Component(s)
exporter/opensearch
What happened?
When OpenSearch is unreachable or a network failure occurs, logs and traces are silently dropped — even when retry_on_failure is fully configured. The retry machinery is never invoked.
Root Cause
onIndexerError in both logBulkIndexer and traceBulkIndexer unconditionally wraps every error as consumererror.Permanent:
func (lbi *logBulkIndexer) onIndexerError(_ context.Context, indexerErr error) {
if indexerErr != nil {
lbi.appendPermanentError(consumererror.NewPermanent(indexerErr))
}
}This fires for infrastructure-level failures (connection refused, timeout, DNS failure) — errors that are always transient. Marking them Permanent causes exporterhelper to drop the data immediately without retrying.
Additionally, the default case in processItemFailure marks all status=0 errors as Permanent, including net.OpError (transport failures) which should be retryable.
Expected Behavior
Transient network failures should be retried by exporterhelper. Only truly unrecoverable errors (bad documents, encoding failures, auth errors) should be Permanent.
Proposed Fix
// onIndexerError — never permanent at infrastructure level
func (lbi *logBulkIndexer) onIndexerError(_ context.Context, indexerErr error) {
if indexerErr != nil {
lbi.errs = append(lbi.errs, indexerErr) // retryable, not permanent
}
}
// processItemFailure — distinguish transport vs encoding errors
default:
var netErr *net.OpError
if errors.As(itemErr, &netErr) {
lbi.appendRetryLogError(itemErr, logs) // transport error, retryable
} else {
lbi.appendPermanentError(itemErr) // encoding error, permanent
}Same fix applies to traceBulkIndexer.
Affected Versions
Tested on current main branch.
Severity
Data Loss — silent, no error surfaced to the user despite retry_on_failure being enabled.
Collector version
0.152.0
Environment information
Environment
OS: (e.g., "Centos 7")
OpenTelemetry Collector configuration
receivers:
kafka:
brokers: ["brokerHost:9092"]
protocol_version: 2.0.0
group_id: consumer-group
logs:
topics: topic
encoding: otlp_proto
processors:
memory_limiter:
limit_mib: 4000
spike_limit_mib: 800
check_interval: 1s
batch:
send_batch_size: 2000
send_batch_max_size: 2000
exporters:
opensearch:
http:
endpoint: "opensearch-endpoint:9200"
auth:
authenticator: basicauth/client
tls:
insecure_skip_verify: true
logs_index: "logs"
mapping:
mode: "flatten_attributes"
dedot: true
dedup: true
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 0s
sending_queue:
enabled: true
num_consumers: 15
queue_size: 1000000
storage: file_storage
block_on_overflow: true
debug:
verbosity: basic
extensions:
file_storage:
directory: /data1/otelcol-contrib/exporter-queue
basicauth/client:
client_auth:
username: "#USER#"
password: "#PASSWORD#"
health_check:
endpoint: 0.0.0.0:13133
service:
telemetry:
logs:
level: info
encoding: json
output_paths:
- "/var/log/otelcol/otelcol.log"
error_output_paths:
- "/var/log/otelcol/otelcol-error.log"
resource:
service.name: "otel-egress-collector"
service.instance.id: "1"
metrics:
level: detailed
readers:
- pull:
exporter:
prometheus:
host: '0.0.0.0'
port: 8888
# Explicitly include the resource attribute as a label
with_resource_constant_labels:
included:
- service.instance.id
- service.name
extensions: [file_storage, basicauth/client, health_check]
pipelines:
logs:
receivers: [kafka]
processors: [memory_limiter, batch]
exporters: [opensearch, debug]Log output
Additional context
Attaching the PR for this Fix #49207
Component(s)
exporter/opensearch
What happened?
When OpenSearch is unreachable or a network failure occurs, logs and traces are silently dropped — even when
retry_on_failureis fully configured. The retry machinery is never invoked.Root Cause
onIndexerErrorin bothlogBulkIndexerandtraceBulkIndexerunconditionally wraps every error asconsumererror.Permanent:This fires for infrastructure-level failures (connection refused, timeout, DNS failure) — errors that are always transient. Marking them
Permanentcausesexporterhelperto drop the data immediately without retrying.Additionally, the
defaultcase inprocessItemFailuremarks allstatus=0errors asPermanent, includingnet.OpError(transport failures) which should be retryable.Expected Behavior
Transient network failures should be retried by
exporterhelper. Only truly unrecoverable errors (bad documents, encoding failures, auth errors) should bePermanent.Proposed Fix
Same fix applies to
traceBulkIndexer.Affected Versions
Tested on current
mainbranch.Severity
Data Loss — silent, no error surfaced to the user despite
retry_on_failurebeing enabled.Collector version
0.152.0
Environment information
Environment
OS: (e.g., "Centos 7")
OpenTelemetry Collector configuration
Log output
Additional context
Attaching the PR for this Fix #49207