The SQS source plugin defaults to acknowledgments: false, which causes messages to be deleted from SQS after being written to the internal pipeline buffer but before the downstream sink confirms successful delivery. If the pipeline fails after buffering (OpenSearch unavailable, processor error, worker crash), messages are permanently lost with no recovery path.
This affects all users who don't explicitly set acknowledgments: true in their pipeline configuration.
What solution would you like?
Change the default value of acknowledgments from false to true in SqsSourceConfig, so messages are only deleted from SQS after the entire pipeline confirms end-to-end delivery. This aligns with the defaults used by iceberg-source and rds-source.
Additionally:
- Log a WARNING when acknowledgments is explicitly disabled
- Add a metric (sqsMessagesDeletedWithoutAcknowledgment) to track messages deleted without end-to-end confirmation
- Document the data loss risk and visibility timeout implications
What alternatives have you considered?
- Requiring explicit configuration via @NotNull Boolean — rejected as too disruptive (breaks all existing configs)
- Keeping the default but adding documentation only — rejected as insufficient for a silent data loss issue
Additional context
This is a breaking change. Users who explicitly require best-effort (fire-and-forget) behavior should set
acknowledgments: false in their pipeline configuration. Users should ensure their SQS visibility timeout exceeds
the expected end-to-end processing time to avoid duplicate delivery.
Affected file:
data-prepper-plugins/sqs-source/src/main/java/org/opensearch/dataprepper/plugins/source/sqs/SqsSourceConfig.java
The SQS source plugin defaults to acknowledgments: false, which causes messages to be deleted from SQS after being written to the internal pipeline buffer but before the downstream sink confirms successful delivery. If the pipeline fails after buffering (OpenSearch unavailable, processor error, worker crash), messages are permanently lost with no recovery path.
This affects all users who don't explicitly set acknowledgments: true in their pipeline configuration.
What solution would you like?
Change the default value of acknowledgments from false to true in SqsSourceConfig, so messages are only deleted from SQS after the entire pipeline confirms end-to-end delivery. This aligns with the defaults used by iceberg-source and rds-source.
Additionally:
What alternatives have you considered?
Additional context
This is a breaking change. Users who explicitly require best-effort (fire-and-forget) behavior should set
acknowledgments: false in their pipeline configuration. Users should ensure their SQS visibility timeout exceeds
the expected end-to-end processing time to avoid duplicate delivery.
Affected file:
data-prepper-plugins/sqs-source/src/main/java/org/opensearch/dataprepper/plugins/source/sqs/SqsSourceConfig.java