When the S3 source is configured to consume S3 event notifications through SQS, Data Prepper rejects messages whose bucket object contains the awsGeneratedTags field.
The SQS message cannot be deserialized into S3EventNotification because S3BucketEntity does not recognize or ignore this additional field. As a result, the SQS message is received but no event is processed.
This appears to be caused by AWS adding awsGeneratedTags to the S3 event notification payload.
To Reproduce
- Configure an S3 source to consume event notifications from an SQS queue.
- Configure the S3 bucket to send event notifications to that queue.
- Add one object, in my case is compress gz file with the logs
- Allow Data Prepper to poll the SQS queue.
- Observe that the message is rejected and zero messages are processed.
Actual behavior
Data Prepper logs the following parsing error:
2026-07-17T11:09:30,498 [s3-source-sqs-1] ERROR org.opensearch.dataprepper.plugins.source.s3.parser.S3EventNotificationParser - SQS message with message ID:3be54bfc-4bd9-43cf-8e7c-bcafeb0b0836 has invalid body which cannot be parsed into S3EventNotification. Unrecognized field "awsGeneratedTags" (class org.opensearch.dataprepper.plugins.source.s3.S3EventNotification$S3BucketEntity), not marked as ignorable (3 known properties: "ownerIdentity", "name", "arn")
at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: org.opensearch.dataprepper.plugins.source.s3.S3EventNotification["Records"]->java.util.ArrayList[0]->org.opensearch.dataprepper.plugins.source.s3.S3EventNotification$S3EventNotificationRecord["s3"]->org.opensearch.dataprepper.plugins.source.s3.S3EventNotification$S3Entity["bucket"]->org.opensearch.dataprepper.plugins.source.s3.S3EventNotification$S3BucketEntity["awsGeneratedTags"]).
2026-07-17T11:09:30,500 [s3-source-sqs-1] INFO org.opensearch.dataprepper.plugins.source.s3.SqsWorker - Received 1 messages from SQS. Processing 0 messages.
Expected behavior
Data Prepper should successfully process valid S3 event notifications even when AWS includes additional fields such as awsGeneratedTags.
When the S3 source is configured to consume S3 event notifications through SQS, Data Prepper rejects messages whose bucket object contains the
awsGeneratedTagsfield.The SQS message cannot be deserialized into
S3EventNotificationbecauseS3BucketEntitydoes not recognize or ignore this additional field. As a result, the SQS message is received but no event is processed.This appears to be caused by AWS adding
awsGeneratedTagsto the S3 event notification payload.To Reproduce
Actual behavior
Data Prepper logs the following parsing error:
Expected behavior
Data Prepper should successfully process valid S3 event notifications even when AWS includes additional fields such as
awsGeneratedTags.