Skip to content

[BUG] S3 source fails to parse SQS notifications containing awsGeneratedTags #7005

Description

@alwashali

When the S3 source is configured to consume S3 event notifications through SQS, Data Prepper rejects messages whose bucket object contains the awsGeneratedTags field.

The SQS message cannot be deserialized into S3EventNotification because S3BucketEntity does not recognize or ignore this additional field. As a result, the SQS message is received but no event is processed.

This appears to be caused by AWS adding awsGeneratedTags to the S3 event notification payload.

To Reproduce

  1. Configure an S3 source to consume event notifications from an SQS queue.
  2. Configure the S3 bucket to send event notifications to that queue.
  3. Add one object, in my case is compress gz file with the logs
  4. Allow Data Prepper to poll the SQS queue.
  5. Observe that the message is rejected and zero messages are processed.

Actual behavior

Data Prepper logs the following parsing error:

2026-07-17T11:09:30,498 [s3-source-sqs-1] ERROR org.opensearch.dataprepper.plugins.source.s3.parser.S3EventNotificationParser - SQS message with message ID:3be54bfc-4bd9-43cf-8e7c-bcafeb0b0836 has invalid body which cannot be parsed into S3EventNotification. Unrecognized field "awsGeneratedTags" (class org.opensearch.dataprepper.plugins.source.s3.S3EventNotification$S3BucketEntity), not marked as ignorable (3 known properties: "ownerIdentity", "name", "arn")
 at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: org.opensearch.dataprepper.plugins.source.s3.S3EventNotification["Records"]->java.util.ArrayList[0]->org.opensearch.dataprepper.plugins.source.s3.S3EventNotification$S3EventNotificationRecord["s3"]->org.opensearch.dataprepper.plugins.source.s3.S3EventNotification$S3Entity["bucket"]->org.opensearch.dataprepper.plugins.source.s3.S3EventNotification$S3BucketEntity["awsGeneratedTags"]).
2026-07-17T11:09:30,500 [s3-source-sqs-1] INFO  org.opensearch.dataprepper.plugins.source.s3.SqsWorker - Received 1 messages from SQS. Processing 0 messages.

Expected behavior

Data Prepper should successfully process valid S3 event notifications even when AWS includes additional fields such as awsGeneratedTags.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    Done

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions