Deduplicate messages in deleteMessageBatch

[shyam2794]

Background

When there are messages with same id in a single batch , deletion fails with the error BatchEntryIdsNotDistinct . We could dedup the message before calling the DeleteMessageBatchCommand.

From AWS ,

If you receive a message more than once, each time you receive it, you get a different receipt handle. You must provide the most recently received receipt handle when you request to delete the message (otherwise, the message might not be deleted).

we could do something similar to this - https://github.com/aws/aws-sdk-ruby/blob/version-3/gems/aws-sdk-sqs/lib/aws-sdk-sqs/queue_poller.rb#L434

Ideally doing this in AWS sdk will be best, but had skimmed through the sqs-client , they just use the messages which we send as arguments when calling DeleteMessageBatchCommand . So we can dedup the messages before calling sdk's command.

Would be happy to raise a PR for this incase if we agree to implement .

Objectives

Remove the duplicate messages within a single batch when deleting.

[nicholasgriffintn]

Hey, thanks for raising the discussion.

I've not looked too deeply into the problem, but a solution to this problem would make sense if the SDK does not do it inherently, so we'd be open to a PR.

That said, it might be worth seeing if the SDK is willing to do this for JavaScript.

Is there an issue over there already?

[shyam2794]

Have raised an issue in AWS JS SDK - aws/aws-sdk-js-v3#5950 .

[shyam2794]

@nicholasgriffintn the SDK team said they can't support this usecase and asked to handle the deduplication logic at the consumers end. Can i raise a PR for this ?

[nicholasgriffintn]

Hey, sorry for the delay.

It's certainly something we'd be willing to look at.

[shyam2794]

@nicholasgriffintn thanks for the reply . Haven't checked in AWS SDK, will see if there are any existing issues. If not will try raise one with them.