[s3] Avoid ListBucket requirement in snapshot commit path#8263
Open
hhhizzz wants to merge 2 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #8261
This PR avoids missing-object status checks in the normal snapshot commit path on object stores. On S3/S3-compatible stores, probing a missing key with
exists/getFileStatuscan require bucketListBucketpermission; without that permission S3A can surface the probe as403 Forbiddeninstead of a cleanfalse/not-found result.The changes are:
exists(snapshot-N)check for object stores inRenamingSnapshotCommit, where external lock protection is already required.snapshot/LATESThint on object stores instead of probingsnapshot-(latest + 1).snapshot-*,LATEST,EARLIEST), use direct put on overwrite writes to avoid Hadoop S3A create/status checks for known metadata objects.Listing is still expected for operations that need to enumerate snapshot files; this only targets the append/commit fast path.
Tests
API and Format
No public API or file format changes.
Documentation
No documentation changes. This is a bug fix for object-store snapshot commit behavior.