Skip to content

[s3] Avoid ListBucket requirement in snapshot commit path#8263

Open
hhhizzz wants to merge 2 commits into
apache:masterfrom
hhhizzz:fix-s3-snapshot-access
Open

[s3] Avoid ListBucket requirement in snapshot commit path#8263
hhhizzz wants to merge 2 commits into
apache:masterfrom
hhhizzz:fix-s3-snapshot-access

Conversation

@hhhizzz

@hhhizzz hhhizzz commented Jun 17, 2026

Copy link
Copy Markdown

Purpose

Linked issue: close #8261

This PR avoids missing-object status checks in the normal snapshot commit path on object stores. On S3/S3-compatible stores, probing a missing key with exists/getFileStatus can require bucket ListBucket permission; without that permission S3A can surface the probe as 403 Forbidden instead of a clean false/not-found result.

The changes are:

  • Skip the pre-commit exists(snapshot-N) check for object stores in RenamingSnapshotCommit, where external lock protection is already required.
  • Verify a failed or direct object-store commit by reading the committed snapshot content instead of probing target existence first.
  • Trust a valid snapshot/LATEST hint on object stores instead of probing snapshot-(latest + 1).
  • For S3 snapshot metadata files (snapshot-*, LATEST, EARLIEST), use direct put on overwrite writes to avoid Hadoop S3A create/status checks for known metadata objects.

Listing is still expected for operations that need to enumerate snapshot files; this only targets the append/commit fast path.

Tests

mvn -nsu -pl paimon-core -am \
  -DskipITs -Dcheckstyle.skip -Dspotless.check.skip=true -Drat.skip=true \
  -DfailIfNoTests=false \
  -Dtest=RenamingSnapshotCommitTest,HintFileUtilsTest test

mvn -nsu -pl paimon-filesystems/paimon-s3-impl -am \
  -DskipITs -Dcheckstyle.skip -Dspotless.check.skip=true -Drat.skip=true \
  -DfailIfNoTests=false \
  -Dtest=S3FileIOTest test

API and Format

No public API or file format changes.

Documentation

No documentation changes. This is a bug fix for object-store snapshot commit behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] S3 snapshot commit requires ListBucket due to missing-object status checks

1 participant