Skip to content

Support "full-object" checksum validation for multipart uploads #10314

Description

@boukeas

S3 supports "full-object" checksum validation for multipart uploads:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity-upload.html

The lakeFS API doesn't expose any of that and neither does the lakefs Python SDK, making it impossible to validate large multipart uploads.

Gaps

Compared with the S3 multipart API, the lakeFS multipart API/SDK is missing:

  • Create-time checksum fields: S3 supports checksum configuration when initiating a multipart upload
  • Part-level checksum fields: S3 UploadPart supports checksum-related request parameters/headers
  • Completion-time full-object checksum fields: S3 multipart completion supports sending the final full-object checksum and multipart object size
  • Corresponding SDK surface: the generated Python SDK mirrors those omissions, so the checksum information cannot be supplied from clients even before reaching the backend

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions