S3 supports "full-object" checksum validation for multipart uploads:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity-upload.html
The lakeFS API doesn't expose any of that and neither does the lakefs Python SDK, making it impossible to validate large multipart uploads.
Gaps
Compared with the S3 multipart API, the lakeFS multipart API/SDK is missing:
- Create-time checksum fields: S3 supports checksum configuration when initiating a multipart upload
- Part-level checksum fields: S3 UploadPart supports checksum-related request parameters/headers
- Completion-time full-object checksum fields: S3 multipart completion supports sending the final full-object checksum and multipart object size
- Corresponding SDK surface: the generated Python SDK mirrors those omissions, so the checksum information cannot be supplied from clients even before reaching the backend
S3 supports "full-object" checksum validation for multipart uploads:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity-upload.html
The lakeFS API doesn't expose any of that and neither does the
lakefsPython SDK, making it impossible to validate large multipart uploads.Gaps
Compared with the S3 multipart API, the lakeFS multipart API/SDK is missing: