File tree Expand file tree Collapse file tree
Expand file tree Collapse file tree Original file line number Diff line number Diff line change @@ -7,6 +7,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
77
88## [ Unreleased]
99
10+ ## [ 1.4.3] - 2025-11-07
11+
12+ ### Fixed
13+ - ** Multipart Upload Caching:**
14+ - Fixed issue where multipart uploads (files >100MB) would always be re-processed even when cached metadata was available
15+ - Added proper multipart ETag caching and comparison to avoid re-extracting and re-uploading files that haven't changed
16+ - Previously, multipart uploads would bypass cache checks because S3 ETag format for multipart uploads differs from simple MD5 hashes
17+
18+ ### Technical Details
19+ - Added ` MultipartETag ` field to ` PartitionCacheEntry ` struct
20+ - Created ` calculateMultipartETagFromFile() ` to compute S3 multipart ETag from temp files using the same 5MB part size as s3manager
21+ - Updated cache storage to calculate and save multipart ETag after upload
22+ - Updated cache retrieval to return multipart ETag along with MD5
23+ - Modified ` shouldSkipPartition ` logic to compare multipart ETags when appropriate
24+ - S3 multipart ETag format: "MD5_of_MD5s-partCount" (e.g., "abc123-56")
25+ - Part size: 5MB (matches s3manager.NewUploader default)
26+ - Multipart threshold: 100MB (matches uploadTempFileToS3 logic)
27+ - Backward compatible: Old cache entries without multipartETag still work
28+
29+ ### Impact
30+ - Eliminates unnecessary re-extraction and re-upload of large files
31+ - Reduces database load from redundant queries
32+ - Improves performance for incremental archival runs
33+ - Maintains data integrity through proper ETag validation
34+
1035## [ 1.4.2] - 2025-11-07
1136
1237### Added
You can’t perform that action at this time.
0 commit comments