fix(cloud): emit fixed-size multipart upload parts for R2 compatibility#1161
Open
alleneubank wants to merge 4 commits into
Open
fix(cloud): emit fixed-size multipart upload parts for R2 compatibility#1161alleneubank wants to merge 4 commits into
alleneubank wants to merge 4 commits into
Conversation
Cloudflare R2 requires all non-trailing parts in a multipart upload to have identical sizes. The previous CloudTarUploader flushed its buffer whenever it exceeded chunk_size, producing variable-sized parts because compressed output varies per chunk. Refactor CloudTarUploader to fill each disk-backed temp file to exactly chunk_size bytes before uploading, so every non-trailing multipart part is uniform. The trailing part (uploaded via flush/close) may be smaller. This is backward compatible with AWS S3 and other providers that accept uniform parts. Fixes: EnterpriseDB#954
Multi-stage build producing a minimal image with barman-cloud tools (backup, wal-archive, restore, wal-restore) and snappy compression support. Uses --no-deps install to avoid building psycopg2 from source, substituting psycopg2-binary for container use. Image: ghcr.io/0xsend/barman:3.17.0-r2fix
…fety - Move chunk_size validation to __init__ (fail fast on bad config) - Add is_final flag to _upload_part_buffer to enforce uniform part sizes for non-trailing parts (R2 requirement) - Preserve buffer and part counter on upload failure for retry callers - Add tests for validation, final-part semantics, and failure recovery
Contributor
|
Hello, we had an initial look at the PR, but couldn't get to it due to conflicting priorities. We want to have a closer look at the proposal. We'll get back to you here after the next release is over. |
GureevLeonid
added a commit
to maestra-io/barman
that referenced
this pull request
May 6, 2026
Standalone barman-cloud CLI image (python:3.13-slim + snappy + lz4 + boto3). Built with our patched cloud.py, pushed to public ECR public.ecr.aws/g5f9s8a4/barman on main + tags via the existing Teleport-Workload-Identity → AWS-STS pattern. Why this fork exists: PR EnterpriseDB#1161 (R2-compatible multipart parts) is open since 2026-02 against EDB/barman with no upstream review traction; we need it shipped to fix daily backups on omicron's R2 buckets for any postgres relation file >5 MiB. Retire when 3.19+ merges the fix. See MAESTRA_FORK.md for branch layout and consumption notes.
|
is this still on the table? |
CloudNativePG's in-tree barmanObjectStore runs barman-cloud-backup inside the postgres operand image, so the R2 fix ships as an overlay image: ghcr.io/0xsend/cnpg-postgresql:18-r2 built from the upstream operand with the forked barman force-reinstalled over stock 3.17.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CloudTarUploaderto fill each disk-backed temp file to exactlychunk_sizebytes before uploading, so every non-trailing multipart part is uniform in sizechunk_size, producing variable-sized partsflush/close) may be smaller, which is allowed by the multipart upload specChanges
barman/cloud.py: Extract_write_data()(fills fixed-size part buffers) and_upload_part_buffer()(uploads a single part) from the existingwrite()/flush()methodstests/test_cloud.py: Add two new tests verifying partial-buffer behavior and fixed-size part emission; fix existing throttle test to set up buffer state correctlyTest plan
test_write_uses_disk_buffer_for_partial_data— partial writes stay in buffer, no upload triggeredtest_write_emits_fixed_size_parts_and_retains_tail— 10 bytes at chunk_size=4 produces two 4-byte parts + 2-byte tailTestCloudTarUploadertests continue to passFixes: #954