barman-cli-cloud 3.19.x: backup.info double-write fails against retention-protected destinations (regression from 3.18.0)

[rtumaykin]

barman-cli-cloud 3.19.x: backup.info double-write fails against retention-protected destinations (regression from 3.18.0)

Searched first

I searched the issue tracker for retention, WORM, object lock, backup.info, immutability, RetentionPolicy, and BAR-1235 and could not find an existing report for this regression on the upload path. The closest adjacent items I found:

• Consider creating backup.info file before running the pre-backup check #570 (closed, feature request — "Consider creating backup.info file before running the pre-backup check") appears to be the conceptual lineage that motivated BAR-1235.
• Feature Request: Use S3 ObjectLock to enforce RecoveryWindowRetentionPolicy #1119 (closed, feature request — "Use S3 ObjectLock to enforce RecoveryWindowRetentionPolicy") relates to object-lock retention semantics but on the delete side.
• Unclear error message when DeleteObjects returns AccessDenied during backup retention #1164 (open — "Unclear error message when DeleteObjects returns AccessDenied during backup retention") is also about retention, but on the delete path rather than the upload path discussed here.

If a duplicate already exists and I missed it, please point me at it and feel free to close this one.

Summary

Since 3.19.0, every cloud backup writes backup.info to the destination twice — once at _start_backup() with status=STARTED and again in the finally: block with status=DONE (or FAILED). On any object store that enforces a retention / immutability window on objects, the second PUT is an intentional overwrite of an object that is still within its retention window, the request is rejected, and every backup ends in FAILED.

3.18.0 wrote backup.info exactly once and works fine against the same destinations.

The double-write lives in the provider-agnostic base class (CloudBackup.coordinate_backup in barman/cloud.py), so the regression applies to all providers (S3, GCS, Azure) — directly observed on GCS; analytically the same code path on S3 with Object Lock or a bucket default retention, and on Azure Blob with a time-based immutability policy or legal hold. See the Provider impact section below.

Affected versions

• Confirmed broken: barman-cli-cloud 3.19.1-1.pgdg22.04+1
• Confirmed working: barman-cli-cloud 3.18.0-2.pgdg22.04+2
• Likely also broken: 3.19.0 (the START-write commit landed for that release)

Environment of the observed failure

• Ubuntu 22.04.5 LTS, Python 3.10.12
• Cloud provider: --cloud-provider=google-cloud-storage
• Bucket: standard GCS bucket with a bucket-retention policy of 90 days (retentionPeriod: 7776000)
• Authentication: GCP service account JSON via GOOGLE_APPLICATION_CREDENTIALS

Reproducer (GCS — what we directly observed)

# any GCS bucket with an active retention policy
gcloud storage buckets update gs://repro-bucket --retention-period=1d

GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json \
  barman-cloud-backup \
    --cloud-provider=google-cloud-storage \
    --lz4 -J 4 -S 5GB --immediate-checkpoint \
    -n test-$(date -u +%Y%m%dT%H%M%S) \
    gs://repro-bucket/barman/ <server-name>

The basebackup itself completes; the final write of backup.info in the finally: block fails:

ERROR: 403 PUT https://storage.googleapis.com/upload/storage/v1/b/<bucket>/o
  ...base/<backup-id>/backup.info...
  {
    "code": 403,
    "message": "Object '<bucket>/...base/<backup-id>/backup.info' is subject
                to bucket's retention policy or object retention and cannot
                be deleted or overwritten until <retention-end>",
    "errors": [{
      "message": "...",
      "domain": "global",
      "reason": "retentionPolicyNotMet"
    }]
  }
ERROR: Backup failed uploading backup.info file (...)

Expected reproducers on other providers

We haven't run these ourselves, but the same code path produces the same shape of failure on any object-retention configuration:

• S3 with Object Lock (Compliance mode, or Governance mode without x-amz-bypass-governance-retention: true), with either a bucket default Object Lock retention setting or per-object retention. Expected response: 403 AccessDenied with reason indicating a retention violation on the second PUT of backup.info.
• Azure Blob with a time-based immutability policy or a legal hold on the container, where upload_blob(..., overwrite=True) is rejected. Azure's response is typically 409 BlobImmutableDueToPolicy / 409 BlobHasImmutabilityPolicy.

Root cause

Introduced in commit d88d385f ("Upload backup.info marked as STARTED at the start of a cloud backup", referencing BAR-1235) and finalized for the 3.19.0 release. The relevant block in barman/cloud.py (CloudBackup.coordinate_backup):

self._start_backup()

# Mark as STARTED and upload backup.info so the backup is visible
# in the catalog immediately.  The finally block will overwrite
# this with the final status (DONE or FAILED).
self.backup_info.set_attribute("status", BackupInfo.STARTED)
self._upload_backup_info()    # ← 1st PUT

paired with the existing call in finally::

finally:
    ...
    self._upload_backup_info()    # ← 2nd PUT, intentional overwrite

coordinate_backup is provider-agnostic (defined on CloudBackup), so this affects every backend.

Provider impact (per source inspection)

None of the providers call _upload_backup_info() through a path that would honor an "if-not-exists" or precondition contract, because coordinate_backup() doesn't pass any such hint:

• GCS (barman/cloud_providers/google_cloud_storage.py): upload_fileobj issues a plain blob.upload_from_file(fileobj) with no preconditions. fail_if_exists raises NotImplementedError. A # TODO: implement a mechanism to avoid overrides comment is already in the code.
• Azure (barman/cloud_providers/azure_blob_storage.py): upload_fileobj calls container_client.upload_blob(..., overwrite=True) — overwrite is explicit. fail_if_exists raises NotImplementedError.
• S3 (barman/cloud_providers/aws_s3.py): fail_if_exists IS implemented (delegates to _put_object), but coordinate_backup() doesn't pass it, so the default path is still an unconditioned s3.meta.client.upload_fileobj(...) and Object Lock will reject the second write.

Notably, the same 3.19.0 release introduced aws_check_object_lock (BAR-1113) to make the delete path WORM-aware on S3. The new START write in BAR-1235 is the equivalent gap on the write path, and it applies to all three providers, not just S3.

Impact

Any user running barman-cli-cloud ≥ 3.19.0 against a destination with any active object-retention / immutability protection — bucket-wide or per-object — will see every backup report FAILED after the basebackup data is uploaded. The data side of the backup is on the object store, but no backup.info is finalized, so the catalog never marks the backup DONE, and barman-cloud-restore and friends won't see the backup as restorable.

Workaround we're using

Pinning barman-cli-cloud and python3-barman at 3.18.0-2.pgdg22.04+2 via apt-mark hold until a fix lands. Posting in case it helps anyone else running into the same on GCS, S3 with Object Lock, or Azure Blob with immutability policies.

---

Thanks for your time looking at this. Happy to provide additional diagnostics, gather logs at a different verbosity, or run targeted repros — just say what would be most useful.

[martinmarques]

I see the issue, which we have to investigate, but I do want to say that on AWS S3 you wouldn't get an error, because S3 lock protects objects by versioning the file. So it creates a new object with each overwrite.

We never supported object store lock protection or WORM on anything that was not S3:

https://docs.pgbarman.org/release/3.19.1/user_guide/barman_cloud.html#object-lock-support

So this would fall more as a feature request

[rtumaykin]

Thanks for the quick look — a few clarifications.

You're right about S3 — that part of the report is mistaken. I extrapolated from the GCS observation without verifying S3 behavior. Per the AWS Object Lock docs, an in-retention PUT to an existing protected key creates a new version rather than failing, so the second _upload_backup_info() would succeed on S3. The S3 paragraph in my "Expected reproducers on other providers" section is wrong; please disregard it. I'd rather not repeat the same mistake on Azure, so I'll mark the Azure bullet as unverified speculation too — the reproducible failure I actually observed is GCS-specific.

On the support scope — understood that Object Lock is only officially supported on S3 per the docs you linked. I want to flag a subtlety for context, not to push back on the classification itself:

In 3.18.0, backup.info was written exactly once (in the finally: block). That single PUT happened to be compatible with retention-policy buckets because there was no overwrite for the policy to reject. Our setup ran cleanly against the same GCS bucket for weeks under 3.18.0. The BAR-1235 change in 3.19.0 added the new START write that the finally: write is intentionally meant to overwrite — and that's the new pattern that conflicts with retention.

If you do triage this as a feature request rather than a regression, would you consider a config flag (e.g. cloud_backup_upload_started_state, default true to preserve the new visibility behavior) that lets users opt out of the START write? That would keep BAR-1235's catalog-visibility win for everyone who wants it, while leaving the previously-working code path available for users on retention-protected non-S3 buckets — without barman needing to take on any Object Lock semantics on GCS or Azure.

Happy to test any shape you'd consider, and to update the issue body to remove the S3 and Azure speculation if that's useful.