feat: support presigned multipart upload (PMPU) on GCS

[ggoggam]

Change Description

Background

lakeFS already emulates multipart upload on GCS: CreateMultiPartUpload drops a marker object, each part is written as its own object (<key>.part_00001), and CompleteMultiPartUpload composes them. What was missing is the presign multipart flow, which was S3-only — the GS adapter returned ErrOperationNotSupported from GetPresignUploadPartURL and never set PreSignSupportMultipart.

The effect is that on GCS, POST /repositories/{repo}/branches/{branch}/staging/pmpu answers 501, and helpers.PreSignUploader (so lakectl fs upload --pre-sign, lakectl local, and anything else on the Go client) falls back to a single pre-signed PUT for the whole object, no matter how large.

Since a GCS part upload is just a PUT of the part's own object, it can be pre-signed exactly the way a regular write already is.

New Feature

Presigned multipart upload on the GS block adapter.

• gs.Adapter.GetPresignUploadPartURL signs a V4 PUT URL for formatMultipartFilename(uploadID, partNumber) — the same object UploadPart writes to — so the rest of the flow (ListParts, validation, compose, abort) is unchanged. Signing options were factored into newSignedURLOptions, shared with GetPreSignedURL.
• GetStorageNamespaceInfo reports PreSignSupportMultipart, which is what the API and lakectl gate on.
• New blockstore.gs.disable_pre_signed_multipart setting, mirroring the existing S3 one.
• Support is off when blockstore.gs.server_side_encryption_customer_supplied is set. A pre-signed PUT writes the part without the customer-supplied key, and GCS requires the same key across every compose source and the destination, so the complete would fail. The adapter reports no support and returns ErrOperationNotSupported rather than handing out a URL that produces an unusable upload.

One client-side fix comes along with it: helpers.presignedURLExpired only recognized X-Amz-Date/X-Amz-Expires, so a GCS URL parsed as "expired" and every part paid for a needless UploadPart refresh round-trip before uploading. It now reads the X-Goog- parameters too.

Testing Details

• pkg/block/gs: new TestPresignMultipartUpload runs the whole flow against fake-gcs-server — create, pre-sign each part, PUT the bytes straight at the store over HTTP, list parts, complete, read the object back and compare. TestGetPresignUploadPartURL covers the signed URL targeting the right part object, part-number bounds, both disable switches, and the CSEK case.
• pkg/block/blocktest: Adapter_PresignUploadPartURL added to the shared multipart suite, asserting every adapter agrees with its own PreSignSupportMultipart flag. Runs for gs/s3/azure/local/mem.
• pkg/api/helpers: TestPresignedURLExpired covers S3 and GCS URLs, fresh and stale, plus the unparseable cases that should fail safe.
• esti: skipPresignMultipart now gates on the capability the server reports instead of blockstore_type == "s3", so the existing esti presign multipart tests cover GCS deployments and still skip where the flow is disabled.

go test ./pkg/block/... ./pkg/config/... ./pkg/api/helpers/... and golangci-lint run ./pkg/... (plus --build-tags esti ./esti/...) pass locally.

One note on the emulator: fake-gcs-server does not return an ETag header on an XML API upload, so the end-to-end test reads the stored part ETags via ListParts rather than the ones a real GCS PUT response would carry. The ETag-is-hex-MD5 assumption the complete path validates against is the same one the existing GCS single-object presign upload already relies on.

Breaking Change?

No. Additive — a new capability on a blockstore that previously reported none, and a new opt-out setting. Existing GCS deployments get presigned multipart uploads for large files; setting blockstore.gs.disable_pre_signed_multipart: true restores the previous behavior. No API, CLI, or client surface changed.

Additional info

The API surface already existed and needed no changes — CreatePresignMultipartUpload / UploadPart / CompletePresignMultipartUpload / AbortPresignMultipartUpload all route through the block adapter and simply work once the adapter advertises support.

Contact Details

@ggoggam on GitHub.

[CLAassistant]

All committers have signed the CLA.