Skip to content

Support S3 One Zone-IA directory buckets in AWS Local Zones #10523

Description

@egummett-aws

Amazon S3 directory buckets are available in selected AWS Local Zones. They use the S3 One Zone-Infrequent Access (One Zone-IA) storage class, rather than S3 Express One Zone. See Unlocking data residency use cases with Amazon S3 in AWS Local Zones, and the directory bucket documentation.

lakeFS already supports S3 Express One Zone directory buckets, adding Git-like version control above a bucket type that does not support S3 Versioning. I tested lakeFS v1.86.0 against a new One Zone-IA directory bucket in the Istanbul Local Zone (eu-central-1, euc1-ist1-az1).

With this configuration:

blockstore:
  s3:
    region: eu-central-1
    discover_bucket_region: false

lakeFS successfully completed:

  • repository creation;
  • presigned object upload and download;
  • branch creation;
  • commit;
  • diff;
  • merge;
  • logical deletion; and
  • historical recovery by creating a branch at the pre-delete commit and reading the deleted object.

lakeFS wrote both its immutable _lakefs/ metadata objects and repository data objects to the One Zone-IA directory bucket.

With the default discover_bucket_region: true, repository creation fails. lakeFS attempts bucket-region discovery using HeadBucket, then panics in the AWS SDK S3 Express credentials path with a nil-pointer dereference. The API client sees EOF. Disabling discovery avoids that code path because the configured blockstore.s3.region is already correct.

I think this needs two things:

  1. Handle directory-bucket region discovery failures without panicking.
  2. Test and, if appropriate, add official support for One Zone-IA directory buckets in Local Zones.

The workflow above did not need directory-bucket append or rename APIs. I’m happy to collaborate on a reproducible integration test or provide the full setup and stack trace.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions