REST catalog with S3T

[gsoundar]

I'm trying to use the REST catalog (https://github.com/apache/iceberg-rust/blob/main/crates/catalog/rest/src/catalog.rs) with S3Tables similar to what is described in https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-open-source.html.

The rough snippet of code is:

    let mut catalog_props = HashMap::new();
    catalog_props.insert("warehouse".to_string(), TABLE_BUCKET_ARN.to_string());
    catalog_props.insert("rest.sigv4-enabled".to_string(), "true".to_string());
    catalog_props.insert("rest.signing-name".to_string(), "s3tables".to_string());
    catalog_props.insert("rest.signing-region".to_string(), region.to_string());
    let catalog_config = RestCatalogConfig::builder()
        .uri(TABLE_BUCKET_CATALOG.to_string())
        .props(catalog_props)
        .build();
    let catalog = RestCatalog::new(catalog_config);

and I get the error

Context:
   status: 403 Forbidden
   headers: {"date": "Wed, 23 Apr 2025 19:24:10 GMT", "content-type": "application/json", "content-length": "42", "connection": "keep-alive", "x-amzn-requestid": "XXXXX", "access-control-allow-origin": "*", "x-amzn-errortype": "MissingAuthenticationTokenException", "x-amz-apigw-id": "XXXXX"}
   json: {"message":"Missing Authentication Token"}

I think it's because we don't have Sigv4 suppport in the code. Anyone confirm this is the RCA ?
Also does anyone know how Sigv4 support can be plumbed in, maybe through the HttpClient perhaps ?

Thanks!

[liurenjie1024]

Hi, there is a S3TableCatalog, could you have a try?

[fstephany]

Yes, the doc says This can be used to load credentials from a custom source, such as the AWS SDK. but I cannot get it to work. I don't understand how it is supposed to be constructed.

I see that AwsDefaultLoader implements the trait AwsCredentialLoad. It should be able to load credentials from ~/.aws/config. But it looks like reqsign was heavily updated since 0.16 and the 0.20 version is not (yet?) used by iceberg-rust.

I'll let you know if/when I manage to get it working.

[fstephany]

It looks like the issue was not the profile loading but the SSO configuration under that profile. I'll wait that iceberg-rust update to the latest reqsign to have this built-in (or use the REST catalog when ready).

[dannycjones]

Perhaps you can create a new type around a Rust SDK credential provider implementing AwsCredentialLoad in the meantime?

Rust SDK's credential chain implements ProvideCredential, which I'd say is the equivalent of AwsCredentialLoad. https://docs.rs/aws-config/latest/aws_config/default_provider/credentials/struct.DefaultCredentialsChain.html#impl-ProvideCredentials-for-DefaultCredentialsChain

[fstephany]

It works with the following (beware, super naive code in here 😁 ):

struct NonRenewedCreds(AwsCredential);

#[async_trait]
impl iceberg_storage_opendal::AwsCredentialLoad for NonRenewedCreds {
    async fn load_credential(
        &self,
        _: reqwest::Client,
    ) -> Result<Option<AwsCredential>, anyhow::Error> {
        futures::future::ok(Some(self.0.clone())).await
    }
}

let cfg = aws_config::from_env()
        .profile_name("profile-name-to-use")
        .load()
        .await;
let credential_loader = cfg.credentials_provider().unwrap();
let creds = credential_loader.provide_credentials().await.unwrap();
let opendal_creds = AwsCredential {
    access_key_id: creds.access_key_id().to_string(),
    secret_access_key: creds.secret_access_key().to_string(),
    session_token: creds.session_token().map(ToOwned::to_owned),
    expires_in: None,
};
let naive_creds = NonRenewedCreds(opendal_creds);
let custom_loader = CustomAwsCredentialLoader::new(Arc::new(naive_creds));

// Regular AWS SDK Client
let s3_tables_client = aws_sdk_s3tables::Client::new(&cfg);

let catalog = S3TablesCatalogBuilder::default()
    .with_client(s3_tables_client)
    .with_storage_factory(Arc::new(OpenDalStorageFactory::S3 {
        configured_scheme: "s3".to_string(), // Use s3:// instead of s3a://
        customized_credential_load: Some(custom_loader),
    }))

[fstephany]

I had to add futures, anyhow, reqsign, reqwest and async-trait to my dependencies. The most annoying one is anyhow as I don't use it.

[dannycjones]

AWS SigV4 isn't supported for the REST Catalog implementation yet, there's a tracking issue here: #1236

In the meantime, there is an S3 Tables catalog implementation available: https://crates.io/crates/iceberg-catalog-s3tables