Add support for AWS IAM Roles for Service Accounts (IRSA)

[Rodrigo Molina (rodrigo-molina)]

Problem

AWS IAM roles for service accounts (IRSA) is a recommended approach for applications such as Kubernetes to authenticate with AWS services without managing static credentials.

From AWS's docs:

In 2014, AWS Identity and Access Management added support for federated identities using OpenID Connect (OIDC). This feature allows you to authenticate AWS API calls with supported identity providers and receive a valid OIDC JSON web token (JWT). You can pass this token to the AWS STS AssumeRoleWithWebIdentity API operation and receive IAM temporary role credentials. You can use these credentials to interact with any AWS service.

Solution

Introduce io.confluent.connect.s3.auth.AwsWebIdentityTokenCredentialsProvider which can be set using the s3.credentials.provider.class connector property.

This provider is a wrapper around AWS’s native WebIdentityTokenCredentialsProvider, similar to how AwsAssumeRoleCredentialsProvider is implemented. It enables configuring IRSA credentials directly via connector properties.

AwsWebIdentityTokenCredentialsProvider configuration properties:

• irsa.role.arn: Role ARN to use when starting a session.
• irsa.session.name: Role session name to use when starting a session.
• irsa.token.file: Path to the web identity token file.

Does this solution apply anywhere else?

• yes
• no

If yes, where?

Any connector that supports instances of AWSCredentialsProvider and Configurable in its configurations.

Test Strategy

The Confluent S3 Sink connector allows custom AWS credential providers via configuration. We have tested this change by deploying a JAR containing the proposed class and using it within the connector's classpath (confluentinc-kafka-connect-avro-converter-7.8.0/lib).

Testing done:

• Unit tests
• Integration tests
• System tests
• Manual tests

Release Plan

This change has no external release dependencies.

[confluent-cla-assistant]

🎉 All Contributor License Agreements have been signed. Ready to merge.
✅ rodrigo-molina
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

[ashwinpankaj]

Thanks Rodrigo Molina (@rodrigo-molina) -

1. Is this being added for CP or CCloud ? can you please share the INIT ?
2. shouldn't this go in the common provider integration repo ? https://github.com/confluentinc/provider-integration/blob/master/src/main/java/io/confluent/provider/integration/aws/v2/ChainedAssumeRoleCredentialsProvider.java

CC: Tarun Jain (@tarunjain-confluent)

[Rodrigo Molina (rodrigo-molina)]

hey ashwinpankaj

Thanks for the quick response! 🙌

1. Is this being added for CP or CCloud ? can you please share the INIT ?

The feature is intended for use in a Self-Hosted Kafka Connect deployment and could be valuable for anyone running Kafka Connect on EKS.

Could you please clarify what INIT refers to in this context?

2. shouldn't this go in the common provider integration repo ? https://github.com/confluentinc/provider-integration/blob/master/src/main/java/io/confluent/provider/integration/aws/v2/ChainedAssumeRoleCredentialsProvider.java

I currently don’t have access to that repository. Let me know if there’s a way to proceed or if I should request access.

[Philipp Trulson (der-eismann)]

We're looking forward to this feature here. How can this proceed ashwinpankaj Tarun Jain (@tarunjain-confluent) ?

[Rodrigo Molina (rodrigo-molina)]

Hey Philipp Trulson (@der-eismann),
As a temporary solution, we're currently building a jar with the AwsWebIdentityTokenCredentialsProvider class and adding it to the S3 plugin dependencies.

When I submitted this pull request, I also wrote some documentation on the process here:
https://medium.com/@molina.rodrigo/how-to-use-iam-roles-for-service-accounts-irsa-in-confluent-aws-s3-sink-connector-563cefe79156

Hope it helps!

[tanvp112]

A much needed feature, wonder why is it lingering?

[Rodrigo Molina (rodrigo-molina)]

tanvp112

Hey ashwinpankaj Tarun Jain (@tarunjain-confluent),

Any suggestions on how to move this forward?

The change itself respects the current repository structure: it extends an existing kafka-connect-storage interface with a wrapper around AWS’s native WebIdentityTokenCredentialsProvider, similar to how AwsAssumeRoleCredentialsProvider is implemented.