Is your feature request related to a problem? Please describe.
Data Prepper's Kafka source can consume from Apache Kafka and Amazon MSK, but there is no built-in way to consume from Azure Event Hubs' Kafka-compatible endpoint.
Event Hubs requires SASL/OAUTHBEARER authentication against Azure Entra ID (Azure AD), which the Kafka source does not currently support. Today a user wanting to ingest
telemetry from Event Hubs into an AWS pipeline has no first-class option, and stitching it together externally means managing and storing an Azure client secret in the
pipeline configuration — an unnecessary long-lived credential.
Describe the solution you'd like
Add an azure_federated SASL/OAUTHBEARER authentication mechanism to the Kafka source that authenticates to Azure Entra ID using AWS-to-Azure workload-identity
federation, with no stored Azure secret:
- At runtime the source resolves AWS credentials through Data Prepper's standard AWS configuration, obtains an AWS web-identity token, and exchanges it at the Azure
Entra token endpoint for a short-lived OAuth access token that Kafka uses to authenticate. Kafka's built-in login refresher keeps the token current.
- Configuration under
authentication.sasl.azure_federated: azure_tenant_id, azure_client_id, azure_token_endpoint, scope, plus aws.region (required) and an
optional aws.sts_role_arn.
- Because AWS credentials flow through the standard
AwsCredentialsSupplier, the same pipeline configuration works across different AWS credential setups.
Example:
kafka-source:
bootstrap_servers:
- "my-namespace.servicebus.windows.net:9093"
topics:
- name: my-eventhub
group_id: dp-consumer
authentication:
sasl:
azure_federated:
azure_tenant_id: "<tenant-id>"
azure_client_id: "<app-client-id>"
azure_token_endpoint: "https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token"
scope: "https://my-namespace.servicebus.windows.net/.default"
aws:
region: "us-east-1"
encryption:
type: ssl
Describe alternatives you've considered (Optional)
- Static Azure client secret in the SASL config — rejected: stores a long-lived credential in the pipeline, with rotation burden and leakage risk.
- A separate Event Hubs source plugin — rejected: Event Hubs speaks the Kafka protocol, so reusing the existing Kafka source is the smaller, more consistent surface;
only the auth mechanism is new.
Is your feature request related to a problem? Please describe.
Data Prepper's Kafka source can consume from Apache Kafka and Amazon MSK, but there is no built-in way to consume from Azure Event Hubs' Kafka-compatible endpoint.
Event Hubs requires SASL/OAUTHBEARER authentication against Azure Entra ID (Azure AD), which the Kafka source does not currently support. Today a user wanting to ingest
telemetry from Event Hubs into an AWS pipeline has no first-class option, and stitching it together externally means managing and storing an Azure client secret in the
pipeline configuration — an unnecessary long-lived credential.
Describe the solution you'd like
Add an
azure_federatedSASL/OAUTHBEARER authentication mechanism to the Kafka source that authenticates to Azure Entra ID using AWS-to-Azure workload-identityfederation, with no stored Azure secret:
Entra token endpoint for a short-lived OAuth access token that Kafka uses to authenticate. Kafka's built-in login refresher keeps the token current.
authentication.sasl.azure_federated:azure_tenant_id,azure_client_id,azure_token_endpoint,scope, plusaws.region(required) and anoptional
aws.sts_role_arn.AwsCredentialsSupplier, the same pipeline configuration works across different AWS credential setups.Example:
Describe alternatives you've considered (Optional)
only the auth mechanism is new.