Skip to content

Support Azure Event Hubs authentication in the Kafka source via AWS-to-Azure identity federation #7007

Description

@bagmarnikhil

Is your feature request related to a problem? Please describe.

Data Prepper's Kafka source can consume from Apache Kafka and Amazon MSK, but there is no built-in way to consume from Azure Event Hubs' Kafka-compatible endpoint.
Event Hubs requires SASL/OAUTHBEARER authentication against Azure Entra ID (Azure AD), which the Kafka source does not currently support. Today a user wanting to ingest
telemetry from Event Hubs into an AWS pipeline has no first-class option, and stitching it together externally means managing and storing an Azure client secret in the
pipeline configuration — an unnecessary long-lived credential.

Describe the solution you'd like

Add an azure_federated SASL/OAUTHBEARER authentication mechanism to the Kafka source that authenticates to Azure Entra ID using AWS-to-Azure workload-identity
federation
, with no stored Azure secret:

  • At runtime the source resolves AWS credentials through Data Prepper's standard AWS configuration, obtains an AWS web-identity token, and exchanges it at the Azure
    Entra token endpoint for a short-lived OAuth access token that Kafka uses to authenticate. Kafka's built-in login refresher keeps the token current.
  • Configuration under authentication.sasl.azure_federated: azure_tenant_id, azure_client_id, azure_token_endpoint, scope, plus aws.region (required) and an
    optional aws.sts_role_arn.
  • Because AWS credentials flow through the standard AwsCredentialsSupplier, the same pipeline configuration works across different AWS credential setups.

Example:

kafka-source:
  bootstrap_servers:
    - "my-namespace.servicebus.windows.net:9093"
  topics:
    - name: my-eventhub
      group_id: dp-consumer
  authentication:
    sasl:
      azure_federated:
        azure_tenant_id:      "<tenant-id>"
        azure_client_id:      "<app-client-id>"
        azure_token_endpoint: "https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token"
        scope:                "https://my-namespace.servicebus.windows.net/.default"
  aws:
    region: "us-east-1"
  encryption:
    type: ssl

Describe alternatives you've considered (Optional)

  • Static Azure client secret in the SASL config — rejected: stores a long-lived credential in the pipeline, with rotation burden and leakage risk.
  • A separate Event Hubs source plugin — rejected: Event Hubs speaks the Kafka protocol, so reusing the existing Kafka source is the smaller, more consistent surface;
    only the auth mechanism is new.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Fields

No fields configured for issues without a type.

Projects

Status
Unplanned

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions