Component(s)
loki.source.kafka
Request
Expose Sarama's Config.RackID as an optional rack_id argument in loki.source.kafka.
When brokers run with replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector, a consumer that sends a rack id fetches from the closest replica instead of the partition leader (KIP-392). Sarama supports this already, the component just doesn't wire it through. otelcol.receiver.kafka already exposes rack_id
Use case
We run Kafka and Alloy across 3 availability zones and all log traffic goes through loki.source.kafka. Today every fetch hits the partition leader, so roughly 2/3 of consumed bytes cross zones and are billed as inter-AZ egress. With rack_id set to the pod's zone this traffic stays zone-local.
Mimir does exactly this on its Kafka read path (-ingest-storage.kafka.client-rack, added in grafana/mimir#14434) — we'd like the same for logs.
AI disclosure
Tip
React with 👍 if this issue is important to you.
Component(s)
loki.source.kafka
Request
Expose Sarama's
Config.RackIDas an optionalrack_idargument inloki.source.kafka.When brokers run with
replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector, a consumer that sends a rack id fetches from the closest replica instead of the partition leader (KIP-392). Sarama supports this already, the component just doesn't wire it through.otelcol.receiver.kafkaalready exposesrack_idUse case
We run Kafka and Alloy across 3 availability zones and all log traffic goes through
loki.source.kafka. Today every fetch hits the partition leader, so roughly 2/3 of consumed bytes cross zones and are billed as inter-AZ egress. Withrack_idset to the pod's zone this traffic stays zone-local.Mimir does exactly this on its Kafka read path (
-ingest-storage.kafka.client-rack, added in grafana/mimir#14434) — we'd like the same for logs.AI disclosure
Tip
React with 👍 if this issue is important to you.