feat: add --generic-hostname-worker-task-metric flag - #388
Open
bafulton wants to merge 1 commit into
Open
Conversation
Mirror --generic-hostname-task-sent-metric for the worker-executed metrics: label the celery_task_* counters and celery_task_runtime with a generic hostname instead of the executing worker's, to bound label cardinality when workers are autoscaled (for example Kubernetes/KEDA pods). track_task_event builds a single labels dict that feeds every counter and the runtime observation, so one gated assignment collapses them together. celery_task_sent keeps its own flag, and celery_worker_up / celery_worker_tasks_active are unaffected since they do not pass through track_task_event. Refs danihodovic#387
bafulton
force-pushed
the
feat/generic-hostname-worker-task-metric
branch
from
July 24, 2026 15:45
837eea6 to
6e5e924
Compare
bafulton
marked this pull request as ready for review
July 24, 2026 15:47
Author
|
@danihodovic, I'd love to get your thoughts when you have a moment! I'm currently running into cardinality issues in my prod environment; Celery Exporter is generating more than 90,000 independent metrics and it's breaking prometheus and datadog scrapes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The executing worker's
hostnamelabel drives task-metric cardinality on autoscaled deployments:celery_task_runtimeand thecelery_task_*counters are all labeled with the worker's hostname, which in Kubernetes/KEDA setups is an ephemeral pod name. Thename × hostname × queueset therefore churns and grows with the worker fleet.--generic-hostname-task-sent-metricsolves this on the client side, but there is no equivalent for the worker-executed metrics.Solution
Add
--generic-hostname-worker-task-metric, mirroring the task-sent flag: it labels thecelery_task_*counters andcelery_task_runtimewith a generic hostname instead of the worker's.track_task_eventbuilds a singlelabelsdict that feeds every counter and the runtime observation, so one gated assignment collapses them together.celery_task_sentkeeps its own flag so the two compose, andcelery_worker_up/celery_worker_tasks_activeare untouched since they do not pass throughtrack_task_event.The test mirrors the existing
generic_hostname_task_sent_metrictest and passes on the memory, redis, and rabbitmq brokers.Refs #387