Additional metrics exported from Celery workers by rbagd · Pull Request #3463 · open-telemetry/opentelemetry-python-contrib

rbagd · 2025-05-05T10:31:09Z

Description

This PR includes an enhancement regarding metrics exported from Celery workers and implements these measurements

Prefetched task up-down counter (number of tasks queued locally in the worker) and time spent in prefetch mode
Active task up-down counter and task processing duration

Currently, there's only one metric being exported - flower.task.runtime.seconds, and it was renamed to follow semantic conventions (reported in the changelog).

Changes in this PR also fix a memory leak present in current instrumentation.

Fixes #3458

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

How Has This Been Tested?

Unit tests added for all newly included and updated metrics

Does This PR Require a Core Repo Change?

No.

Checklist:

Followed the style guidelines of this project
Changelogs have been updated
Unit tests have been added

rbagd · 2025-05-22T15:21:36Z

@xrmx @emdneto May I ask one of you to give a first look at this? I understand there's nobody who owns Celery component, so I'm afraid this will drift into oblivion.

bschoenmaeckers · 2025-05-28T12:38:25Z

It would also be nice to have task result counters, like the following

total finished tasks
total sucessfull tasks
total failed tasks
total retried tasks

rbagd · 2025-06-23T12:23:29Z

It would also be nice to have task result counters, like the following
* total finished tasks

* total sucessfull tasks

* total failed tasks

* total retried tasks

Agreed that it'd be nice to have. I don't want to increase the scope of this MR as it's already quite large but I can look into this in the near future.

...on/opentelemetry-instrumentation-celery/src/opentelemetry/instrumentation/celery/__init__.py

xrmx · 2025-06-30T09:27:34Z

...on/opentelemetry-instrumentation-celery/src/opentelemetry/instrumentation/celery/__init__.py

 _TASK_NAME_KEY = "celery.task_name"

+# Metric names
+_TASK_COUNT_ACTIVE = "messaging.client.active_tasks"


These 3 new metrics are not in semantic conventions right? I don't think we should add them to the same namespace as the others. If adding unspecified metrics at all.

Just to summarize my viewpoint

# Already in semconv * _TASK_PROCESSING_TIME = messaging_metrics.MESSAGING_PROCESS_DURATION # Probably should be in semconv * _TASK_COUNT_ACTIVE = "messaging.client.active_tasks" # Debatable as this is is more linked to Celery * _TASK_COUNT_PREFETCHED = "messaging.client.prefetched_tasks" * _TASK_PREFETCH_TIME = "messaging.prefetch.duration"

By different namespace, do you mean replacing messaging by something like celery for the last two (or three) of these?

xrmx · 2025-06-30T09:29:15Z

CHANGELOG.md

+
+### Fixed
+
+- `opentelemetry-instrumentation-celery` Fix a memory leak where a reference to a task identifier is kept indefinitely


Could you please extract this change in another PR so we can get this reviewed and hopefully merged before next release?

I would say the same. The memory leak seems more feasible to review and merge.

I opened this small PR to just fix the leak.

emdneto · 2025-06-30T16:52:07Z

...on/opentelemetry-instrumentation-celery/src/opentelemetry/instrumentation/celery/__init__.py

-
-    def create_celery_metrics(self, meter) -> None:
-        self.metrics = {
-            "flower.task.runtime.seconds": meter.create_histogram(


need to discuss if we can keep this and the new metric to avoid breakage.

It's fine of course to keep this for backwards compatibility, albeit not sure it's worth it in this case. Metric name has nothing to do with OTel or semantic conventions, and references an unrelated project. :/

rbagd · 2025-08-13T07:06:43Z

Sorry for delays - summer time and all that. I'll go through the review comments.

diogosilva30 · 2026-02-26T20:02:57Z

What's the status of this MR? @rbagd
Need help?

github-actions · 2026-03-13T03:57:33Z

This PR has been automatically marked as stale because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 days of this comment.
If you're still working on this, please add a comment or push new commits.

github-actions · 2026-03-27T04:07:07Z

This PR has been closed due to inactivity. Please reopen if you would like to continue working on it.

rbagd requested a review from a team as a code owner May 5, 2025 10:31

rbagd force-pushed the include_more_celery_metrics branch 4 times, most recently from 3755a64 to 8867346 Compare May 5, 2025 11:40

rbagd force-pushed the include_more_celery_metrics branch 2 times, most recently from 09038ae to 2d192ca Compare May 13, 2025 09:40

rbagd force-pushed the include_more_celery_metrics branch from 2d192ca to 63b559b Compare May 22, 2025 10:02

rbagd mentioned this pull request May 26, 2025

Flaky celery test with pypy-3.8 ubuntu #3521

Open

rbagd force-pushed the include_more_celery_metrics branch from 2221552 to e9e80e1 Compare May 26, 2025 07:05

rbagd force-pushed the include_more_celery_metrics branch from 99db9da to a417078 Compare June 11, 2025 11:08

rbagd force-pushed the include_more_celery_metrics branch from e3a2c0f to c42bbb8 Compare June 23, 2025 12:24

xrmx added this to Python PR digest Jun 23, 2025

xrmx moved this to Ready for review in Python PR digest Jun 23, 2025

xrmx reviewed Jun 30, 2025

View reviewed changes

emdneto reviewed Jun 30, 2025

View reviewed changes

rbagd added 3 commits August 13, 2025 09:34

Additional metrics exported from Celery workers

3c2fade

Fix wrong comment

164e89c

Add timeout for waiting functions

9fcfe0f

rbagd force-pushed the include_more_celery_metrics branch from 0ef3755 to d9f3fee Compare August 13, 2025 07:36

Replace perf_counter by default_timer

1df9e8d

rbagd force-pushed the include_more_celery_metrics branch from d9f3fee to 1df9e8d Compare August 13, 2025 07:39

github-actions bot added the Stale label Mar 13, 2026

github-actions bot closed this Mar 27, 2026

github-project-automation bot moved this from Ready for review to Done in Python PR digest Mar 27, 2026


		### Fixed

		- `opentelemetry-instrumentation-celery` Fix a memory leak where a reference to a task identifier is kept indefinitely

Conversation

rbagd commented May 5, 2025

Description

Type of change

How Has This Been Tested?

Does This PR Require a Core Repo Change?

Checklist:

Uh oh!

rbagd commented May 22, 2025

Uh oh!

bschoenmaeckers commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rbagd commented Jun 23, 2025

Uh oh!

Uh oh!

xrmx Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

rbagd Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

xrmx Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

emdneto Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

rbagd Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

emdneto Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

rbagd Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

rbagd commented Aug 13, 2025

Uh oh!

diogosilva30 commented Feb 26, 2026

Uh oh!

github-actions bot commented Mar 13, 2026

Uh oh!

github-actions bot commented Mar 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bschoenmaeckers commented May 28, 2025 •

edited

Loading