Celery (re)instrumentation by lrafeei · Pull Request #1429 · newrelic/newrelic-python-agent

lrafeei · 2025-07-23T22:12:07Z

This PR picks new instrumentation points for Celery support to allow for Custom Task classes.

github-actions · 2025-07-23T22:13:38Z

🦙 MegaLinter status: ✅ SUCCESS

Descriptor	Linter	Files	Fixed	Elapsed time
✅ ACTION	actionlint	6		0.71s
✅ MARKDOWN	markdownlint	7	0	1.23s
✅ MARKDOWN	markdown-link-check	7		17.59s
✅ PYTHON	ruff	922	0	0.89s
✅ PYTHON	ruff-format	922	0	0.32s
✅ YAML	prettier	13	0	1.36s
✅ YAML	v8r	13		6.37s
✅ YAML	yamllint	13		0.61s

See detailed report in MegaLinter reports

codecov-commenter · 2025-07-23T22:22:54Z

Codecov Report

❌ Patch coverage is 60.60606% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.34%. Comparing base (02d0071) to head (ef587d7).
⚠️ Report is 263 commits behind head on main.

Files with missing lines	Patch %	Lines
newrelic/hooks/application_celery.py	60.00%	16 Missing and 10 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1429      +/-   ##
==========================================
- Coverage   81.37%   81.34%   -0.04%     
==========================================
  Files         208      208              
  Lines       23566    23585      +19     
  Branches     3717     3719       +2     
==========================================
+ Hits        19178    19185       +7     
- Misses       3132     3142      +10     
- Partials     1256     1258       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

TimPansino · 2025-08-01T19:16:29Z

+class CustomCeleryTaskWithSuper(Task):
+    def __call__(self, *args, **kwargs):
+        transaction = current_transaction()
+        if transaction:
+            transaction.add_custom_attribute("custom_task_attribute", "Called with super")
+        return super().__call__(*args, **kwargs)
+
+
+class CustomCeleryTaskWithRun(Task):
+    def __call__(self, *args, **kwargs):
+        transaction = current_transaction()
+        if transaction:
+            transaction.add_custom_attribute("custom_task_attribute", "Called with run")
+        return self.run(*args, **kwargs)
+
+


Nitpick, can we move these (and the tasks that use them) to the bottom of the file so the test app's functionality escalates in complexity rather than starting with the most confusing things?

TimPansino · 2025-08-01T19:27:22Z

+        task = bound_args.get("task", None)
+
+        task = TaskWrapper(task, wrap_task_call)
+        task.__module__ = wrapped.__module__  # Ensure module is set for monkeypatching detection


I'm not sure I understand what this is doing, and the comment is insufficient.

From inspecting execution, task.__module__ = 'celery.app.builtins' and wrapped.__module__ = 'celery.app.trace'.

"Ensure the module is set" doesn't really clarify what it's being set to, and "for monkeypatching detection" is confusing. What is doing the detecting and why is that important?

I disabled this line and the tests pass 100%, so if there's some reason to be doing this we need a test for it.

Defining the TaskWrapper where the run() method just calls the existing __call__ method covers the case that the overriding of the __module__ attribute covers. I believe this was a way to optimize functionality.

Using the scenario where the __call__ method has not been overridden and the run() method has been defined:
Without overriding the __module__ attribute, the task tracer will simply use the run() method and bypass the __call__ function. Then, the TaskWrapper class will reroute that back to the __call__ method. When overriding the __module__ attribute, the task tracer will see that the __call__ method has been overridden (which it does by traversing the MRO), and use that.

That being said, whoops. That is supposed to be task.__module__ = wrap_task_call.__module__ (so wrapper function instead of wrapped function)

I figured it would be faster execution to keep that line in there, but I don't have strong feelings about removing it either.

TimPansino · 2025-08-01T19:29:04Z

-    _process_module_definition("celery.task.base", "newrelic.hooks.application_celery", "instrument_celery_app_task")
-    _process_module_definition("celery.app.task", "newrelic.hooks.application_celery", "instrument_celery_app_task")


I presume with removing these hooks we don't support some older versions of Celery outside our support window, but let's be sure to call that out in the release notes as I'm sure some customers out there use ancient versions of Celery.

TimPansino · 2025-08-01T19:32:52Z

-    result = result.get()
-    assert result == 3
+@pytest.mark.parametrize("dt_enabled", [True, False])
+def test_celery_task_distributed_tracing_inside_background_task(dt_enabled):


The names of all these tests in this file have gotten so long they don't actually describe what they do. Can we shorten all of these and clarify what we're actually testing? Something like this maybe to trim out as many words as possible.

Suggested change

def test_celery_task_distributed_tracing_inside_background_task(dt_enabled):

def test_distributed_tracing_inside_txn_apply_async(dt_enabled):

TimPansino · 2025-08-01T19:39:43Z

+# In this case, the background task creating the transaction
+# has not generated a distributed trace header, so the Celery
+# task will not have a distributed trace header to accept.


This comment and the test don't make sense to me.

This test is for Celery's .apply() method which is immediate and local execution of the task no?

So the reason there's no distributed tracing headers to accept is because this isn't a distributed system, and Celery isn't handling the transaction?

We should validate that no DT headers are accepted for this then with forgone metrics by setting it to None like the above tests do when DT is disabled.

If all of that is true, why is the test after this seeing DT headers and accepting them? Isn't that also local exection? Where is it even getting these headers from to accept?

I've added a separate test for this, where a task is called within a transaction (background task) and insert_distributed_trace_headers()

Basically, delay/apply_async create worker processes and apply/__call__ use the main process to execute. In the case of a task being called in a function that is wrapped with the background_task decorator, any tasks called with delay/apply_async will create a worker process to execute the task and will no longer be in the context of the transaction. Conversely, any tasks called with apply/__call__ will execute in the main process and will be a function trace of the existing transaction.

TimPansino · 2025-08-01T19:41:08Z

+    )
+    @validate_transaction_count(1)  # In the same process, so only one transaction
+    def _test():
+        result = add.apply((1, 2))


Additional thought, can't you call tasks directly like functions? If that's not different execution-wise from calling .apply(), shouldn't we test that here as well the same as we test .apply()?

Good point, I meant to add a direct call test as well. The execution path is different from .apply() even though they both are executed on the main process.

lrafeei added 4 commits July 17, 2025 18:21

Add support for custom task classes

1c0166f

Initial commit

926ee94

Distributed Tracing tests

faabca7

Fix remaining tests and add custom task tests

b7fa0ec

lrafeei and others added 2 commits July 23, 2025 15:31

Merge branch 'main' into celery-custom-task-instrumentation

a4d8232

Clean up comments in file

27cb3a9

lrafeei marked this pull request as ready for review July 23, 2025 22:42

lrafeei requested a review from a team as a code owner July 23, 2025 22:42

mergify Bot added 5 commits July 23, 2025 23:42

Merge branch 'main' into celery-custom-task-instrumentation

ad58275

Merge branch 'main' into celery-custom-task-instrumentation

e7113a3

Merge branch 'main' into celery-custom-task-instrumentation

5c6bda0

Merge branch 'main' into celery-custom-task-instrumentation

0a58054

Merge branch 'main' into celery-custom-task-instrumentation

041cccb

TimPansino self-assigned this Jul 28, 2025

Merge branch 'main' into celery-custom-task-instrumentation

1c7df3c

mergify Bot added tests-failing Tests failing in CI. and removed tests-failing Tests failing in CI. labels Jul 30, 2025

Ruff format

e296bd0

This was linked to issues Jul 31, 2025

Celery tracing has stopped reporting to new relic console after version 9.9 #1219

Closed

Unable to add custom attribute to Celery trace #1243

Closed

Fix comment typo

b792b55

TimPansino requested changes Aug 1, 2025

View reviewed changes

Merge branch 'main' into celery-custom-task-instrumentation

1052d28

mergify Bot added the tests-failing Tests failing in CI. label Aug 6, 2025

mergify Bot added 2 commits August 6, 2025 19:05

Merge branch 'main' into celery-custom-task-instrumentation

9de2530

Merge branch 'main' into celery-custom-task-instrumentation

c77a0a2

mergify Bot removed the tests-failing Tests failing in CI. label Aug 6, 2025

Reviewer suggestions and more tests

088fdb1

Clean up commented code

11cbdbf

lrafeei requested a review from TimPansino August 8, 2025 16:38

lrafeei added 4 commits August 8, 2025 11:20

Add comments and shuffle code

7f2493c

Fix celery linter errors

d908ba6

Fix azure linter errors

17954ab

Fix linter error

ba90b90

mergify Bot added tests-failing Tests failing in CI. and removed tests-failing Tests failing in CI. labels Aug 8, 2025

mergify Bot added 3 commits August 8, 2025 23:14

Merge branch 'main' into celery-custom-task-instrumentation

7634008

Merge branch 'main' into celery-custom-task-instrumentation

fbd50a0

Merge branch 'main' into celery-custom-task-instrumentation

e3b9bef

mergify Bot added the tests-failing Tests failing in CI. label Aug 13, 2025

TimPansino previously approved these changes Aug 13, 2025

View reviewed changes

mergify Bot added 2 commits August 13, 2025 20:09

Merge branch 'main' into celery-custom-task-instrumentation

01b56fb

Merge branch 'main' into celery-custom-task-instrumentation

b2b19b3

mergify Bot added tests-failing Tests failing in CI. and removed tests-failing Tests failing in CI. labels Aug 13, 2025

lrafeei enabled auto-merge (squash) August 18, 2025 15:50

lrafeei disabled auto-merge August 18, 2025 15:51

lrafeei enabled auto-merge (squash) August 18, 2025 15:58

MegaLinter Format

ef587d7

TimPansino dismissed their stale review via ef587d7 August 18, 2025 16:56

TimPansino approved these changes Aug 18, 2025

View reviewed changes

lrafeei merged commit 3cfce55 into main Aug 18, 2025
60 checks passed

lrafeei deleted the celery-custom-task-instrumentation branch August 18, 2025 17:22

umaannamalai added this to the v10.17.0 milestone Sep 4, 2025

		_process_module_definition("celery.task.base", "newrelic.hooks.application_celery", "instrument_celery_app_task")
		_process_module_definition("celery.app.task", "newrelic.hooks.application_celery", "instrument_celery_app_task")

	def test_celery_task_distributed_tracing_inside_background_task(dt_enabled):
	def test_distributed_tracing_inside_txn_apply_async(dt_enabled):

Uh oh!

Conversation

lrafeei commented Jul 23, 2025

Uh oh!

github-actions Bot commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦙 MegaLinter status: ✅ SUCCESS

Uh oh!

codecov-commenter commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lrafeei Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lrafeei Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented Jul 23, 2025 •

edited

Loading

codecov-commenter commented Jul 23, 2025 •

edited

Loading

lrafeei Aug 7, 2025 •

edited

Loading

lrafeei Aug 7, 2025 •

edited

Loading