Skip to content

fix: tell a stopped run apart from one the service killed - #14407

Open
ogabrielluiz wants to merge 1 commit into
feat/flow-span-protocolfrom
fix/cancelled-vs-aborted-span
Open

fix: tell a stopped run apart from one the service killed#14407
ogabrielluiz wants to merge 1 commit into
feat/flow-span-protocolfrom
fix/cancelled-vs-aborted-span

Conversation

@ogabrielluiz

Copy link
Copy Markdown
Contributor

Stacked on #14403 and merges after it. The diff below is against that branch, not release-1.12.0, and green CI here only proves this change against its parent.

What

asyncio delivers two very different events as the same CancelledError, and the flow span reported both the same way:

  • a user pressing stop
  • a server-imposed execution ceiling (asyncio.wait_for around the v2 build driver, a2a, or the agentic assistant), a worker shutdown, a parent task being torn down

Both came out as status="cancelled" with OTel span status UNSET. So every timeout stayed out of error-rate alerting, while the client was served a terminal error and the job row was written FAILED. The span was the only place in the system claiming nothing had gone wrong.

They are separate outcomes now:

Ending status Span status
User pressed stop cancelled UNSET
Ceiling, shutdown, teardown aborted ERROR, error.type=CancelledError

A withdrawn request is genuinely not a service fault, so the stop button keeps its UNSET. Everything else now agrees with what the client and the job row already say.

One constant instead of five strings

The discriminator already existed: the producers stamp a marker on the exception args and the job service reads it to choose CANCELLED over FAILED. It was a bare "LANGFLOW_USER_CANCELLED" repeated at five sites, which is why the span could not have trusted it. It is one constant now, imported by both the producers and the span.

It lives in lfx.constants, not next to the graph exceptions where it more naturally belongs. Importing lfx.graph.exceptions from a langflow service module runs lfx/graph/__init__.py, which imports Graph from a half-initialized base.py, and from langflow.main import create_app then fails outright. lfx.constants imports nothing but pathlib. I hit that while writing this and would rather leave the reason here than have someone move it back.

Verification

New test drives a real asyncio.wait_for timeout rather than hand-raising CancelledError, so it exercises the way the ceiling actually cancels a run. The user-stop test raises with the marker the producers stamp.

Collapsing the split back to one value turns the timeout test red with 'cancelled' == 'aborted'.

Suites: 12 flow-span tests, 1763 in services/, 145 v2 background/cancellation tests, plus an explicit from langflow.main import create_app check for the import cycle above.

Known gaps

aborted still covers a few distinguishable things (ceiling, shutdown, teardown). They all mean "the service did not deliver this run", which is the distinction that matters for alerting, so I did not split further.

@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)
  • release-.*

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: c4b234c9-f19a-4adb-8914-9d1797b0a7d0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

✅ Test Coverage Advisor

No source changes detected without accompanying tests. Thanks for keeping coverage up! 🎉

Advisory check only — never blocks merge.

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Migration Validation Passed

All migrations follow the Expand-Contract pattern correctly.

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Frontend Unit Test Coverage Report

Coverage Summary

Lines Statements Branches Functions
Coverage: 49%
49.57% (71705/144626) 70.35% (10042/14274) 46.77% (1644/3515)

Unit Test Results

Tests Skipped Failures Errors Time
5529 0 💤 0 ❌ 0 🔥 20m 14s ⏱️

@codecov

codecov Bot commented Aug 4, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 64.70588% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.03%. Comparing base (1d6c5ee) to head (7b9466f).
⚠️ Report is 2 commits behind head on feat/flow-span-protocol.

Files with missing lines Patch % Lines
src/lfx/src/lfx/graph/graph/base.py 28.57% 5 Missing ⚠️
src/backend/base/langflow/services/task/service.py 50.00% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@                     Coverage Diff                     @@
##           feat/flow-span-protocol   #14407      +/-   ##
===========================================================
+ Coverage                    61.50%   62.03%   +0.53%     
===========================================================
  Files                         2406     2416      +10     
  Lines                       240085   242243    +2158     
  Branches                     36189    36190       +1     
===========================================================
+ Hits                        147654   150286    +2632     
+ Misses                       90496    90022     -474     
  Partials                      1935     1935              
Flag Coverage Δ
backend 70.35% <88.88%> (+2.66%) ⬆️
lfx 61.35% <37.50%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...e/langflow/services/background_execution/runner.py 98.49% <100.00%> (+0.51%) ⬆️
...ackend/base/langflow/services/job_queue/service.py 85.09% <100.00%> (+2.93%) ⬆️
src/backend/base/langflow/services/jobs/service.py 87.40% <100.00%> (-1.02%) ⬇️
src/lfx/src/lfx/constants.py 100.00% <100.00%> (ø)
src/backend/base/langflow/services/task/service.py 62.90% <50.00%> (+0.60%) ⬆️
src/lfx/src/lfx/graph/graph/base.py 64.11% <28.57%> (-0.15%) ⬇️

... and 142 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ogabrielluiz
ogabrielluiz force-pushed the feat/flow-span-protocol branch from be57c49 to 6a7b0f9 Compare August 4, 2026 22:26
@ogabrielluiz
ogabrielluiz force-pushed the fix/cancelled-vs-aborted-span branch from 897d15a to e2b1752 Compare August 4, 2026 22:27
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Aug 4, 2026
@ogabrielluiz
ogabrielluiz force-pushed the fix/cancelled-vs-aborted-span branch from e2b1752 to f94bce7 Compare August 5, 2026 15:12
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Aug 5, 2026
@ogabrielluiz
ogabrielluiz force-pushed the feat/flow-span-protocol branch from d330b30 to 1d6c5ee Compare August 5, 2026 15:24
asyncio delivers a user pressing stop and a server-imposed execution ceiling
as the same CancelledError, and the flow span reported both as cancelled with
OTel status UNSET. That kept every timeout out of error-rate alerting, while
the client was served a terminal error and the job row was written FAILED.
The span was the only place claiming nothing had gone wrong.

The two are now separate outcomes. A user stop stays cancelled with status
UNSET, because a withdrawn request is not a service fault. Anything else that
cancels a run reports aborted with span status ERROR, which is what the rest
of the system already says about it.

The discriminator already existed: the producers stamp a marker on the
exception args, and the job service reads it to choose CANCELLED over FAILED.
It was a bare string repeated at five sites, so the span could not have
trusted it. It is one constant now, and it lives in lfx.constants rather than
next to the graph exceptions: importing lfx.graph.exceptions from a langflow
service module runs lfx/graph/__init__, which imports Graph, and that made
langflow.main fail to import at all. lfx.constants imports nothing but
pathlib.
@ogabrielluiz
ogabrielluiz force-pushed the fix/cancelled-vs-aborted-span branch from f94bce7 to 7b9466f Compare August 5, 2026 15:25
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Aug 5, 2026
@ogabrielluiz
ogabrielluiz requested a review from erichare August 5, 2026 16:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant