fix(llm-agents): delete the flows three agent specs left behind (#1346)

[rafaelgiln]

Closes #1346.

What the issue asked

Name which of the four candidate paths puts a run's tokens in the platform's unattributed bucket. On run 31093877484 (2026-08-06) that bucket was 4,120 tokens / 4 calls / $0.0052 — 8.0% of the run.

The answer: path 4 — the flow is never deleted at all

Reconstructed trace by trace from the run's own artifacts (token-probes-*.jsonl × token-attrib-*.jsonl × results.json):

trace	shard	flow	tokens / calls	owner
1027dfd2	2	abb0557a	936 / 1	agent-empty-refusal-response.spec.ts — model refusal…
6676e05d	2	c52cca3e	918 / 1	agent-empty-refusal-response.spec.ts — empty response…
e7c60610	3	b5a8c06a	2266 / 2	agent-max-iterations.spec.ts — causal control… (@stable)
3 traces	1	—	0 / 0	window of the failing openai-provider + traces-list-filters

936 + 918 + 2266 = 4,120; 1 + 1 + 2 = 4 calls. The three zero-token traces cost nothing.

Both specs call SimpleAgentTemplatePage.load() and never delete the flow it returns. Token attribution lives on the delete path (#1197), so a flow that is never deleted is never attributed.

What made this hard to read: both specs do emit an attrib_cost record per test. That record comes from loadTemplateByName's own cleanup of the surplus flows it creates — those never ran, carried no traces, and the attribution read came back empty.

The other three paths are refuted by the same data, not by argument:

• Path 1 (deleted through the UI) — none of the six flow ids appears in any token-attrib-*.jsonl, so no delete ran for them at all.
• Path 2 (no test.info() to name) — the attrib_cost records name both the test and the file, so resolution worked.
• Path 3 (deleted between two poller ticks) — measured the test end − trace start gap for all 29 attributed token-bearing traces on this run: 2.5 s to 13.1 s. The unattributed ones sit at 1.9 / 2.4 / 4.6 s, inside that range. The gap does not discriminate; the missing delete does.

Contra-example on the same run: agent-multi-tool-selection.spec.ts — same template, same model, has an id-scoped afterEach, attributed 2 of 2 traces.

The change

A repo-wide scan found exactly three specs that load a template and clean up nothing. All three are wired to the shared tracker (#1108): capture the creation POST, delete id-scoped in afterEach, never a delete-all sweep (#553). cleanup() derives the attribution from the running test on its own (#1197 §1.1) and stays inert unless the lane sets TOKENS_ATTRIB, so local runs and the PR lane are unaffected.

• agent-max-iterations.spec.ts
• agent-empty-refusal-response.spec.ts
• agent-model-connection-isolation.spec.ts — never executes its flow, so it produced no unattributed tokens; the leak was its whole cost, and the same hook closes both.

The same stale claim is corrected in all three specs and their docs: SimpleAgentTemplatePage.load() does not wipe existing flows — the cross-worker delete-all was removed in #553 — which is why the missing cleanup read as intentional.

Verification — local A/B

start-langflow-docker.sh sets LANGFLOW_DEACTIVATE_TRACING=true by design, so a second nightly container was started with tracing on to measure the attribution half.

	flows before → after	TRACE lines in TOKENS_ATTRIB	result
without the fix (origin/main)	26 → 29 (3 leaked)	0	3 passed
with the fix	26 → 26	3	3 passed

The three lines, attributed to the right spec and test, reproducing the daily almost token for token:

TRACE tok=936  claude-haiku-4-5 calls=1 | agent-empty-refusal-response | model refusal…   (daily: 936)
TRACE tok=917  claude-haiku-4-5 calls=1 | agent-empty-refusal-response | empty response…  (daily: 918)
TRACE tok=2251 claude-haiku-4-5 calls=2 | agent-max-iterations        | causal control…   (daily: 2266, 2 calls)

npm run typecheck clean, npm run lint 0 errors, both QA-CHECKLIST guards pass.

Reproduce:

docker run -d --name lf-trace -p 7861:7860 \
  -e LANGFLOW_AUTO_LOGIN=true -e LANGFLOW_DEACTIVATE_TRACING=false \
  -e LANGFLOW_ALLOW_CUSTOM_COMPONENTS=true -e LANGFLOW_A2A_ENABLED=true -e LANGFLOW_WORKERS=1 \
  langflowai/langflow-nightly:latest

PLAYWRIGHT_BASE_URL=http://localhost:7861 MODEL_TEST_PROVIDER=anthropic MODEL_TEST_ID=claude-haiku-4-5 \
TOKENS_ATTRIB=/tmp/attrib.jsonl npx playwright test \
  tests/tests-automations/regression/core-functionality/llm-agents/agent-max-iterations.spec.ts \
  tests/tests-automations/regression/core-functionality/llm-agents/agent-empty-refusal-response.spec.ts \
  tests/tests-automations/regression/core-functionality/llm-agents/agent-model-connection-isolation.spec.ts \
  --workers=1 --retries=0 --grep "@stable"

grep -c '"trace_id"' /tmp/attrib.jsonl    # 3 with the fix, 0 without

Expected: 4 passed, 3 TRACE lines, flow count unchanged.

The issue's own cheap check still applies as the final proof: the unattributed row on the platform's By spec table should fall to the zero-token traces on the next daily.

Not in scope, observed while verifying

agent-max-iterations › "agent stops when max iterations is reached" fails on claude-haiku-4-5 (and on claude-opus-5 / claude-opus-4-5 in a multi-model sweep): the agent replies "I'll fetch that URL for you." with no limit message. It fails identically on origin/main, so it is not this diff. Already tracked as #1264 — this run adds that it reproduces locally, off CI load, across several Anthropic models. That test is @regression, never @stable, which is why the daily never surfaces it.

🤖 Generated with Claude Code