fix(llm-agents): bound the multi-tool sequence run and report its real failure (#1378)

[Victor-w-Madeira]

Closes #1378.

Problem

Test 3 of agent-multi-tool-selection.spec.ts — agent runs the URL then Web Search tools in sequence — failed 3/3 on the 2026-08-08 PR run (31244373991) with:

expect(getByTestId('div-chat-message').last()).toBeVisible() failed
Error: element(s) not found

That message named the wrong thing. The bubble was absent because the run never produced a reply: the agent blew past the model's context window and the provider rejected the request. The spec was selected transitively on an unrelated PR, so the red looked like someone else's problem.

Root cause — upstream and unbounded

WebSearchComponent.perform_web_search() iterates every div.result DuckDuckGo returns, fetches each linked page and extracts its entire text. No result cap, no content truncation:

for result in soup.select("div.result"):          # no limit
    page = self._safe_get_url(final_url, headers=headers)
    content = BeautifulSoup(page.text, "lxml").get_text(separator=" ", strip=True)

Measured in the container on 1.12.0.dev20, query "Sample Slide Show": 10 results, 182,316 chars ≈ 45.6k tokens in ONE call (largest single page: 40,755 chars). git log -S"max_results" across all refs: never existed — long-standing behavior, not a regression. Filed upstream as langflow-ai/langflow#14469.

Test 3 is the only test in this file whose instruction permits an open-ended sequence, so when the agent does not converge it keeps calling perform_search and the conversation is re-sent every turn. Observed: 206,881 / 206,902 / 271,317 tokens in CI and 5,060,863 locally.

Not a rate-limit tier problem, and that decides the fix. The CI org caps at 200k TPM; the local one at 4M — 20× larger — and the local run blew through it anyway. A bigger tier or a wider model buys nothing: no window on the market holds 5M tokens.

Changes

1. Cap max_iterations at 8 for test 3, and assert the field actually holds that value — a fill that silently no-ops leaves the default 15 in place and re-opens this on a run that still looks green.

2. Drop the reply-bubble assert from test 3. The spec doc never specified one for it — tests 1–2 do specify it (their step 6), test 3 goes from step 4 straight to the sequence assert. It was orphan code, and it was the line that mis-reported every blow-up. A crashed run is still caught by the fixture, which owns that verdict (no allowFlowErrors).

What this does NOT do

It does not make the test deterministic. Measured, --retries=0, 1.12.0.dev20 / gpt-4o-mini:

max_iterations	Pass rate	Failure mode
15 (default)	4/5	context blow-up, up to 5,060,863 tokens
8 (this PR)	5/6	context blow-up, 129,150 tokens — same rate, far smaller blast radius
4	0/2	Recursion limit of 13 reached without hitting a stop condition

The cap buys cost and attribution, not reliability. A single perform_search call is already unbounded — measured the same day across three queries: 15,857 / 53,714 / 78,848 tokens for one call, a 5× spread, with the query chosen by the agent. Two calls at the top of that range exceed 128k on their own, so no iteration cap can guarantee this test. Real stabilisation needs #14469.

Test 3 therefore stays out of @stable and its checklist bullet stays [-].

Rejected — the arithmetic that produced 4. An earlier version derived the cap from a token budget (three calls needed, plus headroom, worst case inside 128k) and got 4. It is wrong in a way worth recording: max_iterations=4 sets a LangGraph recursion_limit of 13, the agent hits it, and a run that stops that way persists no AI message at all — so the sequence assert fails on absent data rather than wrong data, and the cap meant to fix the test breaks it a second way. The spec doc records this so it is not re-derived.

Covered tests

#	Test	What it validates
3	agent runs the URL then Web Search tools in sequence for a chained prompt	The run's ordered tool_use list contains fetch_content before perform_search. Cap applied and verified on the field before the run

Tests 1–2 are untouched in logic; the file gained one import and one helper used only by test 3.

Validation (nightly 1.12.0.dev20, --workers=1 --retries=0, MODEL_TEST_ID=gpt-4o-mini)

• npm run typecheck ✅ · npm run lint ✅ 0 errors · npm run test:units ✅ 552/552 · npm run test:scripts ✅ 797/797 · QA-CHECKLIST guard ✅ · coverage guard ✅ · spec-dep guard ✅ (exit 0)
• Test 3: 5 passed / 1 failed of 6 (28.1s–1.3m). The one failure is the documented residual.
• Test 2 isolated: ✅ 1 passed (14.9s)
• Orphan check: 26 flows / 1 project before and after — 0 delta (26 = the starter projects)

Force-fail — executed

Mutation	Result
M5 — point the cap at a non-existent field id	✅ failed: TimeoutError: locator.click … inspector-add-max_iterations_FF_MUTATION. Proves the cap actually applies
M4 — invert the sequence assert (perform_search before fetch_content)	✅ failed for the right reason: tools out of order: ["fetch_content","perform_search"] … expected ["perform_search","fetch_content"]. Needed 2 attempts — the first failed on a blow-up, not the mutation. A third attempt showed ["fetch_content","perform_search","perform_search"], the double search that drives the problem

Revert proven: grep FF_MUTATION → 0 hits, file byte-identical to the pre-FF copy, gates re-run green.

Gap — stated, not papered over

M1/M3 on the two @stable siblings were not executable locally. httpbin.org returns 503 (the #631 failure mode), test 1 fails on it, and the describe is mode: "serial" so the siblings are skipped. Test 2 passes isolated; test 1 is unvalidated locally. It is covered by this PR's lane, where ECHO_BASE_URL points at the in-network go-httpbin and the 503 does not occur — the PR runs all three tests, including both @stable ones. Closing the gap locally would need the Langflow container restarted with LANGFLOW_SSRF_ALLOWED_HOSTS, discarding its credentials.

CI expectation

The import graph selects exactly this spec (1 spec, 0 dropped, not suite-wide) and needsModels=true, so Collect models runs and all three tests execute. Note the residual: the CI org caps at 200k TPM and retries land inside the same one-minute window, so failures there correlate rather than flake independently — that is why the original run was 3/3 rather than intermittent. The cap cuts consumption by ~40×, which makes it far less likely, but this was validated against a 4M-TPM org and cannot be asserted for CI without running there.

Follow-ups

• Web Search component injects unbounded scraped page content into agent context (no max_results, no content truncation) langflow-ai/langflow#14469 — the upstream cap this test's stability actually depends on.
• Extract the duplicated setMaxIterations() helper shared by agent-max-iterations and agent-multi-tool-selection #1380 — extract the setMaxIterations() helper now duplicated with agent-max-iterations.spec.ts (left duplicated here on purpose: extracting would edit an @stable spec and widen this PR's blast radius).

Dependencies

None beyond the usual provider key. No new external dependency; max_iterations is set through existing testids.

[Copilot]

Pull request overview

This PR addresses a flaky/deterministically-failing agent sequence test (agent-multi-tool-selection test 3) by bounding the agent run’s iteration budget and removing a misleading UI assertion, while documenting the rationale in the spec doc and checklist.

Changes:

• Add a max_iterations cap (8) for the chained multi-tool sequence test and verify the value is actually applied.
• Remove the “reply bubble renders” UI assertion from test 3 so failures are attributed to the underlying run/fixture verdict instead of “element(s) not found”.
• Update the spec doc and QA checklist entry to reflect the new bounded behavior and rationale.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
tests/tests-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts	Adds setMaxIterations() helper + applies an iteration cap for test 3; removes the orphan reply-bubble assertion.
QA-CHECKLIST.md	Updates the Tools & Integrations bullet for the multi-tool sequence test to note the new cap and why it remains [-].
docs/core-functionality/llm-agents/agent-multi-tool-selection.md	Expands test 3 documentation with the cap rationale and updates the step-by-step contract.

Suppressed comments (1)

docs/core-functionality/llm-agents/agent-multi-tool-selection.md:328

• This section contradicts the updated test contract above (Step 4 caps max_iterations at 8). It still claims a cap of 4 / 4 model calls / ~90k tokens, which is inconsistent with the spec (MAX_ITERATIONS_SEQUENCE = "8") and the new doc section.

  tool round-trip for tests 1–2. **Test 3 is not one round-trip** — it runs a
  multi-tool sequence bounded by the `max_iterations` cap of 4, so it costs up
  to 4 model calls and sends up to ~90k tokens (see the context-budget note in
  Step by step). It was unbounded before #1378, at up to 15 calls and millions
  of tokens.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.