Skip to content

fix(llm-agents): bound the multi-tool sequence run and report its real failure (#1378) #1381

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Victor-w-Madeira wants to merge 2 commits into
base: main
Choose a base branch
from
+165 −11
Open

fix(llm-agents): bound the multi-tool sequence run and report its real failure (#1378) #1381

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
86d4b8f
fix(llm-agents): bound the multi-tool sequence run and report its rea…
Victor-w-Madeira Aug 8, 2026
2811852
docs(llm-agents): fix claims that contradict the merge and the measur…
Victor-w-Madeira Aug 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion QA-CHECKLIST.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -370,7 +370,7 @@

#### 6.4 Tools and Integrations
- [ ] Agent with integrated external MCP tool executes action and returns result
- [-] Agent executes multiple tools in sequence → `llm-agents/agent-multi-tool-selection.spec.ts` (Test 3 — chained fetch→search, ordered `tool_use` assert; `@stable` gated on the clean baseline #818, per #827)
- [-] Agent executes multiple tools in sequence → `llm-agents/agent-multi-tool-selection.spec.ts` (Test 3 — chained fetch→search, ordered `tool_use` assert; `@stable` gated on the clean baseline #818, per #827. #1378 bounded the run with a measured `max_iterations` cap after the unbounded Web Search payload blew the context up — still `[-]`: the cap raises the pass rate, it does not make the agent's convergence deterministic)
- [x] Tool returns error — agent handles it and continues execution → `core-functionality/llm-agents/agent-tool-error-handling.spec.ts`
- [x] Multiple connected tools — agent selects the correct one for each prompt → `agent-multi-tool-selection.spec.ts`
- [x] Tool with invalid name — validation prevents execution with clear message → `core-functionality/llm-agents/agent-tool-name-validation.spec.ts`
Expand Down
93 changes: 88 additions & 5 deletions docs/core-functionality/llm-agents/agent-multi-tool-selection.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,15 +166,72 @@ describe with two tests:
3. Seed a task that makes the second tool depend on the first's result:
*"First fetch `${FETCH_URL}` and read its exact slideshow title. Then search
the web for that title and summarize one result. (probe `<nonce>`)"*.
4. Open the Playground, send, wait for the run to finish (Stop button hidden).
5. **Sequence assert (API):** poll `GET /api/v1/monitor/messages` — nonce-keyed
4. **Cap `max_iterations` at 8** on the Agent node (advanced field, exposed via
the inspector — same handles `agent-max-iterations.spec.ts` uses), and assert
the field actually holds that value. This is load-bearing, not tuning: a cap
that silently fails to apply leaves the default 15 and re-opens #1378 on a
run that still looks green. See the note below.
5. Open the Playground, send, wait for the run to finish (Stop button hidden).
6. **Sequence assert (API):** poll `GET /api/v1/monitor/messages` — nonce-keyed
session lookup (same as tests 1–2); collect the **ordered** list of
`tool_use` block names across the session's AI message(s). Assert the list
contains both `fetch_content` and `perform_search`, with
`indexOf(fetch_content) < indexOf(perform_search)` — the agent ran the two
tools one after another in the required order. Only names/order are
asserted (search content is non-deterministic).
6. No `allowFlowErrors`.
7. No `allowFlowErrors`.

> **Why `max_iterations` is capped here and nowhere else (#1378).** Unlike
> tests 1–2, this test's instruction *permits* a multi-tool sequence, and the
> agent does not reliably converge on it. When it doesn't, it keeps calling
> `perform_search`, and each call injects the Web Search component's full
> result set into the conversation. That component caps nothing:
> `perform_web_search()` iterates every `div.result` DuckDuckGo returns and
> scrapes each linked page's **entire** text (upstream
> `langflow-ai/langflow#14469`). Measured on `1.12.0.dev20`, query
> `"Sample Slide Show"`: **10 results, 182,316 chars ≈ 45.6k tokens in one
> call** (largest single page: 40,755 chars). The conversation is re-sent
> every turn, so a non-converging run grows without bound and the provider
> rejects it — the 2026-08-08 PR run reported requests of
> **206,881 / 206,902 / 271,317** tokens and a local run reported
> **5,060,863**, after which the run returns no reply at all.
>
> **This is a volume problem, not a rate-limit tier problem, and the
> distinction decides the fix.** The CI organization's cap is 200k TPM and
> the local one's is 4M TPM — 20× larger — and the local run blew through it
> anyway. A bigger tier, or a model with a wider context window, buys
> nothing: no window on the market holds 5M tokens. Bounding the iteration
> count is what bounds the run.
>
> **What the cap does — and what it does not.** It bounds the worst case; it
> does **not** make this test deterministic. Measured on `1.12.0.dev20` /
> `gpt-4o-mini` with `--retries=0`:
>
> | `max_iterations` | Pass rate | Failure mode |
> |---|---|---|
> | 15 (default) | 4/5 | context blow-up, up to 5,060,863 tokens |
> | **8** (this spec) | **5/6** | context blow-up, 129,150 tokens — same rate within noise, far smaller blast radius |
> | 4 | **0/2** | `Recursion limit of 13 reached without hitting a stop condition` |
>
> An earlier version of this note derived the cap from a token budget (three
> calls needed, plus headroom, worst case inside a 128k window) and arrived
> at **4**. That was wrong in both directions and is recorded so it is not
> repeated. The failure at 4 is not a smaller version of the failure at 15:
> `max_iterations=4` sets a LangGraph `recursion_limit` of 13, the agent hits
> it, and **a run that stops that way persists no AI message at all** — so
> the sequence assert fails on *absent* data rather than wrong data, and the
> cap meant to fix the test breaks it a second way.
>
> **The residual flake cannot be fixed from this repo, and that is the honest
> bottom line.** A *single* `perform_search` call is already unbounded —
> measured the same day across three queries: **15,857 / 53,714 / 78,848**
> tokens for one call, a 5× spread, with the query chosen by the agent and
> not by us. Two calls at the top of that range exceed 128k on their own, so
> **no iteration cap can guarantee this test.** Real stabilisation needs the
> payload bounded upstream (`langflow-ai/langflow#14469`). Until then this
> test stays out of `@stable`, its checklist bullet stays `[-]`, and #1378
> stays open. **Re-measure before changing the number — do not re-derive it
> on paper.**

---

Expand Down Expand Up @@ -222,14 +279,31 @@ tools in the wrong order, fails).
*sequence*. The instruction permits multiple calls but the ORDER is the
agent's, driven by the prompt's data dependency (it cannot search for the
title before fetching it).
- **No live-bubble assert in test 3 (#1378)** — tests 1–2 assert a rendered
`div-chat-message` because their contract includes a completed reply (test 1
additionally pins the deterministic title on the persisted tool output).
Test 3's contract is the ordered `tool_use` list and nothing else, so the
spec doc never specified a bubble assert for it. The code carried one
anyway — never documented here — and it was the line that failed on every
context blow-up, reporting `element(s) not found` instead of the real cause.
It is removed rather than relaxed: the run still executes with no
`allowFlowErrors`, so a crashed run is caught by the fixture (the gate that
owns that verdict), not by a proxy assert on the reply bubble. Note the v2
run path is ADVISORY-only today (#1165) — when it flips to failing, a
context blow-up will fail this test at the fixture, which is the correct
attribution and the reason the cap above is the actual fix.
- **Force-failure checks** (CONTRIBUTING §2): M1 — expect the sibling tool
as first call in test 1 ⇒ selection assert must fail; M2 — assert an
impossible title (e.g. `Sample Slide Show XYZ`) ⇒ the `fetch_content`
tool-output execution assert must fail (verified against go-httpbin: the assert
surfaced the real tool output containing *"title": "Sample Slide Show"* and
failed the impossible pattern); M3 — same first-call swap in test 2 ⇒ must
fail; M4 — invert the sequence assert (require `perform_search` before
`fetch_content`) in test 3 ⇒ must fail against the real ordered tool list.
`fetch_content`) in test 3 ⇒ must fail against the real ordered tool list;
M5 — point the `max_iterations` cap at a non-existent field id in test 3 ⇒
the cap step must fail rather than silently leaving the default 15 in place
(a cap that quietly does not apply is exactly the #1378 failure, and a
passing run would not reveal it).

---

Expand All @@ -247,7 +321,16 @@ tools in the wrong order, fails).
## External dependencies *(required)*

- **LLM provider API** (per `models.json` target): one completion with one
tool round-trip per test.
tool round-trip for tests 1–2. **Test 3 is not one round-trip** — it runs a
multi-tool sequence bounded by the `max_iterations` cap of 4, so it costs up
to 4 model calls and sends up to ~90k tokens (see the context-budget note in
Step by step). It was unbounded before #1378, at up to 15 calls and millions
of tokens.
- **Web Search → DuckDuckGo + every linked page** (tests 2 and 3): the
component scrapes each result's full page text, so this test's token cost is
set by whatever pages DuckDuckGo returns that day. Unbounded upstream
(`langflow-ai/langflow#14469`); the `max_iterations` cap is what keeps the
total finite on our side.
- **URL-tool fetch endpoint** (test 1) — `${ECHO_BASE_URL}/json`, defaulting to
`https://httpbin.org/json` (fixed `Sample Slide Show` payload). httpbin.org is
chronically unreliable (sustained 503s/timeouts hard-failed this test on the
Expand Down
82 changes: 77 additions & 5 deletions ...s-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@
import type { Page } from "@playwright/test";
import type { APIRequestContext } from "@playwright/test";
import { expect, test } from "../../../../fixtures/fixtures";
import {
closeAdvancedOptions,
openAdvancedOptions,
} from "../../../../helpers/ui/open-advanced-options";
import { SimpleAgentTemplatePage, type LoadSimpleAgentOptions } from "../../../../pages";
import { waitForFlowSaveSettled } from "../../../../helpers/flows/wait-for-flow-save-settled";
import { getAuthToken } from "../../../../helpers/auth/get-auth-token";
Expand Down Expand Up @@ -72,6 +76,10 @@
const SYSTEM_PROMPT_SEQUENCE =
"Use the connected tools to complete the task. You may call multiple tools in " +
"sequence as the task requires; never answer from memory and never refuse.";
// Iteration budget for the sequence test only (#1378). Empirical -- see
// setMaxIterations for the measurements, and for why it bounds the blast radius
// without making this test deterministic.
const MAX_ITERATIONS_SEQUENCE = "8";

// Flows created by each test are tracked here and deleted by id in
// afterEach — loadTemplateByName does NO cleanup (post-#553 contract), and
Expand Down Expand Up @@ -135,6 +143,63 @@
await field.blur();
}

// Cap the Agent's max_iterations (test 3 only). This test is the only one whose
// instruction permits an open-ended sequence, and the agent does not reliably
// converge on it: when it doesn't, it keeps calling perform_search, and every
// such call injects the Web Search component's whole result set. That component
// caps nothing -- it scrapes each DuckDuckGo hit's full page text (measured on
// 1.12.0.dev20: 10 results, 182,316 chars ~= 45.6k tokens in ONE call; upstream
// langflow-ai/langflow#14469). Since the conversation is re-sent every turn, a
// non-converging run grows without bound and the provider rejects it: #1378
// recorded requests of 206,881/206,902/271,317 tokens in CI and 5,060,863
// locally, after which the run returns no reply at all.
//
// Not a rate-limit tier problem: the local org allows 4M TPM (20x CI's 200k) and
// blew through it anyway, and no model's context window holds 5M tokens. A
// bigger tier or a wider model buys nothing; bounding the iteration count does.
//
// What this cap DOES and DOES NOT do -- read this before trusting it. It bounds
// the worst case (the 5,060,863-token run above cost ~2 min of wall clock and
// real money); it does NOT make this test deterministic. Measured on
// 1.12.0.dev20 / gpt-4o-mini, --retries=0:
//
// max_iterations pass rate
// 15 (default) 4/5
// 8 (this value) 5/6 <- same rate within noise, smaller blast radius
// 4 0/2
//
// 4 fails for a DIFFERENT reason and that asymmetry is why the cap cannot be
// tightened into a fix: max_iterations=4 sets a LangGraph recursion_limit of 13,
// the agent hits it ("Recursion limit of 13 reached without hitting a stop
// condition"), and a run that stops that way persists no AI message at all -- so
// the sequence assert fails on absent data rather than wrong data.
//
// The residual flake is NOT fixable from here, because ONE search call is
// already unbounded. Measured, same day, three queries: 15,857 / 53,714 / 78,848
// tokens for a single perform_search -- a 5x spread, and the query is the
// agent's choice, not ours. Two calls at the top of that range exceed 128k on
// their own, so no iteration cap can guarantee this test. Real stabilisation
// needs the payload bounded upstream (langflow-ai/langflow#14469); until then
// this test stays out of @stable and #1378 stays open. Re-measure before
// changing this number -- do not re-derive it on paper.
//
// max_iterations is an advanced field: expose it on the node body via the
// inspector, then fill it -- the same handles agent-max-iterations.spec.ts uses.
async function setMaxIterations(page: Page, maxIterations: string): Promise<void> {
await page.locator('[data-testid^="rf__node-Agent"]').first().click();
await openAdvancedOptions(page);
await page.getByTestId("inspector-add-max_iterations").click();
await closeAdvancedOptions(page);
const maxIter = page.getByTestId("int_int_max_iterations");
await expect(maxIter).toBeVisible({ timeout: 15000 });
await maxIter.scrollIntoViewIfNeeded();
await maxIter.fill(maxIterations);
await maxIter.blur();
// The cap is load-bearing, not cosmetic: a fill that silently no-ops leaves
// the default 15 in place and re-opens #1378 on a run that still looks green.
await expect(maxIter).toHaveValue(maxIterations, { timeout: 10000 });
}

// Set the task on the ChatInput node (the Playground prompt pre-fills from it;
// typing into the Playground races an async default re-injection).
async function setChatInputText(page: Page, text: string): Promise<void> {
Expand Down Expand Up @@ -294,7 +359,7 @@
expectedOrder: string[],
): Promise<void> {
const bearer = await getAuthToken(request);
await expect

Check failure on line 362 in tests/tests-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts

View workflow job for this annotation

GitHub Actions / Run impacted E2E specs

[chromium] › tests/tests-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts:499:9 › Agent Multi-Tool Selection [openai / gpt-4o-mini] › agent runs the URL then Web Search tools in sequence for a chained prompt @regression @agents @playground 

  1) [chromium] › tests/tests-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts:499:9 › Agent Multi-Tool Selection [openai / gpt-4o-mini] › agent runs the URL then Web Search tools in sequence for a chained prompt @regression @agentS @playground 

    Retry #2 ───────────────────────────────────────────────────────────────────────────────────────
    Error: expect(received).toBe(expected) // Object.is equality

    Expected: "tool-sequence-in-order"
    Received: "AI message for the session not persisted yet"

    Call Log:
    - Timeout 90000ms exceeded while waiting on the predicate

      360 | ): Promise<void> {
      361 |   const bearer = await getAuthToken(request);
    > 362 |   await expect
          |   ^
      363 |     .poll(
      364 |       async () => {
      365 |         const res = await request.get("/api/v1/monitor/messages", {
        at expectToolSequencePersisted (/home/runner/work/langflow-e2e/langflow-e2e/tests/tests-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts:362:3)
        at /home/runner/work/langflow-e2e/langflow-e2e/tests/tests-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts:543:11
        at /home/runner/work/langflow-e2e/langflow-e2e/tests/tests-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts:542:9

Check failure on line 362 in tests/tests-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts

View workflow job for this annotation

GitHub Actions / Run impacted E2E specs

[chromium] › tests/tests-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts:499:9 › Agent Multi-Tool Selection [openai / gpt-4o-mini] › agent runs the URL then Web Search tools in sequence for a chained prompt @regression @agents @playground 

  1) [chromium] › tests/tests-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts:499:9 › Agent Multi-Tool Selection [openai / gpt-4o-mini] › agent runs the URL then Web Search tools in sequence for a chained prompt @regression @agentS @playground 

    Retry #1 ───────────────────────────────────────────────────────────────────────────────────────
    Error: expect(received).toBe(expected) // Object.is equality

    Expected: "tool-sequence-in-order"
    Received: "AI message for the session not persisted yet"

    Call Log:
    - Timeout 90000ms exceeded while waiting on the predicate

      360 | ): Promise<void> {
      361 |   const bearer = await getAuthToken(request);
    > 362 |   await expect
          |   ^
      363 |     .poll(
      364 |       async () => {
      365 |         const res = await request.get("/api/v1/monitor/messages", {
        at expectToolSequencePersisted (/home/runner/work/langflow-e2e/langflow-e2e/tests/tests-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts:362:3)
        at /home/runner/work/langflow-e2e/langflow-e2e/tests/tests-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts:543:11
        at /home/runner/work/langflow-e2e/langflow-e2e/tests/tests-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts:542:9

Check failure on line 362 in tests/tests-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts

View workflow job for this annotation

GitHub Actions / Run impacted E2E specs

[chromium] › tests/tests-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts:499:9 › Agent Multi-Tool Selection [openai / gpt-4o-mini] › agent runs the URL then Web Search tools in sequence for a chained prompt @regression @agents @playground 

  1) [chromium] › tests/tests-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts:499:9 › Agent Multi-Tool Selection [openai / gpt-4o-mini] › agent runs the URL then Web Search tools in sequence for a chained prompt @regression @agentS @playground 
    Error: expect(received).toBe(expected) // Object.is equality

    Expected: "tool-sequence-in-order"
    Received: "AI message for the session not persisted yet"

    Call Log:
    - Timeout 90000ms exceeded while waiting on the predicate

      360 | ): Promise<void> {
      361 |   const bearer = await getAuthToken(request);
    > 362 |   await expect
          |   ^
      363 |     .poll(
      364 |       async () => {
      365 |         const res = await request.get("/api/v1/monitor/messages", {
        at expectToolSequencePersisted (/home/runner/work/langflow-e2e/langflow-e2e/tests/tests-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts:362:3)
        at /home/runner/work/langflow-e2e/langflow-e2e/tests/tests-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts:543:11
        at /home/runner/work/langflow-e2e/langflow-e2e/tests/tests-automations/regression/core-functionality/llm-agents/agent-multi-tool-selection.spec.ts:542:9
.poll(
async () => {
const res = await request.get("/api/v1/monitor/messages", {
Expand Down Expand Up @@ -454,6 +519,11 @@

await test.step("permit a multi-tool sequence, seed the chained task", async () => {
await setSystemPrompt(page, SYSTEM_PROMPT_SEQUENCE);
// Bound the run BEFORE it starts: this is the only test whose
// instruction permits an open-ended sequence, so it is the only one
// that can accumulate Web Search payloads past the model's context
// window (#1378 — see setMaxIterations).
await setMaxIterations(page, MAX_ITERATIONS_SEQUENCE);
await setChatInputText(page, task);
await waitForFlowSaveSettled(page);
});
Expand All @@ -462,11 +532,13 @@
await openPlaygroundAndSend(page, task);
});

await test.step("execution: a reply bubble renders in the Playground", async () => {
const bubble = page.getByTestId("div-chat-message").last();
await expect(bubble).toBeVisible({ timeout: 30000 });
});

// No reply-bubble assert here, unlike tests 1-2. Their contract includes
// a completed reply; this test's contract is the ordered tool_use list
// and the spec doc never specified a bubble assert for it. The code
// carried one anyway and it was the line that failed on every context
// blow-up, reporting "element(s) not found" instead of the real cause.
// A crashed run is still caught -- by the fixture, which owns that
// verdict (no allowFlowErrors above), not by a proxy on the bubble.
await test.step("sequence: fetch_content is called before perform_search", async () => {
await expectToolSequencePersisted(request, nonce, [URL_TOOL, SEARCH_TOOL]);
});
Expand Down
Loading