Skip to content

fix(mcp): harden mcp-client-agent against pre-1.10.x Playground prefill race - #228

Merged
daniellicnerski1 merged 2 commits into
mainfrom
fix/mcp-client-agent-playground-prefill-226
May 18, 2026
Merged

fix(mcp): harden mcp-client-agent against pre-1.10.x Playground prefill race#228
daniellicnerski1 merged 2 commits into
mainfrom
fix/mcp-client-agent-playground-prefill-226

Conversation

@daniellicnerski1

Copy link
Copy Markdown
Collaborator

Summary

  • Closes test(mcp): mcp-client-agent echo test sends template default prompt on Langflow < 1.10.x #226. The agent calls echo MCP tool and returns echoed message test in tests/tests-automations/regression/mcp/client/mcp-client-agent.spec.ts failed deterministically on Langflow 1.9.2 despite the underlying MCP echo integration working when reproduced manually.
  • Real root cause (the issue's hypothesis was partially off — the prefill does exist on 1.9.2): in flow-page-sliding-container.tsx the prefill useEffect has inputs and nodes in its dep array. Their array references change across renders, so the effect re-fires and resets chatValue back to the Chat Input node's template input_value ("Hello, how are you?") between Playwright's .fill() and the send-button click — send() then dispatches the stale value while the textarea still displays our prompt. Issue's Option 1 + Option 2 alone do not close the race (verified empirically).
  • Hardening applied: keep .clear() + .fill() + toHaveValue() (issue acceptance contract), then run the actual send as an atomic DOM operation inside a single page.evaluate — set the textarea value, dispatch the synthetic input event so React's controlled component updates, and call .click() on the send button in the same synchronous tick. React useEffects only run after the current task completes, so the prefill cannot race the click. Added an exact-match assertion on the user message that landed in chat to catch any residual race loudly at the input step.

Test plan

  • npm run typecheck — 0 errors
  • npm run lint on the touched file — 0 errors, 0 new warnings (5 pre-existing)
  • 5 consecutive npx playwright test ... --workers=1 --retries=0 runs PASS on local Langflow 1.9.2 with MODEL_TEST_ID=gpt-4o-mini (~18–22s each, one build-events stream per run = no retried agent calls, deterministic)
  • Force-fail validation: broke toHaveValue assertion → failed at the input step with a clear diagnostic showing the textarea reverting to "Hello, how are you?" mid-typing; reverted → PASS
  • --trace=on run produced a coherent trace matching the declared test.step() blocks
  • Zero 🚨 Backend Error occurrences in the test output
  • Re-validate against langflowai/langflow-nightly:latest before merge (issue acceptance criterion)

Spec doc docs/mcp/client/mcp-client-agent.md intentionally not touched — the test logic (send the echo prompt, assert the echoed response) is unchanged, and Last validated is not bumped because end-to-end re-validation against nightly is still pending.

🤖 Generated with Claude Code

Daniel Licnerski Borges and others added 2 commits May 14, 2026 16:06
…ll race

The `agent calls echo MCP tool and returns echoed message` test failed
deterministically on Langflow 1.9.2 even though the underlying MCP echo
integration worked when reproduced manually. Investigation showed the
prefill `useEffect` in `flow-page-sliding-container.tsx` has `inputs` and
`nodes` in its dep array; their array references change across renders,
so the effect re-fires and resets `chatValue` back to the Chat Input
node's template `input_value` ("Hello, how are you?") between Playwright's
`.fill()` and the send button click. The send then dispatches the stale
value while the textarea still displays our prompt — the agent never
receives the echo instruction.

Hardening:
- Keep `.clear()` + `.fill()` + `toHaveValue()` (issue #226 acceptance
  criteria) for the surface contract on 1.10.x.
- Run the actual send as an atomic DOM operation inside a single
  `page.evaluate`: set the textarea value, dispatch a synthetic `input`
  event so React's controlled component picks up the change, and call
  `.click()` on the send button in the same synchronous tick. React
  useEffects only run after the current task completes, so the prefill
  cannot race the click.
- Assert the user message that reached the chat history equals the echo
  prompt before waiting on the agent response, so a residual race fails
  loudly at the input step instead of silently at the final assertion.

Verified green on local Langflow 1.9.2 across 5 consecutive `--retries=0`
runs (~18–22s each, one build stream per run = no retried agent calls).
Resolves #226.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…atch)

The previous validation only checked the agent's response text for "hello
mcp", which passed coincidentally when the LLM echoed the prompt back from
memory without invoking any tool. Strengthen the assertion chain to prove
the tool was actually called:

Proof #1 — Playground renders a "Called tool" indicator (ContentBlockDisplay
  returns null when no tool was invoked, so this row only exists after a
  tool call).
Proof #2 — the trigger reads "Called tool ECHO" (formatToolTitle uppercases
  the tool name); case-insensitive match tolerates rawTitle variants.
Proof #3 — the echoed payload "hello mcp" appears in the final response
  (kept from the previous validation, now last in the chain).

Notes for future maintainers, documented inline:
- header-icon is OUTSIDE div-chat-message — they are siblings under the
  untestid'd "flex w-[94%]" wrapper in chat-message.tsx.
- Langflow's AccordionTrigger (src/components/ui/accordion.tsx) uses asChild
  and wraps a <div>, NOT a <button>; selector must be tag-agnostic.
- chat-message.tsx defaults hideHeader=false (accordion collapsed); a
  best-effort DOM click on the row's cursor-pointer chevron expands it.

Quote the tool name in single quotes in the prompt — empirically the most
reliable phrasing for gpt-4o-mini.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@Victor-w-Madeira Victor-w-Madeira left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM on the code side — validation passed cleanly against project conventions.

Validation summary

  • --trace=on run: 2 passed (1.1m) — openai/gpt-5.4 (36.1s) + google/gemini-2.5-flash (26.4s), no retries, no 🚨 Backend Error
  • ESLint: 0 errors / 5 pre-existing warnings (none introduced here)
  • tsc --noEmit: clean
  • @stable present; spec imports from fixtures/fixtures; uses SimpleAgentTemplatePage + models.json per mcp/CLAUDE.md
  • New Proof #1 (Called tool) and Proof #2 (ECHO) assertions remove the "presence-only" false-positive risk that was flagged in CONTRIBUTING.md

Blocker before merge — docs/mcp/client/mcp-client-agent.md is out of sync with the code

  1. Step 7 (line 27) still shows the old prompt "Use the echo tool to echo: hello mcp". The code now sends "Use the 'echo' tool to echo: hello mcp" — the single quotes around echo are a deliberate choice documented in the inline comment (lower refusal rate on gpt-4o-mini). The doc must reflect the actual prompt the test sends.

  2. Step 7 does not describe the atomic page.evaluate send, which is the actual fix for #226 (set the textarea value, dispatch the synthetic input event, and click send in one synchronous tick so the prefill useEffect cannot race the dispatch). The doc still describes the naive fillclick flow that this PR is replacing — so a reader of the spec doc cannot tell what the PR actually changed.

  3. Validation criterion (line 35) lists only "Agent response contains hello mcp". The test now asserts three independent proofs:

    • Playground renders a Called tool indicator (agent invoked some tool)
    • The tool invoked is echo (case-insensitive match on the ECHO formatted title)
    • The echoed payload hello mcp appears in the final agent message

    Without these in the doc, the rationale of commit 7e78416 ("assert echo MCP tool was actually invoked (not just text match)") is invisible to readers of the spec.

Please update the spec doc to reflect the new prompt, the atomic-send technique, and the three-proof validation criterion before merging.

@daniellicnerski1
daniellicnerski1 merged commit 8e0432b into main May 18, 2026
2 checks passed
@Victor-w-Madeira
Victor-w-Madeira deleted the fix/mcp-client-agent-playground-prefill-226 branch May 18, 2026 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(mcp): mcp-client-agent echo test sends template default prompt on Langflow < 1.10.x

2 participants