test(a2a): promote the A2A server batch to @stable (#1349) - #1351
Merged
Conversation
The A2A batch shipped 7 specs / 17 tests across #1242, #1244 and #1247 and none of them carried @stable, so daily-stable.yml — which filters on --grep "@stable" — never ran a single one. Ten of the eighteen §16 checklist bullets were automated and watching nothing: an A2A regression between two nightlies was invisible to every scheduled lane, and the specs only ran when a PR happened to touch them or someone dispatched manual.yml. The reason the docs gave for holding the tag back does not survive contact with CONTRIBUTING.md, which states the standard plainly: every new test enters with @stable, in the PR itself, with the reviewer confirming the five validation steps on approval. The four documented exceptions are inherited-not-yet-reviewed, temporarily-removed-while-failing, utility specs and @destructive; this batch is none of them. "No daily history" was also circular — without the tag these specs can never enter the daily, so they can never accumulate the history their own docs cited as the precondition for it. The technical blocker is gone too: LANGFLOW_A2A_ENABLED=true has been on every lane since #1240, and the whole batch is LLM-free (a Chat Input -> Chat Output passthrough with a per-run sentinel, no provider, no key, no quota). Evidence: 51/51 green (17 tests x 3, --retries=0 so a flake could not hide) on nightly 1.12.0.dev18 — four nightlies past the dev14 the docs recorded — in 53.4s at the default worker count, with flows 26 -> 26 (nothing leaked) and zero "Backend Error" logged. Re-run after the tag conversion: 17/17. The tags had to move as well, and that is the larger half of the diff. These specs declared their tags inside the test TITLE; the rest of the suite uses an inline `tag:` array, and scripts/lib/stable-tests.ts — the shared parser behind Phase 0 and the checklist guard — counts ONLY a `tag` array containing the literal "@stable". Appending the tag to the title would have put these 17 tests in the daily while leaving them out of every generated count, which is precisely the split #985 exists to prevent. The describe blocks lose their title tags too: all 7 title-tagged describes in the entire suite were A2A's. The parser now reports 465 @stable test() calls, up from 448, with zero warnings. §16.1's nine [-] bullets become [x]. The [~] Try it bullet stays [~] on purpose: the gap it names is a product absence, not a coverage one — agentTab.viewExchange is a dead i18n key (7 locales, zero call sites, absent from the DOM after a completed turn, #1244). Its spec still earns the tag; only the bullet symbol is held by the partial. The eight [ ] bullets are untouched — §16.2 needs the loopback-SSRF question answered first, and the other two are already recorded as not automatable. Per #741, the generated blocks are left alone — update-coverage-summary.yml regenerates them on merge. Closes #1349 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1349.
What this does
Promotes the A2A server batch — 7 specs, 17 tests, shipped across #1242, #1244 and #1247 — to
@stable, and flips the nine[-]bullets inQA-CHECKLIST.md§16.1 to[x].Until now
daily-stable.ymlfilters on--grep "@stable"and none of these tests carried it, so ten automated checklist bullets watched nothing. An A2A regression between two nightlies was invisible to every scheduled lane; the specs ran only when a PR happened to touch them (import graph) or someone dispatchedmanual.yml.Why the tag was withheld, and why that no longer holds
Every spec doc gave the same reason:
CONTRIBUTING.mdsays the opposite: "every new test enters with@stable" — in the PR itself, with the reviewer confirming the five validation steps on approval. Its four exceptions are inherited-not-yet-reviewed, temporarily-removed-while-failing, utility specs and@destructive. This batch is none of them."No daily history" is also circular — without the tag these specs never enter the daily, so they can never accumulate the history their own docs cite as the precondition.
The technical blocker is gone too:
LANGFLOW_A2A_ENABLED=truehas been on every lane and both start scripts since #1240, and the batch is LLM-free (Chat Input → Chat Output passthrough with a per-run sentinel — no provider, no key, no quota).Evidence
Against
langflowai/langflow-nightly:latest=1.12.0.dev18(four nightlies past thedev14the docs recorded),LANGFLOW_A2A_ENABLED=true,LANGFLOW_WORKERS=1, default Playwright worker count.🚨 Backend Error:The tag conversion — the larger half of the diff
These specs declared their tags inside the test title. The rest of the suite uses an inline
tag:array, andscripts/lib/stable-tests.ts— the shared parser behind the Phase 0 block and the checklist guard — counts only atagarray containing the literal"@stable".Appending
@stableto the title would have put these 17 tests in the daily while leaving them out of every generated count: exactly the split #985 exists to prevent. So the tags moved totag:arrays and the titles are now clean, matching the convention (stable-tests.tstreats titles as the source of the Phase 0 bullet text).The
describeblocks lose their title tags for the same reason — all 7 title-tagged describes in the entire suite were A2A's.Parser before → after: 448 → 465
@stabletest()calls, zero warnings.Checklist symbols
[-]→[x]in §16.1.[~]Try it bullet stays[~]. The gap it names is a product absence, not a coverage gap:agentTab.viewExchangeis a dead i18n key — 7 locales, zero call sites, absent from the DOM after a completed turn (Create: A2A Agent tab — publish flow and the Try it panel (§16.1 UI) #1244). The spec still earns the tag; only the bullet symbol is held by the partial.[ ]bullets are untouched. §16.2 (A2A Client) needs the loopback-SSRF question answered first (follow-up 4 of the scoping doc); non-owner-publish403and the disabled-server state are already recorded as not automatable.Per #741, the generated blocks are not regenerated here —
update-coverage-summary.ymldoes that on merge.scripts/check-checklist-guard.mjsconfirms no generated block was touched.Docs
All 7 spec docs: the "No
@stableyet" bullet is replaced with the tag plus the measured justification, andLast validatedbumped1.12.0.dev14→1.12.0.dev18.a2a-coverage-scope.mdis deliberately left atdev14— itsLast validatedrefers to the surface map measurement, which this PR did not redo.Gates run locally
npm run typechecknpm run lintnpm run test:unitsnpm run test:scriptsnpm run check:checklist-coverage@stablespec referenced by a Part II bulletnode scripts/check-checklist-guard.mjsnpx playwright test --grep "@a2a"Risk to watch after merge
The 4 UI tests (
agent-tab-*) are the only plausible flake surface: the Agent-tab save has no DOM completion signal (agent-statusflips optimistically), so both specs wait on thePATCH /api/v1/flows/{id}the button fires, andagent-card-nameis mount-time state that a save does not refresh (asserted after a reload). Both were measured in #1244 and are handled; 3/3 green here. Worth watching in the first week of daily history.🤖 Generated with Claude Code