Skip to content

test(a2a): promote the A2A server batch to @stable (#1349) - #1351

Merged
rafaelgiln merged 1 commit into
mainfrom
test/issue-1349-a2a-stable
Aug 7, 2026
Merged

test(a2a): promote the A2A server batch to @stable (#1349)#1351
rafaelgiln merged 1 commit into
mainfrom
test/issue-1349-a2a-stable

Conversation

@rafaelgiln

Copy link
Copy Markdown
Collaborator

Closes #1349.

What this does

Promotes the A2A server batch — 7 specs, 17 tests, shipped across #1242, #1244 and #1247 — to @stable, and flips the nine [-] bullets in QA-CHECKLIST.md §16.1 to [x].

Until now daily-stable.yml filters on --grep "@stable" and none of these tests carried it, so ten automated checklist bullets watched nothing. An A2A regression between two nightlies was invisible to every scheduled lane; the specs ran only when a PR happened to touch them (import graph) or someone dispatched manual.yml.

Why the tag was withheld, and why that no longer holds

Every spec doc gave the same reason:

No @stable yet: granted only after team validation (CONTRIBUTING.md). New in #1244, no daily history.

CONTRIBUTING.md says the opposite: "every new test enters with @stable" — in the PR itself, with the reviewer confirming the five validation steps on approval. Its four exceptions are inherited-not-yet-reviewed, temporarily-removed-while-failing, utility specs and @destructive. This batch is none of them.

"No daily history" is also circular — without the tag these specs never enter the daily, so they can never accumulate the history their own docs cite as the precondition.

The technical blocker is gone too: LANGFLOW_A2A_ENABLED=true has been on every lane and both start scripts since #1240, and the batch is LLM-free (Chat Input → Chat Output passthrough with a per-run sentinel — no provider, no key, no quota).

Evidence

npx playwright test --grep "@a2a" --repeat-each=3 --retries=0

Against langflowai/langflow-nightly:latest = 1.12.0.dev18 (four nightlies past the dev14 the docs recorded), LANGFLOW_A2A_ENABLED=true, LANGFLOW_WORKERS=1, default Playwright worker count.

Metric Result
Executions 51 / 51 passed (17 × 3)
Retries 0 — the local default of 3 disabled so a flake could not hide
Wall clock 53.4 s
🚨 Backend Error: 0
Flows before → after 26 → 26 (nothing leaked)
Re-run after the tag conversion 17 / 17, 21.3 s, 0 backend errors

The tag conversion — the larger half of the diff

These specs declared their tags inside the test title. The rest of the suite uses an inline tag: array, and scripts/lib/stable-tests.ts — the shared parser behind the Phase 0 block and the checklist guard — counts only a tag array containing the literal "@stable".

Appending @stable to the title would have put these 17 tests in the daily while leaving them out of every generated count: exactly the split #985 exists to prevent. So the tags moved to tag: arrays and the titles are now clean, matching the convention (stable-tests.ts treats titles as the source of the Phase 0 bullet text).

The describe blocks lose their title tags for the same reason — all 7 title-tagged describes in the entire suite were A2A's.

Parser before → after: 448 → 465 @stable test() calls, zero warnings.

Checklist symbols

  • Nine [-][x] in §16.1.
  • The [~] Try it bullet stays [~]. The gap it names is a product absence, not a coverage gap: agentTab.viewExchange is a dead i18n key — 7 locales, zero call sites, absent from the DOM after a completed turn (Create: A2A Agent tab — publish flow and the Try it panel (§16.1 UI) #1244). The spec still earns the tag; only the bullet symbol is held by the partial.
  • The eight [ ] bullets are untouched. §16.2 (A2A Client) needs the loopback-SSRF question answered first (follow-up 4 of the scoping doc); non-owner-publish 403 and the disabled-server state are already recorded as not automatable.

Per #741, the generated blocks are not regenerated here — update-coverage-summary.yml does that on merge. scripts/check-checklist-guard.mjs confirms no generated block was touched.

Docs

All 7 spec docs: the "No @stable yet" bullet is replaced with the tag plus the measured justification, and Last validated bumped 1.12.0.dev141.12.0.dev18. a2a-coverage-scope.md is deliberately left at dev14 — its Last validated refers to the surface map measurement, which this PR did not redo.

Gates run locally

Gate Result
npm run typecheck ✅ 0 errors
npm run lint ✅ 0 errors (1319 pre-existing warnings, unchanged)
npm run test:units ✅ 502 pass
npm run test:scripts ✅ 795 pass
npm run check:checklist-coverage ✅ every @stable spec referenced by a Part II bullet
node scripts/check-checklist-guard.mjs ✅ no generated block edited
npx playwright test --grep "@a2a" ✅ 17/17

Risk to watch after merge

The 4 UI tests (agent-tab-*) are the only plausible flake surface: the Agent-tab save has no DOM completion signal (agent-status flips optimistically), so both specs wait on the PATCH /api/v1/flows/{id} the button fires, and agent-card-name is mount-time state that a save does not refresh (asserted after a reload). Both were measured in #1244 and are handled; 3/3 green here. Worth watching in the first week of daily history.

🤖 Generated with Claude Code

The A2A batch shipped 7 specs / 17 tests across #1242, #1244 and #1247 and none
of them carried @stable, so daily-stable.yml — which filters on --grep "@stable"
— never ran a single one. Ten of the eighteen §16 checklist bullets were
automated and watching nothing: an A2A regression between two nightlies was
invisible to every scheduled lane, and the specs only ran when a PR happened to
touch them or someone dispatched manual.yml.

The reason the docs gave for holding the tag back does not survive contact with
CONTRIBUTING.md, which states the standard plainly: every new test enters with
@stable, in the PR itself, with the reviewer confirming the five validation
steps on approval. The four documented exceptions are inherited-not-yet-reviewed,
temporarily-removed-while-failing, utility specs and @destructive; this batch is
none of them. "No daily history" was also circular — without the tag these specs
can never enter the daily, so they can never accumulate the history their own
docs cited as the precondition for it. The technical blocker is gone too:
LANGFLOW_A2A_ENABLED=true has been on every lane since #1240, and the whole
batch is LLM-free (a Chat Input -> Chat Output passthrough with a per-run
sentinel, no provider, no key, no quota).

Evidence: 51/51 green (17 tests x 3, --retries=0 so a flake could not hide) on
nightly 1.12.0.dev18 — four nightlies past the dev14 the docs recorded — in
53.4s at the default worker count, with flows 26 -> 26 (nothing leaked) and zero
"Backend Error" logged. Re-run after the tag conversion: 17/17.

The tags had to move as well, and that is the larger half of the diff. These
specs declared their tags inside the test TITLE; the rest of the suite uses an
inline `tag:` array, and scripts/lib/stable-tests.ts — the shared parser behind
Phase 0 and the checklist guard — counts ONLY a `tag` array containing the
literal "@stable". Appending the tag to the title would have put these 17 tests
in the daily while leaving them out of every generated count, which is precisely
the split #985 exists to prevent. The describe blocks lose their title tags too:
all 7 title-tagged describes in the entire suite were A2A's. The parser now
reports 465 @stable test() calls, up from 448, with zero warnings.

§16.1's nine [-] bullets become [x]. The [~] Try it bullet stays [~] on purpose:
the gap it names is a product absence, not a coverage one — agentTab.viewExchange
is a dead i18n key (7 locales, zero call sites, absent from the DOM after a
completed turn, #1244). Its spec still earns the tag; only the bullet symbol is
held by the partial. The eight [ ] bullets are untouched — §16.2 needs the
loopback-SSRF question answered first, and the other two are already recorded as
not automatable.

Per #741, the generated blocks are left alone — update-coverage-summary.yml
regenerates them on merge.

Closes #1349

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rafaelgiln
rafaelgiln merged commit 3a3e062 into main Aug 7, 2026
7 checks passed
@Victor-w-Madeira
Victor-w-Madeira deleted the test/issue-1349-a2a-stable branch August 8, 2026 07:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Promote the A2A server batch to @stable — 17 tests that watch nothing today

1 participant