fix(collect-models): wait for the credential write, not for a clock (#1355)

[rafaelgiln]

Closes #1355.

This PR's framing changed twice while it was open, both times because a run refuted it. The history is kept in the commits on purpose — the first commit's instrumentation is what produced the diagnosis the third one fixes. What follows is the final, measured version.

The measurement

Reproduced locally with a funded OpenAI key, a fresh container and every /api/v1/variables/ response logged:

collect-models: provider "anthropic" credential write took 103.1s (HTTP 201)

All three credential writes answer 201. None is rejected, none is refused. anthropic's simply does not come back for ~103 s — and the providers panel keeps its Save marked aria-busy for as long as a write is in flight. The next provider's click then lands on a busy button.

So the failure was never a broken button, a bad key, or a rejected save. It was a fixed clock waiting on a variable duration, and that duration grows with how many providers are already configured.

What it looked like before

Models found (openai):    [41 models]
⚠️  provider "anthropic" never showed the configured state ("Disconnect") within 60s of Save
⚠️  provider "anthropic" has a key configured but collected ZERO models
Models found (anthropic): []
Error: the "Save" button for provider "google" never became actionable after 60.0s over 232 poll(s)

Why it started now: the CI OPENAI_API_KEY was replaced with a funded key. A drained key failed fast; a funded one makes the save real, and the panel exposes 41 visible models (52 counting the collapsed deprecated ones) where it exposed none worth toggling.

The fix

page.waitForResponse for the credential write, registered before the click (or the response can land first and be missed), resolving the moment the write answers. The 180 s backstop is set well clear of the worst measurement rather than just above it, precisely because a fixed clock here is what failed.

The outcome is reported three ways, none silent: no write observed, a write that answered non-2xx, and a write that succeeded but took over 10 s. The last one is a trend line — the next time this cost approaches the ceiling, it is visible before it crosses it.

Two supporting changes from the earlier commits stay:

• The warnings — a provider that never reaches its configured state, and a provider with a key that collects zero models. These are what produced the diagnosis above, and the zero-models one matters beyond this bug: Collect models is continue-on-error in daily-stable.yml (ci: make the provider-key resolver warn-only in the daily (#976) #980), so a missing provider makes every parametrized LLM spec skip and the daily reports green having tested no agent — collect-models: single inaccessible lead model disables whole provider (16 OpenAI agent tests silently skipped) #570/shardguard counts blob files, not tests — a 0-test run reaches triage looking benign #1012's green-by-absence.
• The toggle serialisation, with its comment corrected to say what it actually is: added on a hypothesis the next CI run refuted, it fixed nothing, and it is kept only because a burst of 41 unconfirmed writes against a LANGFLOW_WORKERS=1 backend is worth avoiding at a measured cost of ~0 (23.9 s against 24.9 s).

Verification

• Fresh container, --retries=0: 1 passed (2.6m), all three providers collected — where the same command previously left anthropic at zero models and died on google's Save. This is the reproduction and the proof, on the same machine.
• 28 unit tests pass; mutation checks kill 5 of 6 and 3 of 4 on the two waits.
• tsc --noEmit clean; ESLint 0 errors (7 pre-existing any warnings in this file).

The lane's own Collect models is the CI-side confirmation.

🤖 Generated with Claude Code

[rafaelgiln]

The lane's own Collect models refuted the diagnosis this PR opened with — and named the real one

The first commit's instrumentation did its job on its first run in CI (job 92747128001):

Models found (openai): [41 models]
⚠️  provider "anthropic" never showed the configured state ("Disconnect") within 60s of Save
⚠️  provider "anthropic" has a key configured but collected ZERO models
Models found (anthropic): []
Error: collect-models: the "Save" button for provider "google" never became actionable after 60.0s
      aria-busy      true
      aria-disabled  true
      enabled        false

anthropic's Save was clicked on an IDLE button and never completed. So the original framing — "the previous validation is slow, the next click arrives early" — is wrong: waiting longer was never going to fix it. 120 s across the two providers and nothing settled.

The failure screenshot closes it. The captured panel is google's, its Save spinning, with nobody having clicked it — so the busy state belongs to the form, not to a button. And in the provider list: OpenAI carries a 52 models badge, anthropic carries none.

The write in flight is anthropic's, and what queues ahead of it is this loop. Enabling a model is a write, and the collector enables every model of every provider as fast as the clicks land. With a funded OpenAI key the panel exposes 41 visible models where a drained key exposed none worth toggling — one provider going from ~0 writes to 41, against a backend the lanes run with LANGFLOW_WORKERS=1.

That also answers "why only now" properly: the funded key did not make validation slower, it created 41 writes that did not exist before.

What the new commit adds

waitForToggleChecked confirms aria-checked="true" after each click, which serialises the writes. Two bounds, mirroring the token-attribution sidecar (#1197 §4.4): 5 s per toggle bounds one write, 60 s per provider bounds the sum — a per-item timeout alone would let 41 slow-but-succeeding confirmations spend 41 × 5 s and blow the spec's own 5-minute budget. Past the budget the clicks continue unconfirmed and the count is warned about rather than dropped (#1012).

• 4 more unit tests, same injected-clock approach: 28 passed, 0 failed
• mutation (delete the confirmation): 3 of the 4 die
• local collect-models: 23.9 s against 24.9 s before, 3 providers, 90 models, no new warning — a confirmation on a responsive panel costs tens of milliseconds

The first commit stays, and not just for history: it is what produced the diagnosis above, and its two warnings are what will make the next occurrence legible instead of a click timeout pointing at the wrong provider.

Still not proven here, same caveat as before: the local key is dry, so the 41-write path does not occur locally. This PR's own lane is the test.