Symptom
Collect models fails, and with it every lane that runs it. Two consecutive runs of the same job on PR #1348, identical failure point:
Models found (openai): [41 models]
Models found (anthropic): [] ← 75 s later, empty
TimeoutError: locator.click: Timeout 20000ms exceeded.
- waiting for getByRole('button', { name: 'Save', exact: true })
- locator resolved to <button aria-busy="true" aria-disabled="true" …>
- 37 × waiting for element to be visible, enabled and stable
- element is not enabled
at tests/helpers/provider-setup/collect-models.ts:388
Runs: 92740997280, 92742195359.
The button is not disabled for lack of input — it is aria-busy, i.e. a save/validation started earlier is still in flight.
What changed to expose it
The CI secret OPENAI_API_KEY was replaced with a funded key on 2026-08-07 01:21 UTC. Before that, saving OpenAI failed fast on You have no credits remaining; now the validation is real and slow. collectModelsForProvider walks the providers in a loop and clicks the next Save without waiting for the previous provider's validation to settle, so the click lands on a busy button.
Last clean run before the swap went openai → anthropic in 34 s and passed in 1.2 min; after it, anthropic takes 75 s and comes back with zero models.
Caveat, stated because it bounds the diagnosis: the correlation is 2/2 after the swap against a green run before it, but the mechanism above is inferred from the DOM state in the error, not from a captured trace of the in-flight request. A reproduction with a funded key would confirm it.
Why this is urgent, and why it is worse on the daily than on a PR
The two lanes fail in opposite ways, and the quieter one is the dangerous one.
Three defects, one symptom
- The click does not wait for the button to be idle.
saveBtn.click() relies on Playwright's actionability check, whose 20 s ceiling is shorter than a provider validation (the code's own comment says Google takes ~35 s). When it expires, the error points at the click rather than at the provider whose validation never finished.
- A validation that never completes is swallowed. The
waitFor(Disconnect, 60_000) right after the click ends in .catch(() => {}), so "the provider was never configured" is indistinguishable from "it configured fine".
- Collecting zero models is not reported as a problem.
Models found (anthropic): [] is printed at the same level as a healthy list. For a provider whose key IS set, an empty collection is a failure — and it is the state that produces the silent daily.
Proposed fix
In collectModelsForProvider:
- wait for the
Save button to leave aria-busy / aria-disabled before clicking, with a ceiling sized against the validation the code already documents (~35 s for Google), and fail naming the provider and the observed attribute state rather than letting a generic click timeout point at the wrong line;
- when
Disconnect never appears after a save, say so — the provider is not configured and what follows will collect nothing;
- when a provider with a key present collects zero models, warn loudly rather than printing an empty array like any other result.
Related, not duplicates: #77 (closed) was the previous race on this same surface — stale button labels plus not waiting for the form to animate in; both of those are fixed and the current code waits for the input. #976 covers the opposite problem (every candidate key dead) and stays open.
Symptom
Collect modelsfails, and with it every lane that runs it. Two consecutive runs of the same job on PR #1348, identical failure point:Runs: 92740997280, 92742195359.
The button is not disabled for lack of input — it is
aria-busy, i.e. a save/validation started earlier is still in flight.What changed to expose it
The CI secret
OPENAI_API_KEYwas replaced with a funded key on 2026-08-07 01:21 UTC. Before that, saving OpenAI failed fast onYou have no credits remaining; now the validation is real and slow.collectModelsForProviderwalks the providers in a loop and clicks the nextSavewithout waiting for the previous provider's validation to settle, so the click lands on a busy button.Last clean run before the swap went openai → anthropic in 34 s and passed in 1.2 min; after it, anthropic takes 75 s and comes back with zero models.
Caveat, stated because it bounds the diagnosis: the correlation is 2/2 after the swap against a green run before it, but the mechanism above is inferred from the DOM state in the error, not from a captured trace of the in-flight request. A reproduction with a funded key would confirm it.
Why this is urgent, and why it is worse on the daily than on a PR
The two lanes fail in opposite ways, and the quieter one is the dangerous one.
pr-validation.yml—Collect modelsis notcontinue-on-error, so the job aborts:Run impacted specsisskipped. Loud. It is what forced PR fix(llm-agents): delete the flows three agent specs left behind (#1346) #1348 to be merged with a red lane.daily-stable.yml— there it iscontinue-on-errorby design (ci: make the provider-key resolver warn-only in the daily (#976) #980, so a drained key cannot kill a scheduled day). With nomodels.json,resolveTestTargets()has nothing to resolve and every parametrized LLM spec skips. The daily reports green having tested no agent at all — the green-by-absence failure collect-models: single inaccessible lead model disables whole provider (16 OpenAI agent tests silently skipped) #570 and shardguard counts blob files, not tests — a 0-test run reaches triage looking benign #1012 exist to prevent.Three defects, one symptom
saveBtn.click()relies on Playwright's actionability check, whose 20 s ceiling is shorter than a provider validation (the code's own comment says Google takes ~35 s). When it expires, the error points at the click rather than at the provider whose validation never finished.waitFor(Disconnect, 60_000)right after the click ends in.catch(() => {}), so "the provider was never configured" is indistinguishable from "it configured fine".Models found (anthropic): []is printed at the same level as a healthy list. For a provider whose key IS set, an empty collection is a failure — and it is the state that produces the silent daily.Proposed fix
In
collectModelsForProvider:Savebutton to leavearia-busy/aria-disabledbefore clicking, with a ceiling sized against the validation the code already documents (~35 s for Google), and fail naming the provider and the observed attribute state rather than letting a generic click timeout point at the wrong line;Disconnectnever appears after a save, say so — the provider is not configured and what follows will collect nothing;Related, not duplicates: #77 (closed) was the previous race on this same surface — stale button labels plus not waiting for the form to animate in; both of those are fixed and the current code waits for the input. #976 covers the opposite problem (every candidate key dead) and stays open.