fix(model-provider): assert the provider, not a credential name upstream stopped writing (#1334)

[rafaelgiln]

Closes #1334

Root cause — product-changed, not a regression and not a test defect

Upstream #14311 ("stop automatic provider field binding", 646bdd6b, forward-ported to release-1.12.0 on 2026-08-04) deleted the block that pre-populated a provider field's value with the credential variable name. template.api_key.value is now "" on every read, from mount onward. This spec asserted that removed binding inline, so its poll could never settle — models half correct, credential: "" against an expected "OPENAI_COMPATIBLE_API_KEY", on 2/2 attempts of run 31093877484.

The product hypothesis was refuted before the upstream lead was adopted, as the issue required. Differential on the same node, instance and session:

Selection	persisted template.model.value	persisted template.api_key
OpenAI Compatible / gpt-3.5-turbo (×4)	[{name, provider: "OpenAI Compatible", …}]	{value: "", load_from_db: false, show: true, required: false}
OpenAI / gpt-5.6	[{name, provider: "OpenAI", …}]	{value: "", load_from_db: false, …}

An empty api_key is build-wide and provider-agnostic, not an OpenAI-Compatible persistence failure. The backend agrees in source: apply_provider_variable_config_to_build_config is documented "Apply the current provider's metadata without changing explicit field values" and writes only show/required/advanced/info. models persisting while credential does not has one explanation under this reading — different writers: the selection is carried by the editor's autosave, api_key.value was written only by the deleted block, and it already reads "" before any selection is made.

The fix — same migration #1274 applied to the shared helper

The credential is no longer stored but is still determined, by the provider of the selected model, so the assertion moves onto that axis. tests/helpers/flows/agent-credential-settle.ts got this in #1274; this spec never did, because it asserted the shape inline — it was the last such copy under that spelling.

Proved causally, not assumed. instantiation.py takes provider = model.get("provider") off the selected entry and calls get_api_key_for_provider, which with an empty api_key resolves get_provider_secret_variable_key(provider). With the pair configured the run answers and the message carries properties.source: "gpt-3.5-turbo"; deleting only OPENAI_COMPATIBLE_API_KEY while a valid OPENAI_API_KEY stays configured account-wide leaves the persisted binding unchanged and turns the same run into 401 - Incorrect API key provided: EMPTY. The run resolves this provider's key and does not borrow the OpenAI provider's — which is what the old assertion existed to protect, and the provider axis is stronger here, because api.openai.com serves ids the plain OpenAI provider also serves.

api_key is now read and printed in the poll's diagnostic and asserted in neither direction (#1274's rule — requiring it empty would swap one dated premise for another and break a manual.yml dispatch at a pre-#14311 build).

Also in this PR — each found and measured while validating the above

• Test 5 enables its own model. gpt-4o-mini is default: false in this provider's live catalog, and the node's dropdown offers default-enabled or explicitly enabled ids only (model_catalog.py), so on a clean instance OpenAI Compatible-gpt-4o-mini-option never renders — 2/2 click timeouts, well before the binding assertion. It passed in CI only on model-status left by other specs: ambient state this spec never declared. Now enabled via POST /api/v1/models/enabled_models (the Azure AI Foundry sibling's call, same folder), disabled again in cleanup, with the catalog and the enabled map both polled first, plus the dropdown's own Refresh List as a bounded repair.
• Test 4 polls the terminal shape. It polled num_models > 0 and then re-read, catching the catalog mid-registration: 124 instead of 248 twice, and once an empty llm list, while the same endpoint seconds later reads {llm: 124, embeddings: 124}. Now one comparison over {llmIds, num_models, is_configured}, so a partial read is "not settled yet" rather than a wrong catalog.
• Test 6's quarantine is LIFTED — LE-2124 fixed. Concurrent POST /api/v1/variables/ writes answer 201/201 3/3 on 1.12.0.dev19, and the test passes 3/3 through the Settings UI with its assertions untouched. test.fixme removed, @stable added. Its rejection bodies are now captured, because an unreachable endpoint leaves the identical end state as the race: Invalid OpenAI-compatible base URL fails (the defect), a transport rejection skips with the body.
• Environment aborts skip instead of reddening. The backend validates the key by calling the endpoint, so a stalled network answers 400 … timed out; that shape is retried once, then skips with the reason. A drained account is read off the run's own message (no credits remaining / insufficient_quota) — invisible to the probe, since GET /v1/models answers 200 for a key with no credit, and left unattributed it surfaces 90 s later as "AI reply … not persisted yet". Deliberately narrow: rate_limit_exceeded still fails.

@stable is restored on test 5 (auto-removed by cb3082d).

Validation

Langflow 1.12.0.dev19 (= langflowai/langflow-nightly:latest, sha256:60cf62a2…), --workers=1 --retries=0.

Check	Result
Deterministic burst	3 clean runs
Final green after force-fail reverts	expected=6, unexpected=0, skipped=0 (116 s)
Further full-file runs on the final code	6/6 repeatedly; every remaining red carries an endpoint-stall signature
Pre-fix baseline	5/5 red
typecheck / lint / validate:specs / both QA-CHECKLIST guards	exit 0
Leftovers	0 flows, 0 OPENAI_COMPATIBLE_* variables, provider's enabled_models empty

Force-fail — all 6 tests verified red, then reverted:

t1  num_models expected 1 instead of 0 on the unconfigured live-only provider   → failed ✓
t2  validate-provider body expected valid:true for an unresolvable .invalid host → failed ✓
t3  expected a substring no validator message contains                          → failed ✓
t4  provider item expected to read '<n> ff-mutation' instead of '<n> models'     → failed ✓
t5  persisted binding expected provider 'FF-MUTATION Provider'                   → failed ✓
t6  variable pair expected to include a never-written FF_MUTATION_VAR            → failed ✓

No // FF-MUTATION marker remains in the diff.

Two findings NOT fixed here — separate issues to follow

1. tests/tests-automations/regression/mcp/client/mcp-client-agent.spec.ts:86 carries a second inline copy of the same dead premise, spelled api_key?.value rather than credential: — which is why the issue's grep missed it — inside a local 20 s toPass [Daily #744] agent/flow execution does not complete — div-chat-message / 'built successfully' never render (3 specs) #751 guard. It is @mcp @agents @regression with no @stable, so it never runs in the daily: latently broken, not currently red.
2. A run can execute a model the persisted flow does not name. Measured twice in 12 runs: the database and the widget both read gpt-4o-mini / OpenAI Compatible at send time, and the run still failed with 404 … This is not a chat model, which only the provider's first default-enabled id (babbage-002) produces. So it is not a persistence reversion and re-selecting cannot fix it — POST /api/v2/workflows did not build what the database holds. This PR attributes that state in one line rather than diagnosing it.

Reviewer note

The validation machine's network stalls ~20 s on 2 of 12 (host) and 3 of 12 (container) calls to api.openai.com, measured directly; it is not DNS or IPv6 (an /etc/hosts pin and forced -4 change nothing, and each IP answers in ~0.6 s when pinned). That is what the transport-skip paths above exist to attribute, and it is why some local runs show an explicit skip rather than a pass.

🤖 Generated with Claude Code

[rafaelgiln]

CI verdict: ambient-red — the E2E lane died in Collect models, and this PR's specs never ran

What failed. The Run impacted E2E specs job exits at its shared Collect models pre-flight, twice (first attempt, re-run). The step list confirms the lane never reached a Playwright run of the changed spec — it goes Collect models → Run destructive lane, with no impacted-specs step in between.

The cause, from the re-run's own log:

build axis: ✅ openai
build axis: ✅ anthropic
build axis: ✅ google
⚠️  collect-models: no credential write observed for provider "anthropic" within 180s of clicking Save.
    The panel stays busy while a write is in flight, so the NEXT provider is what pays for this (#1355).
⚠️  collect-models: no credential write observed for provider "google" within 180s of clicking Save.
1 failed  →  Process completed with exit code 1

All three provider keys validate (build axis: ✅ on each) — this is not a drained or revoked key. The failure is the Settings credential Save write not settling within 180 s, the #1355 signature that #1357 addressed earlier today; the first attempt surfaced the downstream form of the same thing (anthropic — no models collected from the providers panel, classified as a non-billing hard failure).

Evidence that it is ambient and not this branch:

1. This PR cannot reach that code. The diff is three files — openai-compatible-provider-setup.spec.ts, its spec doc, and one hand-written QA-CHECKLIST.md bullet. tests/collect-models.spec.ts is untouched, and no helper or Page Object is touched, so nothing in the import graph of the failing step changed.
2. It is not specific to this branch. The same no credential write observed for provider "anthropic" within 180s warning appears on chore/quarantine-daily-1361 — run 31175663512, 11:51 today — hours before this branch existed.
3. It reproduced on a clean re-run, 2 attempts out of 2, with the provider that stalled differing between them (anthropic on both, google additionally on the second) — the load-dependent shape of collect-models clicks a Save button still aria-busy from the previous provider — the PR lane aborts and the daily degrades to silent LLM skips #1355, not a deterministic consequence of a diff.
4. The spec this PR changes does not depend on the sweep at all. It configures its own OPENAI_COMPATIBLE_* pair over the API and is not part of collect-models.spec.ts; the sweep is forced only because provider-dependent-specs.mjs sees a @model-provider spec changed by the PR itself (pr-validation: needs_models is false for specs that need a configured provider but carry none of the classifier's markers #1216's rule), which is correct in general and simply costly here.

Caveat, stated rather than glossed: because the lane aborted in pre-flight, this PR carries no CI-side execution of its own change. The evidence for the change is local, on the same nightly CI targets:

Check	Result
Langflow version	1.12.0.dev19 — same image as this lane (Langflow Nightly 1.12.0.dev19 appears in this job's own preflight)
Deterministic burst, --workers=1 --retries=0	3 clean runs
Final green after the force-fail reverts	expected=6, unexpected=0, skipped=0
Further full-file runs on the final code	6/6 repeatedly
Force-fail	all 6 tests verified red under a marked mutation, then reverted; no // FF-MUTATION left in the diff
typecheck / lint / validate:specs / both QA-CHECKLIST guards	exit 0 (and all six non-E2E checks are green on this PR)

Why merging is still the right call. The red belongs to a shared pre-flight that is currently failing for other PRs too and that this diff cannot influence; the change itself is validated on the target build, and holding it parks a fix for a @stable spec that is red in the daily behind an unrelated infra recurrence. The #1355 recurrence is worth its own issue — the 180 s wait #1357 introduced is not sufficient on a loaded runner — and I will file it separately rather than fold it in here.

If a reviewer wants CI-side proof of this spec before merging, dispatch manual.yml with test_grep on the changed file once Collect models is healthy.