A run can execute a model the persisted flow does not name — POST /api/v2/workflows does not build what the database holds

[rafaelgiln]

Found while validating #1334 (PR #1369). Neither the database nor the widget predicted what the run executed, so no observable the suite currently reads would have caught it.

Symptom

Measured 2 in 12 full-file runs of openai-compatible-provider-setup.spec.ts on 1.12.0.dev19. The Playground run fails with:

Error code: 404 - {'error': {'message': 'This is not a chat model and thus not supported in the
v1/chat/completions endpoint. Did you mean to use v1/completions?', 'type': 'invalid_request_error',
'param': 'model'}}

gpt-4o-mini is a chat model. The only ids this provider offers that produce that error are its completions-only ones — babbage-002 / davinci-002, the first entries of its default-enabled set — which is the signature of an empty model field falling back to the provider's first default-enabled id.

Why "the selection was reverted" is refuted, not assumed

The obvious reading is the known editor race: a debounced PATCH /api/v1/flows/{id} lands after the pick and carries the pre-selection state. That was tested and does not explain this.

The spec re-reads GET /api/v1/flows/{id} at the last moment the run can still be influenced — after the Playground modal is open, immediately before button-send. On both occurrences that read returned:

{ models: ["gpt-4o-mini"], providers: ["OpenAI Compatible"], credential: "" }

The widget (value-dropdown-model_model) read gpt-4o-mini too. So at send time the database and the UI agreed, and the run still executed something else: POST /api/v2/workflows did not build what the database holds. A repair loop was written, measured against this, and removed for that reason — re-selecting cannot fix a state that is already correct.

The reversion race is real and separate: it was observed once in 5 in the same session, with the node left on the editor's mount default (claude-opus-5 / Anthropic). That one the persisted-binding poll catches. This one it does not.

What is already in place

PR #1369 left the pre-send re-read in the spec as attribution, not repair, so the next occurrence says so in one line instead of implying a stale selection. The failure message is:

the persisted binding must still be this provider's model when the run is sent — if the run then executes another model, the run did not build the persisted flow

That is the extent of what #1334 claimed. The mechanism is not diagnosed.

Investigation directive

Product first, and instrument the boundary. (a) Capture the POST /api/v2/workflows request body on a failing run and compare the model field it carries against GET /api/v1/flows/{id} read at the same instant — that single comparison decides whether the frontend sends a stale in-memory graph or the backend resolves the field differently. (b) If the request is correct, follow the backend: instantiation.py reads model.value[0], and an empty or unresolvable value is what produces the first-default fallback — establish which. (c) Establish the blast radius: this is the unified ModelInput on the v2 run path, so it is not specific to OpenAI Compatible — the same fallback on a provider whose first default is a chat model would run silently and green, which is the more dangerous form and the reason this is worth chasing.

Why it matters more than its rate suggests

Here it fails loudly only because this provider's first default-enabled id happens to be completions-only. A provider whose first default is a working chat model produces a passing run against a model nobody selected — the #1169 silent-substitution class. Any spec that pins a model and asserts on the reply is exposed.

Deliverables (Done when)

• The request/response boundary is instrumented on a failing run, and the verdict names which side substitutes the model.
• If the run does not build the persisted flow, that is recorded as a product finding with a filed upstream ticket, and this issue stays open until it lands in the nightly.
• The blast radius is stated: which providers' first default-enabled id is a chat model, hence where this would pass silently.
• Whatever observable does predict the executed model is identified, so specs can gate on it — the persisted binding demonstrably does not.

[rafaelgiln]

Upstream filed: LE-2156

Verdict recorded: langflow-regression. Investigation on 1.12.0.dev19.

Deliverable 1 — which side substitutes: the editor, and the run is faithful to what it receives

• POST /api/v2/workflows carries data, declared in WorkflowRunRequest as an "Optional live-canvas override of the flow's nodes/edges; takes priority over the saved flow data". Captured on a healthy run — no flake needed:

  keys = flow_id, input_value, mode, stream_protocol, session_id, start_component_id, data
  data = PRESENT   data.nodes = 6
  node LanguageModelComponent-FLeYF model.value = [{"name":"claude-opus-5","provider":"Anthropic",...}]
  

• The run path does not substitute: get_llm (unified_models/instantiation.py) raises "A model selection is required" on an empty list and "The selected model is missing a provider" on a blank provider. It never falls back to a default.
• What substitutes is update_build_config: an empty model value is filled with options[0].

Deliverable 3 — blast radius, worse than this issue assumed

options is a flat list across enabled providers, so the fill need not be the same provider. Measured through POST /api/v1/custom_component/update:

field_value sent	returned
[{name:"gpt-4o-mini", provider:"OpenAI Compatible"}]	unchanged	preserved
[{name:"definitely-not-a-model", provider:"Nope"}]	unchanged	preserved
[{name:"gpt-4o-mini", provider:"OpenAI"}]	unchanged	preserved
[]	claude-opus-5 / Anthropic	replaced

An OpenAI-Compatible node came back as Anthropic. On 1.12.0.dev19, options[0] is a working chat model for Anthropic, Google, OpenAI, Azure AI Foundry, Ollama and OpenRouter — 6 of 8 — so the substitution normally runs green, on a provider nobody selected, billed to that provider's account. OpenAI Compatible is the loud exception only because its default set is endpoint-derived and starts with completions-only ids.

Deliverable 4 — the observable

The data payload of POST /api/v2/workflows, and nothing else. The persisted flow and the widget are both upstream of it. This is not "the observable was weak" — it was the wrong object, so no strengthening of the persisted-binding poll could ever have caught this.

Still open — the trigger, and two refutations that narrow it

What empties the field on a node that had a selection is not established. Both candidate backend paths were tested and refuted, which is the useful half:

1. Not an invalid-selection reset — a deliberately impossible model (definitely-not-a-model / Nope) is preserved, so the [options[0]] branch does not fire for out-of-catalog values.
2. Not an empty options list wiping the value — with every provider credential deleted, options comes back as 1, not 0 (the selection is injected into the options), and the value is preserved.

Both are backend paths, so the trigger looks like editor/frontend state that never round-trips through update_build_config.

Honesty about the rate

Pre-fix baseline, 12 runs of the untouched spec: 0 reproduced the 404. The runner reports "3/12 (25%)", but two of those carry unexpected=0 — no test failed, they were counted because a backend error was logged — and the single genuine failure carries TimeoutError: page.waitForResponse: Timeout 60000ms, a different symptom. The verdict rests on the deterministic experiment, not on frequency.

This issue stays OPEN

Per its own deliverable 2, until the trigger is identified and the fix lands in the nightly. Deliverable 4 is identified but not yet in code — a helper asserting on the run request payload is a separate change with wide reach and deserves its own decision.

Every scout was deleted, its leaked flow removed, and the three provider credentials the second experiment deleted were restored (instance verified back to Anthropic / Google / OpenAI configured).