[Daily #1361] openai-compatible — model discovery stopped registering each served model twice (num_models 248 → 124)

[Victor-w-Madeira]

Spun out of daily-failure triage #1361 (run 31163810520, 2026-08-07).

Upstream: not filed

Symptom

the configured provider discovers exactly the models its endpoint serves hard-failed on the doubling assertion. The endpoint served 124 model ids; the spec asserts num_models === ids.length * 2 — the documented contract that /v1/models does not distinguish chat from embedding, so discovery registers every served model once per type. It got 124, not 248.

The assertion immediately above it — llmIds deep-equals the endpoint's own ids — passed, so discovery is running and the llm half is intact. Only the second registration is missing.

Spec (line)	Waits for	Signature
tests/tests-automations/regression/core-functionality/model-provider/openai-compatible-provider-setup.spec.ts:558 ("the configured provider discovers exactly the models its endpoint serves")	providerCatalog(request, PROVIDER_NAME).num_models	Error: expect(received).toBe(expected) // Object.is equality

Why these failures are one cause

One test, so no cross-spec grouping is claimed. What is claimed is that this is not a first occurrence: the same test failed on the 2026-08-06 daily as a flake, on the neighbouring assertion (expect(received).toEqual(expected) // deep equality, the llmIds comparison), and 2026-08-05 took :620 in the same file. The dataset therefore reports recurrence.same_signature: false — correctly, since the signature moved — and this issue is filed against that default, because the two days are the same test degrading in the same step, and 2026-08-06 was not a guard-tripped day (3 hard failures, well under the threshold), so this reproduced off a mass-failure day.

Preliminary read (descriptive — NOT a verdict)

This surfaced on a mass-failure day — 8 hard failures against the guard threshold of 5, so daily-stable.yml tripped the mass-failure guard and auto-removed @stable from nothing. The run also carried a confirmed backend outage (108 s on shard 1, 09:07:57→09:09:46 UTC). This failure is unrelated to both: it is deterministic (identical on attempts 0 and 2, exact integers both times), it touches no locator and no browser, and it reads a REST catalog rather than the wedged path.

Descriptively: the count halved to exactly the number of llm ids, and the llm id list itself is unchanged. That is the shape of the embedding registration no longer happening — but which side moved (discovery's behaviour, or the endpoint's /v1/models response) has not been measured, and the spec's own comment ties the doubling to #14199's embedding discovery, which is a place to look rather than an answer.

Investigation directive

Product first. (a) On the current nightly, configure the provider and read GET on the provider catalog directly — confirm whether num_models is now ids.length and whether any entry carries metadata.model_type === "embedding". (b) If embeddings are gone from discovery, establish whether that is intentional upstream (a change to discovery.py / #14199) or a regression — an OpenAI-compatible endpoint whose embedding models silently stop being registered is user-facing. (c) Only after (a) and (b): decide whether the spec's * 2 is still the right contract or whether it encoded a behaviour that upstream deliberately dropped. Do not soften the assertion to make it pass before that verdict exists. (d) Separately, check the 2026-08-06 toEqual failure on this same line — the id list differed that day and matches today, so the endpoint's catalog is not stable across days either, which could confound (a).

Deliverables (Done when)

• Root cause confirmed per spec (product regression vs. test/wait-strategy vs. environment), with evidence on the current nightly.
• Each spec passes reliably (multiple clean --retries=0 runs), fixing waits/flow as needed.
• Quarantine lifted in the fix PR — remove test.fixme and restore @stable, re-validated per CONTRIBUTING.md. (Nothing to lift if nothing was quarantined.)
• If the root cause is a product (Langflow) regression: recorded as such here, and this issue stays open until the upstream fix lands in langflowai/langflow-nightly:latest (or the release-1.x.x branch), is re-validated there, and @stable is restored — not on a test-side mute.
• The 2026-08-06 flake on this same line (different assertion) is explained or explicitly ruled unrelated — not left as an unexplained neighbour.
• If discovery legitimately stopped doubling, the spec's contract comment is rewritten to match, not just the number.

[Victor-w-Madeira]

A neighbouring test in the same file is also failing, on the PR lane (2026-08-07)

Observed while merging the quarantine PR #1366, whose impacted-specs lane ran this file against langflowai/langflow-nightly:latest (run 31175663512). Recorded here because it is the same provider surface, not because a shared cause is being claimed.

tests/tests-automations/regression/core-functionality/model-provider/openai-compatible-provider-setup.spec.ts:623 — "a discovered model runs a flow through the OpenAI Compatible provider" (@model-provider @components @playground, no @stable — auto-removed on 2026-08-06 by cb3082d) failed all three attempts:

TimeoutError: locator.click: Timeout 20000ms exceeded.
  - waiting for getByTestId('OpenAI Compatible-gpt-4o-mini-option').first()

and on the third attempt, earlier in the setup:

POST /variables/ OPENAI_COMPATIBLE_API_KEY -> 400
{"detail":"Connection to the OpenAI-compatible endpoint at https://api.openai.com/v1 timed out."}

Why it is worth holding next to this issue while investigating:

• The model option never renders, which is downstream of discovery — the same mechanism this issue is about, seen from the UI instead of the catalog.
• The 400 is the same message the flake at :515 produced on the 2026-08-07 daily, where the assertion expected Authentication failed for the OpenAI-compatible endpoint and received Connection to the OpenAI-compatible endpoint at https://api.openai.com/v1 timed out. That flake was noted-not-filed as a first occurrence; this is a second sighting of the same string, from a different test and a different run.

That raises a reading worth ruling out before concluding discovery changed: if the endpoint is intermittently unreachable from CI, a partial /v1/models response would explain a halved num_models without any upstream change at all. Step (a) of the investigation directive should therefore record whether the endpoint answered fully, not just what the catalog contained.

Both this test and :515 are additional evidence, not additional scope — neither is quarantined and neither is claimed by this issue yet.

[rafaelgiln]

Measured: the doubling is intact on 1.12.0.dev19 — the catalog is observable mid-registration

Directive (a) of this issue, run on the current nightly. Provider configured from scratch over the API, then five consecutive reads of GET /api/v1/models?provider=OpenAI Compatible, counting metadata.model_type:

read 1: num_models 124  by type {'llm': 124}
read 2: num_models 124  by type {'embeddings': 124}     ← the LLM half is the one missing here
read 3: num_models 248  by type {'llm': 124, 'embeddings': 124}
read 4: num_models 248  by type {'llm': 124, 'embeddings': 124}
read 5: num_models 248  by type {'llm': 124, 'embeddings': 124}

This refutes the preliminary read. Embedding registration was not dropped: both halves arrive, and the settled catalog is 248 = 124 llm + 124 embeddings. What varies is when each half is visible — read 2 shows the llm half absent with embeddings present, so neither side is "the second registration". Discovery fills the catalog in a window, and a read inside it returns a partial count.

It also answers deliverable 5 — the 2026-08-06 neighbour on the same line. That day's toEqual failure was the llmIds comparison; read 2 above is exactly that state (llm ids not yet registered) and produces exactly that assertion, from the same cause. The two days are one mechanism, not "the signature moved".

Why the daily looked deterministic across attempts 0 and 2. The spec's afterEach purges the OPENAI_COMPATIBLE_* pair unconditionally, so every attempt starts a cold registration — each retry re-enters the same window rather than reading a warm catalog. A warm-up window and a per-attempt-reproducible failure are not in conflict here.

I hit the same thing while working #1334 on the same file: num_models 124 instead of 248 twice, and once an empty llm list, against a catalog that reads {llm: 124, embeddings: 124} seconds later out of band.

What #1369 already changed on this assertion — and what it deliberately did not

PR #1369 (merged, 9de7772, for #1334) touched this test. Flagging it here because it landed while this issue was open and does not reference it:

• The assertion is NOT softened. num_models === probe.ids.length * 2 is still asserted, exactly, alongside llmIds deep-equalling the endpoint's own id set and is_configured === true. This issue's "do not soften the assertion before the verdict exists" holds.
• What changed is when it is read. It used to poll(num_models > 0) and then re-read — which is what lets a partial catalog reach the assertion. It now polls the terminal shape, comparing {llmIds, num_models, is_configured} in one toEqual, so a partial read counts as "not settled yet" instead of as a wrong catalog.
• A real removal would still fail, and fail clearly. If upstream ever does stop the second registration, the poll cannot reach 2 × ids and the test reds after its timeout with the full expected-vs-received shape. Nothing is masked.

The spec comment and the spec doc now describe this as a registration window, citing the measurements above.

Left for whoever owns this issue

I am not closing it — the verdict is yours to record. What is still open, if anything:

• whether the window is worth an upstream report (a UI read landing inside it shows a user a half-populated model list, which is the user-facing form of this);
• confirming the same reads on the image the failing daily actually ran, since mine is 1.12.0.dev19.

If those are not wanted, the deliverables here look satisfied by the above plus #1369.