feat(provider-setup): build-probe a provider's component, not just its API key (#900) (#1051)

* feat(provider-setup): build-probe a provider's component, not just its API key (#900)

collect-models recorded `active` from the raw-key probe alone. That probe calls
the provider's own API upstream of Langflow, so a valid key on an image that
cannot instantiate the component produced a false `active`, and the real failure
surfaced tens of layers downstream as a generic node-build timeout — a full
triage cycle each time (#898/LE-1974, #907/LE-1987).

Adds a second, independent BUILD axis in two layers, because neither catches the
other's shape. Which shape a provider produces is decided upstream by where a
maintainer puts an import, and it is inconsistent across components — measured on
1.12.0.dev8: OpenAI and Google import langchain at module level (missing package
=> absent from the registry), while Anthropic imports lazily inside build_model
(missing package => registers normally and fails at build). The "Groq shape" from
 #1039 is therefore already present in Anthropic today, so the catalog check this
issue originally proposed would not have been enough.

  1. Catalog — one GET /api/v1/all, exact provider -> component-key map. Replaces
     the substring token match, where `openai` also matched `OpenAI Compatible`,
     and skips the `component_display_names` pseudo-category.
  2. Build — every surviving component as a disconnected vertex of ONE throwaway
     flow, deleted in a finally. A credentials error is a PASS: it proves the
     client class was imported and constructed.

No paid call is possible: every SecretStrInput is neutralised first. This is
load-bearing, not hygiene — the registry template ships api_key as
{ value: "OPENAI_API_KEY", load_from_db: true }, so an un-neutralised probe would
load the global variable collect-models itself just saved and the default
text_output output would invoke the model for real.

The axis gets its own assert rather than riding on the existing env-keyed check:
that check skips providers whose env key is unset, but an unbuildable component is
a broken image regardless of whether anyone configured a key.

Fails OPEN on its own infrastructure, with a 60s build timeout against ~9s
measured. The daily's `Collect models` pre-flight shares its container with the
@stable shard and has a history of wedging it (#922/#927, #1011), so a probe
timeout degrades to a warning, never to a blocked pre-flight or a false inactive.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(provider-setup): make the build axis prove itself in CI, not just locally (#900)

PR #1051's first CI run exposed three defects in the build probe. All three are
fixed here, and the CI evidence is recorded in docs/collect-models.md so they are
not reintroduced.

1. FAIL-OPEN LOOKED LIKE A PASS. The probe timed out and still logged
   "build axis: ✅ openai / ✅ anthropic / ✅ google" — fail-open and real success
   produced identical output, so a probe that proved nothing reported everything
   fine. That is the exact blind spot this issue exists to remove, reintroduced one
   layer up. There is now a third state, `unknown`, distinct from ok and failed:
   logged as "⚠️ … [NOT PROVEN]", never written to providers.json, and never
   allowed to override the key verdict. Only a PROVEN packaging failure does.

2. THE PROBE KEPT THE SERVER WORKING AFTER THE CLIENT GAVE UP. The events GET
   expired at 60s while the build ran on; gunicorn hit WORKER TIMEOUT and the next
   CI step found the backend unreachable for 120s, failing the job. The probe now
   POSTs to /api/v1/build/{job_id}/cancel on timeout — stopping waiting and
   stopping working are the same act.

3. BATCHING DISCARDED ALL SIGNAL WHEN ONE COMPONENT WAS SLOW. All five were built
   in a single request, so exceeding the budget lost the verdict for every provider
   and the log could not even name the slow component. They are now built one at a
   time via `stop_component_id`, each with its own 20s budget (~6x the slowest
   measured) under a 90s ceiling for the axis.

Also measured while diagnosing, and recorded as external-dependency contract:
`event_delivery` defaults to `polling` and the events GET is a LONG-POLL that
blocks until the build finishes and returns every event at once — there is no
incremental read to lean on, which is why the budget is per component and paired
with cancel. Two hypotheses were tested and refuted rather than assumed: cold-start
imports (a restarted worker still probes in 9.3s) and the CI service container
holding provider keys (pr-validation.yml passes none, so no paid call was possible).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(provider-setup): run the build axis before the UI load, not after it (#900)

The reworked probe still reported `unknown` for all three providers on CI, twice,
with several components timing out on the POST that merely STARTS the build. That
is not a slow build — it is a backend that cannot accept the request.

Root cause is placement. `collect-models` saves three provider keys through the
Settings UI, and each save makes Langflow validate the provider and fetch its model
list. On the single-worker CI backend that is enough load to wedge it: both
pr-validation.yml and daily-stable.yml already carry a dedicated "Wait for the
backend to recover from the collect-models load" step immediately after this spec
for exactly that reason (#922/#927/#1044). The build probe was running inside that
load, before the recovery step — the worst possible moment. Two gunicorn WORKER
TIMEOUTs in the same window.

The probe consumes nothing the UI collection produces: not models.json, not the
saved keys, only the component registry. So it now runs FIRST, against an idle
backend, and `collectProviders` takes the verdicts as an argument instead of
probing concurrently with the key axis.

Locally (idle backend either way) this changes nothing — 3 clean runs, real ✅
verdicts, ~15s. The whole point is CI, where the previous placement produced no
signal at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Rafael <rafael@oriontech.me>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: 3 people

QA-CHECKLIST.md

@@ -46,6 +46,7 @@
 - [-] Google Generative AI Provider Setup → `helpers/provider-setup/setup-google.ts`
 - [-] Provider Map (`providerSetupMap`) — central registration point → `helpers/provider-setup/index.ts`
 - [-] Provider validation via API (credit, valid key) → `helpers/provider-setup/collect-models.ts`
+- [-] Provider build-axis probe (registry key present + component instantiates) → `helpers/provider-setup/probe-component-buildable.ts`
 - [-] Collection of available models via UI (Settings → Model Providers) → `helpers/provider-setup/collect-models.ts`
 - [-] `providers.json` — status of each provider (active/inactive + reason) → `data/providers.json`
 - [-] `models.json` — list of models per provider → `data/models.json`
@@ -384,6 +385,7 @@
 
 #### 7.1 Provider Collection and Validation
 - [x] Validate API keys of all providers via real call → `collect-models.spec.ts`
+- [x] Validate the running build can instantiate each provider's component (registry + build, not just the key) → `collect-models.spec.ts`
 - [x] Collect available models per provider via UI → `collect-models.spec.ts`
 - [x] Inactive providers appear as skipped in tests with reason → `agent-component-regression.spec.ts`
 - [x] Configure provider API key via Save Configuration (first setup) → `collect-models.spec.ts`

docs/collect-models.md

@@ -5,6 +5,7 @@ import { expect, test } from "./fixtures/fixtures";
 import { collectAll } from "./helpers/provider-setup/collect-models";
 import type { ProviderRecord } from "./helpers/provider-setup/collect-models";
 import { providerConfigMap, type Provider } from "./helpers/provider-setup";
+import { isBuildAxisReason } from "./helpers/provider-setup/probe-component-buildable";
 
 /**
  * Utility spec that populates the provider data files every LLM spec depends
@@ -79,6 +80,29 @@ test(
       }
     });
 
+    await test.step("this Langflow build can instantiate every provider's component", async () => {
+      // The BUILD axis (#900), asserted independently of the key axis below.
+      //
+      // Why this is not redundant with the env-keyed check that follows: that one
+      // skips every provider whose env key is unset. A component that cannot be
+      // built is a broken IMAGE, and that verdict does not depend on whether
+      // anyone configured a key — so without this step, running without (say)
+      // ANTHROPIC_API_KEY would let an unbuildable Anthropic component pass in
+      // silence. That is the same silent-skip class the whole issue removes.
+      //
+      // It is also never downgraded: a packaging gap is a broken environment, not
+      // a transient billing/quota outage, so it must fail loud every time.
+      const providers = JSON.parse(fs.readFileSync(PROVIDERS_PATH, "utf-8")) as ProviderRecord[];
+      const unbuildable = providers.filter((p) => isBuildAxisReason(p.error));
+      expect(
+        unbuildable.map((p) => p.provider),
+        `provider component(s) this Langflow build cannot instantiate — the image is ` +
+          `missing a distribution or a langchain-* package, and every spec parametrized ` +
+          `on them would fail downstream as a generic node-build timeout: ` +
+          unbuildable.map((p) => `${p.provider} — ${p.error}`).join(" | "),
+      ).toEqual([]);
+    });
+
     await test.step("every env-keyed provider is ACTIVE (transient billing/quota outages warn, don't fail)", async () => {
       // The #570 guarantee: a provider whose key is configured but that ends
       // "inactive" silently test.skip()s every spec parametrized on it (16

tests/collect-models.spec.ts

@@ -4,6 +4,7 @@ import path from "path";
 import fs from "fs";
 import { SettingsPage } from "../../pages/SettingsPage";
 import { providerConfigMap, type Provider } from "./provider-config";
+import { probeBuildAxis, type ProviderVerdict } from "./probe-component-buildable";
 
 const DATA_DIR = path.join(__dirname, "data");
 const PROVIDERS_PATH = path.join(DATA_DIR, "providers.json");
@@ -134,19 +135,9 @@ function modelsFor(models: ModelRecord[], provider: string): string[] {
 // runs green on; the catalog order stays as the tail so a provider with
 // none of the preferred models still validates on whatever it exposes.
 //
-// KNOWN GAP — "active" means the key works, NOT that Langflow can BUILD the
-// model. The probe calls the provider's own API directly, upstream of
-// Langflow, so it cannot see a missing server-side integration package. On a
-// nightly that shipped without `langchain-google-genai`, google probed
-// `active` here while every Google chat/embedding build inside Langflow raised
-// `ImportError: Could not import '...google_generative_ai_model' ... Install
-// the missing package`, surfacing downstream only as a misleading node-build
-// timeout across ~17 @stable specs (agents + Google-embedding KB). Root cause
-// + impact map: #898; upstream ticket: LE-1974. A faithful check would have to
-// BUILD a Language Model flow per provider and inspect the error — there is no
-// standalone endpoint that triggers the class import (`/api/v1/models/*`
-// return static metadata only). That build-probe hardening was deferred to
-// #900; this note is the trap marker for the next triager.
+// SCOPE — this probe answers "does the KEY work", nothing more. Whether Langflow
+// can actually BUILD the provider's component is the separate build axis in
+// `probe-component-buildable.ts`, merged in by `collectProviders` below (#900).
 const CANDIDATE_PREFS: Record<string, RegExp[]> = {
   openai: [/^gpt-4o-mini$/, /^gpt-4o$/, /^gpt-4\.1(-mini|-nano)?$/, /^gpt-4/],
   google: [
@@ -298,22 +289,47 @@ export async function validateProviderWithFallback(
   };
 }
 
-async function collectProviders(models: ModelRecord[]): Promise<ProviderRecord[]> {
-  console.log("Validating providers via API...");
+// A provider is usable only when BOTH axes pass, and they fail independently:
+// the key axis asks whether the provider's cloud API accepts the key, the build
+// axis whether THIS Langflow image can instantiate the component. A build-axis
+// failure overrides an `active` key verdict — a working key on an image that
+// cannot build the model is precisely the false `active` that made #898 and #907
+// cost a triage cycle each. The two run concurrently: the build probe is ~9s of
+// mostly-idle HTTP, and the key probe is network-bound too.
+async function collectProviders(
+  models: ModelRecord[],
+  buildAxis: Record<string, ProviderVerdict>,
+): Promise<ProviderRecord[]> {
+  console.log("Validating providers via API (key axis)...");
 
   const results = await Promise.all([
     validateProviderWithFallback("openai", rankCandidates("openai", modelsFor(models, "openai")), validateOpenAI),
     validateProviderWithFallback("anthropic", rankCandidates("anthropic", modelsFor(models, "anthropic")), validateAnthropic),
     validateProviderWithFallback("google", rankCandidates("google", modelsFor(models, "google")), validateGoogle),
   ]);
 
-  for (const r of results) {
+  const merged = results.map((r) => {
+    const axis = buildAxis[r.provider];
+    // Only a PROVEN build failure overrides the key verdict. `unknown` (the probe
+    // could not reach a verdict) must not: it says nothing about the provider, and
+    // writing it as `inactive` would turn a runner-side hiccup into a hard gate
+    // failure plus the silent downstream skips this mechanism exists to prevent.
+    if (axis && axis.state === "failed") {
+      // Recorded even when the key is fine: the specs parametrized on this
+      // provider cannot run either way, and the reason must name the layer that
+      // is missing rather than blaming the key.
+      return { ...r, status: "inactive" as const, error: axis.reason ?? "build axis failed" };
+    }
+    return r;
+  });
+
+  for (const r of merged) {
     const icon = r.status === "active" ? "✅" : "❌";
     const detail = r.error ? ` — ${r.error}` : "";
     console.log(`${icon} ${r.provider} (${r.model ?? "no model"})${detail}`);
   }
 
-  return results;
+  return merged;
 }
 
 // ─── Model collection (UI navigation) ─────────────────────────────────────────
@@ -441,11 +457,26 @@ export async function collectAll(page: Page): Promise<void> {
     fs.mkdirSync(DATA_DIR, { recursive: true });
   }
 
-  // Step 1: Collect models from UI via Settings
+  // Step 1: BUILD axis first, on a still-idle backend.
+  //
+  // Order is load-bearing, not cosmetic (#900). Step 2 saves three provider keys
+  // through the Settings UI, and each save makes Langflow validate the provider and
+  // fetch its model list — enough load on the single-worker CI backend that
+  // `pr-validation.yml` carries a dedicated "Wait for the backend to recover from
+  // the collect-models load" step after this spec (#922/#927/#1044). Running the
+  // build probe after that load put it at the worst possible moment: on PR #1051's
+  // CI run every component timed out, several on the POST that merely STARTS the
+  // build, so the axis reported `unknown` for all three providers and produced no
+  // signal at all. The probe needs only the component registry — not models.json,
+  // not the keys — so it can and must run before that load.
+  const knownProviders = Object.keys(providerConfigMap) as Provider[];
+  const buildAxis = await probeBuildAxis(page.request, knownProviders);
+
+  // Step 2: Collect models from UI via Settings
   const models = await collectModels(page);
 
-  // Step 2: Validate providers via API, falling back across candidate models
-  const providers = await collectProviders(models);
+  // Step 3: Validate the key axis and merge both verdicts
+  const providers = await collectProviders(models, buildAxis);
   fs.writeFileSync(PROVIDERS_PATH, JSON.stringify(providers, null, 2), "utf-8");
   console.log(`providers.json saved with ${providers.length} providers.`);
 

tests/helpers/provider-setup/collect-models.ts

@@ -41,8 +41,13 @@ import { getAuthToken } from "../auth/get-auth-token";
 // KNOWN LIMITATION: the match is a substring, so a short token can hit a
 // neighbouring component type (`openai` also matches `OpenAI Compatible`). Fine
 // for the current callers, whose tokens (`groq`, `mistral`) are unambiguous.
-// Lifting this probe into the `collect-models` gate needs an explicit
-// provider -> component-key map instead (#900).
+//
+// SCOPE — this probe stays the per-spec gate for providers that are NOT bundled
+// in the image (groq, mistral — #1039). The providers `collect-models` validates
+// go through `probe-component-buildable.ts` instead (#900), which uses exact
+// registry keys rather than this substring match AND adds the build layer this
+// one cannot supply: a registry hit does not prove the component builds, as the
+// trap above describes.
 export async function isProviderComponentAvailable(
   request: APIRequestContext,
   providerToken: string,