Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
869 lines (630 loc) · 58.1 KB

CONTRIBUTING.md

File metadata and controls

869 lines (630 loc) · 58.1 KB
Raw

Contributing to Langflow E2E

This guide is for anyone creating, validating or maintaining tests in this repository.

How to create a new test

1. Choose the correct folder

Find the folder inside tests/tests-automations/regression/ that matches the functional area of the test:

What you are testing Folder
Login, logout, user management core-functionality/auth/
Flow execution, JSON import flow-functionality/
Model provider configuration core-functionality/model-provider/
Canvas, sidebar, sticky notes ui-ux/
REST endpoints api/flows/
File upload, RAG core-functionality/knowledge-ingestion-management/
LLM agents, reasoning, tool calling core-functionality/llm-agents/
MCP server or client mcp/server/ or mcp/client/

Folders with a CLAUDE.md contain specific instructions on how to create tests in that area. Read it before you start.

2. Name the file

The file must end in .spec.ts and have a descriptive kebab-case name that identifies the behavior under test:

login-invalid-credentials.spec.ts
agent-model-provider-selection.spec.ts
canvas-add-custom-component.spec.ts
flow-import-json.spec.ts

3. Basic file structure

import { test, expect } from "../../../fixtures";

test.describe("Area or feature name", () => {
  test("should [expected behavior] when [condition]", async ({ page }) => {
    await test.step("Step 1 description", async () => {
      // action
    });

    await test.step("Step 2 description", async () => {
      // assertion
    });
  });
});

Always import from fixtures — never directly from Playwright. The base fixture adds backend error monitoring: flow execution errors fail the test, HTTP errors are logged only (step 5 below explains why that distinction matters).

4. Use existing helpers and pages

Before writing actions from scratch, check if a helper or page object already exists for what you need:

// navigate to settings
import { SettingsPage } from "../../pages";

// load Simple Agent with configurable provider and model
import { SimpleAgentTemplatePage } from "../../pages";
await new SimpleAgentTemplatePage(page).load({ provider: "openai", model: "gpt-4o-mini" });

5. Add at least one tag

Every test must have a tag so it can be filtered by suite:

test("should configure the model provider", { tag: ["@model-provider"] }, async ({ page }) => {

See the available tags table in the README.

6. Update QA-CHECKLIST.md

After creating the test, find the corresponding item in QA-CHECKLIST.md and mark it as [-] (automated, needs validation). Only change to [x] after following the validation process below.

In a PR, edit ONLY the manual Part II bullets — never the generated blocks. The Coverage Summary table + note, the Phase 0 — Validated list, and the Phase 1/2 module tables are all derived data (from the [x]/[-]/[ ]/[~]/[!] bullets and the @stable tags). They are regenerated automatically by the update-coverage-summary.yml workflow on every push to main (committed with [skip ci]). Do not run npm run coverage:summary and commit its output in a PR — the counts are left to the merge job precisely so that many PRs can be in flight without colliding on the same count lines (two @stable add/remove PRs otherwise conflict on QA-CHECKLIST.md — see issue #741). Change only your bullet (- [x] <item> → <spec>.spec.ts); the pr-validation.yml QA-CHECKLIST guard job fails any PR that edits a generated block. If you regenerated by mistake: git checkout origin/main -- QA-CHECKLIST.md, then re-apply just your bullet edit.

The bullet is mandatory once the spec claims coverage. The same checklist-guard job also runs npm run check:checklist-coverage, which fails the PR when a spec carries @stable or has a spec doc under docs/ but no manual Part II bullet references it. Such a spec is invisible in every generated count — including the @stable lane the daily workflow reads as release signal (issue #985). The reference may be a bare filename or any deeper path slice, and an existing bullet that already describes the behavior can simply gain it; run the command locally to see the exact violation. Only these two claims are required — a spec with neither @stable nor a doc is not forced into the checklist, and a missing mirrored doc is never a defect (docs may be shared under another filename, so docs resolve by reference, not by filename).

7. Create the spec documentation file

For each spec, create a corresponding .md file in docs/, mirroring the relative path from regression/. For example:

tests/tests-automations/regression/core-functionality/playground/playground-session-id.spec.ts
→ docs/core-functionality/playground/playground-session-id.md

Use docs/TEST-SPEC-TEMPLATE.md as a base. The mandatory sections are: What this test validates, Tags, Validation criterion and External dependencies.

In the Last validated field, record the Langflow release cycle in which the test was developed or last reviewed (e.g.: Langflow 1.10.x). Validate preferably against the langflowai/langflow-nightly:latest image, which tracks the release branch under development. If the nightly is unstable, use the corresponding release branch (release-1.x.x) directly. In both cases, the field should reflect the cycle, not the exact build.

The External dependencies section lists files from the upstream Langflow repository that, if changed, could break the test. It is read by file-watcher.yml to determine which tests need review when Langflow changes. Fill it in carefully.

Do not build on legacy components

Langflow marks deprecated components with legacy: true and hides them from the sidebar by default — they only appear when the showLegacy preference is enabled. Do not build test flows on legacy components: they carry no maintenance commitment from upstream, can be removed or change behavior without notice, and are an unstable base for regression.

Use the maintained core equivalents instead. The canonical case:

Legacy (avoid) Use instead
Text Input Chat Input
Text Output Chat Output

Selector mapping when migrating: input_outputText Input/Output → input_outputChat Input/Output; button_run_text * → button_run_chat *; the Chat Input output handle is chat message (not output text); the Chat Output input handle stays inputs. Chat Input/Output are added minimized (minimized = True) — expand the node (more-options-modal → expand-button-modal, wait for hide-node-content to disappear) before reaching run buttons / inspector fields / shownode handles, or connect via the collapsed noshownode handles when the node is only a connection source/sink.

A component missing from the sidebar is not the same problem as a legacy one. Since 1.12, whether a component exists at all is a packaging decision per image: most families moved into per-vendor distributions plus an aggregate lfx-bundles the nightly does not install, so a family can be absent with no product announcement and it surfaces as a waitForSelector timeout 30 s deep. Before treating that as a product bug, read docs/component-distribution-policy.md — it holds the decision table (gate-and-skip, demote the bullet, or a bundle-enabled lane), the measured inventory, and the pre-flight drift report that names the cause at the gate. Do not decide it per incident; that is how #898 and #907 were both diagnosed wrongly.

The only exception is a test whose purpose is to validate legacy behavior itself — e.g. the legacy-visibility toggle (core-components/legacy-components-toggle-regression.spec.ts). In that case enabling showLegacy and using the legacy component is expected; state this explicitly in the spec doc's What this test validates section so a reviewer does not flag it.

Browser locale is pinned to en-US

playwright.config.ts pins the browser context locale (and the Accept-Language header) to en-US for every test in every project. The suite asserts on English strings throughout — toast titles, button labels, status messages, upstream error text — so the locale must stay fixed regardless of host machine settings, CI runner defaults, or future i18n locale detection in Langflow.

Do not override locale per test or per project. If a test ever needs a non-English locale (e.g. deliberate multi-locale coverage), that is a separate, parameterised concern — raise it as its own issue rather than editing the shared default.

Creating tests with LLM (agents, providers, MCP)

Tests that execute an agent with an LLM require a specific setup. Do not hardcode provider, API key or model — use the project infrastructure.

Before creating the test: generate the data

npx playwright test tests/collect-models.spec.ts

This validates the API keys for each provider and collects the available models in the UI, generating:

  • tests/helpers/provider-setup/data/providers.json
  • tests/helpers/provider-setup/data/models.json

Model parameterization pattern

The project uses a pattern where each model from models.json generates a separate test.describe.serial. See agent-component-regression.spec.ts as a complete reference.

Basic structure:

import * as dotenv from "dotenv";
import path from "path";
import { test, expect } from "../../../../fixtures/fixtures";
import { SimpleAgentTemplatePage, type LoadSimpleAgentOptions } from "../../../../pages";
import { hasProviderEnvKeys, type Provider } from "../../../../helpers/provider-setup";
import { resolveTestTargets } from "../../../../helpers/provider-setup/test-targets";

if (!process.env.CI) {
  dotenv.config({ path: path.resolve(__dirname, "../../../../.env") });
}

// One shared resolver — NEVER inline a copy of this (#1184). It reads models.json,
// applies the .env strategy (MODEL_TEST_ID → MODEL_TEST_PROVIDER → ALL_MODELS → one
// per provider) and attaches each provider's inactive-skip reason. Seventeen specs
// used to carry their own copy and they had drifted into five variants, two of which
// silently ignored MODEL_TEST_ID — same drift #1043 removed for providerSkipReasons.
//
// `tier` declares what the spec needs from a lane, so a lane can pick the cheapest
// target that satisfies it (#1185, #1187) instead of guessing per spec:
//   "tool-calling"     — the assertion IS model capability (tools, structured output)
//   "any-completion"   — any model that returns text; the assertion is plumbing
//   "none"             — no inference at all (modals, invalid-key UI)
//
// `any-completion` is the one tier a lane ACTS on today: with
// ANY_COMPLETION_PROVIDER=ollama (+ OLLAMA_TEST_MODEL) those specs run against a
// keyless local model — no key, no quota — and that routing outranks MODEL_TEST_ID /
// MODEL_TEST_PROVIDER for that tier, because #1185's daily pin is global to the run.
//
// The test for `any-completion` is "does ANY assertion depend on the model CHOOSING
// or MANAGING to do something", NOT "is the reply's content read" (#1187). The looser
// wording admitted a spec that fails ~40 % of the time: agent-system-prompt reads the
// reply only for a sentinel its own instruction demanded, which sounds like plumbing
// and is adherence. Dependence includes timing and throughput, not just compliance —
// a test that needs the run to still be in flight, or to finish three turns inside
// the 5-minute cap, depends on the model too. And the declaration is per FILE, so one
// model-dependent test disqualifies the whole file.
//
// Adoption is per spec, on a MEASURED rate over a stated N — never an N/N gate. At
// p=0.60 the old "3/3, no retries" gate certifies with probability ≈22 %, which is
// exactly how the pilot was adopted; no feasible number of consecutive CI dispatches
// fixes that (~59 runs for 95 % confidence in p ≥ 0.95). The rate is necessary and
// not sufficient: agent-component-regression passed 5/5 routed and is still
// `tool-calling`, because the criterion is dependence, not the observed count.
// Measure the SPEC on the LANE — a direct model probe measures a different system
// (10/10 called directly vs 60 % through the Agent, same model, same prompt).
// Add `requires: "vision" | "chat"` when the spec needs a specific capability
// WITHIN the provider.
const targets = resolveTestTargets({ tier: "tool-calling" });

for (const { label, options, skipReason } of targets) {
  const provider = options.provider ?? "openai";

  test.describe.serial(`My Test [${label}]`, () => {
    test("should ...", { tag: ["@agents"] }, async ({ page }) => {
      test.skip(!!skipReason, skipReason ?? "");
      test.skip(!hasProviderEnvKeys(provider), `Missing env vars for "${provider}"`);

      try {
        await new SimpleAgentTemplatePage(page).load(options);
      } catch (e: any) {
        if (e?.message?.startsWith("MODEL_NOT_AVAILABLE")) test.skip(true, e.message);
        throw e;
      }

      // your test here
    });
  });
}

Running agent tests

This is the most important step — and the most overlooked.

Intentionally break the behavior the test should detect and confirm the test fails. If it keeps passing, the assertion is not validating anything real.

# Run all models for a provider
MODEL_TEST_PROVIDER=openai \
  npx playwright test path/to/test.spec.ts --workers=1

# Run only a specific model
MODEL_TEST_ID=gpt-4o-mini \
  npx playwright test path/to/test.spec.ts --workers=1

# Run all models from JSON (default)
npx playwright test path/to/test.spec.ts --workers=1

For agent tests, how to force a failure depends on the behavior being validated:

What the test validates How to force failure
Response contains specific content Change the prompt to one that won't produce that content
Parameter affects behavior (e.g.: max_tokens) Comment out the line that sets the parameter in the test
system_prompt is respected Leave the instructions field empty
Memory is retained between messages Disconnect the Memory component from the flow
Tool result appears in reasoning panel Remove the tool from the agent before executing
New Chat clears the session Do not click New Chat before asking
Streaming is progressive Mock the response as non-streaming via page.route()

If the test passes in all these failure scenarios, it is a false positive and must not be merged.

4. Run in debug mode to walk through step by step

PLAYWRIGHT_BASE_URL=http://localhost:7860 npx playwright test path/to/test.spec.ts --reporter=html --trace=on
npx playwright show-report

2. Confirm that the test steps are documented

Every test must have test.step() describing what each block does.

3. Force a failure to confirm it is not a false positive

Comment out or invert the main assertion. The test must fail. If it passes even with the broken assertion, the scenario is not being truly validated.

4. Run in debug mode to walk through step by step

npx playwright test path/to/test.spec.ts --debug

5. Check the terminal logs

The base fixture prints backend errors automatically. Look for:

  • 🚨 Backend Error: — unexpected HTTP error. Logged, never fails the test (#1084)
  • 🚨 Flow Error Detected — silent failure in flow execution. Fails the test unless the spec called page.allowFlowErrors()

Because an HTTP error cannot fail a test, this step is the only thing standing between a real backend 500 and a green run — the fixture prints ⚠️ N HTTP error(s) detected — ADVISORY to say so out loud. Treat any line under it as a finding to explain, not noise to scroll past.

Which responses reach that log is decided by tests/fixtures/http-error-policy.ts (every 4xx/5xx on an /api/ route, minus documented exemptions: auth endpoints, and the external Langflow Store which is unreachable from CI). If your spec drives an endpoint into a 4xx/5xx on purpose — including mocking one with page.route — call page.allowHttpErrors() so the deliberate error stays out of the log instead of teaching readers to ignore it. Adding an endpoint to the exemption list makes it invisible to all 235 specs, so it needs a reason that survives review; run with PW_HTTP_ERROR_DEBUG=1 to see what is currently being ignored.

If instead your spec merely passes through a state where a known, filed product defect fires, declare that one response with page.expectKnownHttpError({ pathname, status, reason }) (#1008). It differs from allowHttpErrors() in the two ways that matter: only the exact status-plus-pathname it names is quietened, so every other 4xx/5xx in the test — including a different status on the same path — is still reported; and the declaration is verified, so a defect that stops firing fails the test and names the call to delete. That is what keeps the exemption from outliving its justification. The reason must carry the issue reference — it is printed on every run and it is what the next reader judges the exemption by.

6. Update the checklist

Only mark [x] after confirming all 5 steps above. If coverage is partial, use [~].

False positive anti-patterns

The patterns below produce tests that never fail — regardless of the application state. These are the most common mistakes when creating E2E tests and the hardest to detect in code review.

|| true — mathematically impossible assertion to fail

// ❌ WRONG — always passes, validates nothing
expect(hasResponse || hasSteps || true).toBe(true);
expect(claudeVisible || manageBtnVisible || true).toBe(true);

A test with || true in the assertion is a disguised placeholder. If you find this pattern in the codebase, rewrite the assertion or remove the test.

catch(() => false) without justification — silent soft check

// ❌ PROBLEMATIC without context — passes even if the element never appears
const isVisible = await element.isVisible({ timeout: 3000 }).catch(() => false);
if (isVisible) {
  await expect(anotherElement).toBeVisible();
}

The catch(() => false) pattern is acceptable only when the behavior is genuinely optional. In that case, document the intent with a comment explaining why the check is soft:

// ✅ CORRECT — intentional and documented soft check
// header-icon only appears when the agent uses tools.
// Models that respond directly without tools do not generate this icon — expected behavior.
const usedTools = await page.getByTestId("header-icon").last()
  .isVisible({ timeout: 3000 }).catch(() => false);
if (usedTools) {
  await expect(page.getByTestId("duration-display").last()).toBeVisible();
}

If the behavior is not optional, use a direct assertion:

// ✅ CORRECT — direct assertion for mandatory behavior
await expect(page.getByTestId("div-chat-message").last()).toBeVisible({ timeout: 30000 });
const responseText = await page.getByTestId("div-chat-message").last().innerText();
expect(responseText.trim().length).toBeGreaterThan(1);

Presence assertion without content validation

// ❌ WEAK — confirms the element exists, but not that it contains the correct data
await expect(page.getByTestId("div-chat-message").last()).toBeVisible();

// ✅ BETTER — validates that the content is what is expected
const response = await page.getByTestId("div-chat-message").last().innerText();
expect(response).toContain("Paris"); // for "what is the capital of France?"

Hardcoded provider in agent tests

// ❌ WRONG — tests only Anthropic, ignores OpenAI, Google, WatsonX, Ollama
await page.getByText("Anthropic").click();
await page.getByTestId("popover-anchor-input-api_key").fill(process.env.ANTHROPIC_API_KEY);

// ✅ CORRECT — parameterized by the project's provider infrastructure
for (const { label, options, skipReason } of resolveTestTargets({ tier: "tool-calling" })) {
  test.describe.serial(`My Test [${label}]`, () => {
    test("should ...", async ({ page }) => {
      await new SimpleAgentTemplatePage(page).load(options);
    });
  });
}

See agent-component-regression.spec.ts and the CLAUDE.md in the llm-agents/ folder for the complete pattern.

Branches

Use the <type>/<short-description> pattern in kebab-case:

Type When to use Example
feat/ New test, new helper or page object feat/agent-regression-multi-provider
fix/ Fix for a broken or flaky test fix/model-provider-selector-flaky
chore/ CI, checklist, dependencies, internal refactoring chore/update-nightly-workflow
docs/ Documentation update docs/update-contributing

Commits

Use the same prefix as the branch followed by a description in the imperative:

feat: add agent regression tests parametrized by model
fix: replace flaky selector in model provider test
chore: update file-watcher monitored paths
  • Maximum 72 characters on the first line
  • English preferred
  • No trailing period

Pull Requests

All work enters via PR — no direct push to main.

Process:

  1. Open the PR with the branch ready and the test validated
  2. Request review from another organization member before merging
  3. Use squash merge to keep the main history clean and linear
  4. After the merge, delete the local and remote branch: 
    git checkout main && git pull
git branch -d <branch>
git push origin --delete <branch>

What the PR must communicate:

  • What it adds or fixes
  • How the test was validated (the 5 steps from the guide)
  • Related issue, if it comes from a file-watcher alert
  • Roadmap linkage: the current-wave item it advances, or the approved exception issue that justifies off-wave work — a follow-up, a daily-failure triage issue, or a community-labeled regression (worked in severity order, when indicated). See ROADMAP.md

Describing a new test PR

The description of a test PR has a different responsibility than a feature or fix PR: it must communicate not only what was done, but what is being guaranteed. The reviewer must be able, without opening any file, to assess whether the test covers the correct behavior, whether the approach is sound and what the limits of the added coverage are.

For the reviewer: consult QA-SCENARIOS-GUIDE.md and locate the corresponding scenario. Verify that the implemented test covers the specified behavior — objective, preconditions and validation criterion. Divergences between the specification and the implementation must be flagged as blocking.

1. Covered tests table

List each test with a description of the system behavior it validates — not the execution steps, but the property that would break in case of a regression:

# Test What it validates
1 test name Description of the system behavior that would be detected if it regressed
2 test name Description of the system behavior that would be detected if it regressed

2. How each test was built

Describe non-obvious implementation decisions. If the test uses response interception, API injection, a pre-built flow file or any indirect mechanism instead of direct UI interaction, justify the choice — what makes the direct approach infeasible or inappropriate for the scenario:

Ex: field X has no editable UI under normal usage conditions; the test injects the value via API response interception to exercise the behavior without depending on external side effects.

3. Dependencies

Explicitly declare what the test needs to run correctly:

  • PRs or helpers that must be merged first
  • If it requires LLM: provider, model and environment variables needed in .env
  • Execution mode: serial or parallel, and why
  • Presence of cleanup afterEach and what it discards

4. What this test does not cover

Declare the negative scope — related behaviors the reviewer might reasonably expect to be covered, but that are out of scope for this PR and why:

Ex: does not cover the component's behavior when the API returns a 5xx error; does not validate integration with field Y, which belongs to another functional area.

5. Known limitations (if any)

Record workarounds, empirical timeouts, accepted race conditions or any decision that a future maintainer would need to understand to avoid introducing regressions when modifying the test:

Ex: the test waits N ms to ensure autosave before navigating; this value is empirical and may be insufficient in high-latency environments. The correct solution would depend on an explicit backend signal that is not available in the current version.

6. Update QA-SCENARIOS-GUIDE.md

For each new scenario covered, add an entry in QA-SCENARIOS-GUIDE.md with:

  • Objective — what the scenario validates in terms of system behavior
  • Preconditions — what needs to be configured or running
  • Step by step — the sequence of actions the test executes
  • Validation — the criterion that determines success or failure

The guide is the human-language specification of the automated tests. Keeping it up to date allows the reviewer to compare the implemented test with the specified behavior and assess whether the coverage is correct — without having to read the code.

Test maintenance

How the team learns that a test needs review

file-watcher.yml checks whether the official Langflow repository received commits in monitored paths within a window (since, default 24h) and opens an issue in this repository when it finds any.

It cannot run today. The workflow is disabled_manually in Actions and its cron was removed in 9da85fa, so it has no run history at all and a dispatch fails with HTTP 422: Cannot trigger a workflow_dispatch on a disabled workflow. Reviving it takes both: enable the workflow, then restore a cadence if one is wanted. Until then nothing here fires, whatever the monitored paths say. When it is dispatched after a long gap, widen since.

The issue reports:

  • Which functional area changed
  • The exact command to run the affected tests
  • Which section of QA-CHECKLIST.md to review

Monitored areas

The area table is not duplicated here — it lives in scripts/watch-upstream-areas.mjs (13 areas → paths, tags, checklist sections) and is printed by:

node scripts/watch-upstream-areas.mjs --mode=areas

Three rules govern it (issue #1092):

  • A path that cannot be evaluated is a failure, not a pass. The workflow runs --mode=check before the sweep; a monitored path missing from the upstream checkout fails the job by name. Before the guard, git log -- <bad-path> printed nothing and read as "nothing changed" — which is how constants/flow_constants.tsx sat dead in the list. That path has never existed upstream on any ref (the real file is src/frontend/src/flow_constants.tsx), so the entry was wrong from the day it was written and nothing ever said so. The guard is continue-on-error and the job is failed after the report exists: one upstream rename must not suppress the report for the other 12 areas.
  • Every src/lfx/ subtree is classified exactly once, either mapped to an area or recorded as out of scope with a reason (LFX_CLASSIFICATION in the same file). A subtree that appears upstream and matches no entry fails the guard, so the next step of Langflow's lfx migration forces a decision instead of widening a blind spot. That blind spot is what made #1091 hard to catch: the change that broke all six stdio registrations landed in src/lfx/src/lfx/base/mcp/security.py, inside a 70-file commit that four other areas did watch — so the sweep would have fired, but never under MCP Server, leaving @mcp out of the revalidation grep.
  • The window is validated, not guessed. since must be N hours|days|weeks|months ago, yesterday, or an ISO date; anything else is rejected with exit 2. git log --since= is parsed by approxidate, which never errors — undefined silently selects zero commits (a green run that reads exactly like a quiet day) and last thursdya silently widens to 200. The sweep also prints how many commits the window selected repo-wide and the newest commit in the checkout, so an empty result is legible instead of being confused with clean areas.

When you add or repoint a path, run the guard locally against a Langflow clone:

node scripts/watch-upstream-areas.mjs --mode=check --root /path/to/langflow

What to do when a file-watcher issue arrives

  1. Read the commits listed in the issue
  2. Run the tests indicated in the issue table
  3. For each test that fails or seems outdated, follow the validation guide above
  4. Update the necessary tests and mark QA-CHECKLIST.md
  5. Close the issue

Adaptive impacted-tests subset

adaptive-impacted.yml is a finer-grained companion to nightly.yml. It is disabled_manually in Actions today (as are nightly.yml, weekly-stable.yml and file-watcher.yml), so the cadence below describes what it does when enabled, not what is running. When it ran, each day at 04:00 BRT it:

  1. Queries Docker Hub for the current langflowai/langflow-nightly:latest digest and resolves the matching git SHA in the Langflow repo.
  2. Compares to the SHA of the last nightly we tested (repo variable LAST_TESTED_NIGHTLY_SHA).
    • Same → skip the run (no new image).
    • Different → diff Langflow source between the two SHAs and run only the specs whose ## External dependencies reference the changed paths.
  3. On success, advances LAST_TESTED_NIGHTLY_SHA to the current SHA.

The mapping comes from each spec doc's ## External dependencies section — keep that section accurate when you add or rename Langflow source paths a test depends on. Any path starting with src/ is parsed; trailing / matches anything inside the directory.

CLI for local inspection:

npm run impacted -- src/backend/.../webhook.py     # → list of impacted spec files
npm run validate:specs                              # → which specs have/lack External dependencies
npm run check:nightly-delta                         # → would the workflow skip or run today?

State persistence requires the secret GH_PAT_VARIABLES (a PAT with Variables: read & write); without it the workflow still runs but does not advance the cursor.

Unit tests

Playwright specs cover Langflow. Unit tests cover our own code: the scripts that generate the release signal, the guards that gate every PR, the automation that edits specs on main, and the helpers under tests/helpers/. These are pure functions over text — given a spec file, which test() calls carry @stable; given a Playwright JSON report, which tests hard-failed — the cheapest thing to test and the most expensive to get wrong, since a parser that silently under-counts becomes a wrong release signal with no failing test anywhere (issue #1017).

Write one whenever you add or change: a scripts/* code path, a guard's decision logic, or a helper whose defect would surface as a random spec failure instead of a reproducible one (#988, #1002).

Where the file goes

Next to the code it covers, named *.test.ts (or *.test.mjs for the dependency-free .mjs scripts) — same convention scripts/partition-shards.test.mjs already follows:

scripts/lib/stable-tests.ts             →  scripts/lib/stable-tests.test.ts
scripts/remove-stable-from-failures.ts  →  scripts/remove-stable-from-failures.test.ts
tests/helpers/provider-setup/x.ts       →  tests/helpers/provider-setup/x.test.ts
playwright.config.ts                    →  playwright.config.test.ts

The discovery roots are scripts/, tests/ and the repo root's top level (not recursive outside those two) — so a *.test.ts anywhere else is not run, which is the one way to add a unit test that silently never executes.

Never validate a helper in a scratch file outside the repo. That is what left the collect-models fallback with its only regression net living in a PR description (#1011) — the gap this section exists to close.

And never write one as a .spec.ts. Before this lane existed, the only way to test a script was to make it a Playwright spec, so tests/scripts/remove-stable-from-failures.spec.ts was one — which meant three assertions over a pure function booted a Langflow container and waited on the credential pre-flight to run, and a *.unit.spec.ts under tests/ also matches the impacted-specs pathspec (:(glob)tests/**/*.spec.ts) and fires the E2E job. That spec was migrated into scripts/remove-stable-from-failures.test.ts and deleted; if you find another, migrate it rather than adding to it.

Running them

npm run test:units      # TypeScript unit tests (scripts/ + tests/), node --test via ts-node
npm run test:scripts    # the .mjs half — dependency-free scripts/ helpers

Both run in pr-validation.yml's TypeScript Check job, so a failing unit test fails the PR. Neither needs a browser, a backend or provider keys.

To drive a single file (or a single test) while writing it:

node --require ts-node/register --test scripts/lib/stable-tests.test.ts
node --require ts-node/register --test --test-name-pattern "guard trips" scripts/remove-stable-from-failures.test.ts

The runner, and why this one

node --test with ts-node's CommonJS require hook. No new dependency, no second runner to keep current, and it works on the Node 20 that pr-validation.yml pins. A type error in the test file fails it too, so tsc and the runner agree.

Two consequences worth knowing before you fight them:

  • npm run test:units discovers files with find, not a glob. On Node 20 the test runner's own discovery pattern does not recognise .ts, so passing it a directory finds nothing and passes silently. The script therefore lists the files explicitly and fails when the list comes out empty, rather than reporting a green vacuous run.
  • playwright.config.ts pins testMatch: "**/*.spec.ts". Playwright's default pattern also collects *.test.ts, which would pull these into npx playwright test, --grep, the tag counters and the impacted-specs pathspec. Keep every real spec named *.spec.ts and every unit test named *.test.ts.

Rejected, for the record: --loader ts-node/esm (deprecated loader API), bumping CI to Node ≥22.6 for --experimental-strip-types (changes the Node every job runs on, for one lane), Playwright as the unit runner (a *.unit.spec.ts matches the impacted-specs pathspec and would boot a Langflow container to run a pure function), and vitest (better ergonomics, but a standing maintenance decision for what the require hook already does).

Making a script testable

A script whose main() runs at import time cannot be imported by a test. Guard it:

if (require.main === module) {
  main();
}

Then export the seam you want to assert on — a pure function over the parsed input, as collectHardFailures and parseStableTests are. When the behaviour under test is the file mutation (as with the @stable auto-removal guard), drive the real script as a subprocess over throwaway fixtures in a temp dir and assert the files came back byte-identical; a reimplementation of a guard is not that guard.

Tag @stable — validated tests

What it is

@stable identifies tests that have been reviewed by the team and confirmed as correct and reliable. Only these tests run in the stable workflow (daily-stable.yml, every weekday; weekly-stable.yml is a disabled fallback). A hard failure there automatically removes @stable from the offending test and opens an issue for triage — restoring the tag is the only human-gated step (see lifecycle below).

Standard: every new test enters with @stable

@stable is the default for any new test. The test enters with the tag in the PR itself, together with the documentation file in docs/ (see step 7 of the guide above). The reviewer, upon approving the merge, is confirming that:

  1. The test passed all 5 steps of the validation guide (trace, forced failure, debug, no backend errors, checklist)
  2. The validated behavior is real and relevant for daily monitoring
  3. The documentation file in docs/ is present with the mandatory sections filled in

If any of these points is absent or incomplete, the reviewer must request changes before approving.

test("should create a flow and run successfully", { tag: ["@workspace", "@stable"] }, async ({ page }) => {

The @stable tag coexists with other functional and cross-cutting tags — it does not replace them.

Exceptions — when the test will not have @stable

Three cases where the tag is intentionally absent:

  1. Inherited tests not yet reviewed — they exist in the repository but have not yet gone through the validation and documentation process.
  2. Tests temporarily removed while failing — the tag was removed while the test awaits correction (see lifecycle below).
  3. Utility specs — scripts that collect data or configure infrastructure rather than asserting product behavior (e.g. collect-models.spec.ts). These are not regression tests and must never enter the stable workflow.
  4. @destructive tests — a test that mutates account-wide state (deleting every project of the shared superuser, for instance) runs only in the destructive lane: playwright.config.ts excludes @destructive from every normal run and CI runs it alone afterwards with PW_DESTRUCTIVE=1. daily-stable.yml filters on --grep "@stable" and has no destructive lane, so a test carrying both tags would be excluded from the daily and silently never run. Treat the two as mutually exclusive until the daily grows a lane of its own (#1010).

When @stable is permanently absent (case 3): state the reason in the spec doc's Tags section so it is visible without reading PR history.

When @stable is temporarily absent (cases 1 and 2): no spec doc update is required — the absence is tracked via the GitHub issue and the commit that removed the tag (auto-removed by the workflow on a hard failure, or a manual PR for inherited tests).

@stable lifecycle: from the triage issue to restoration

On a red scheduled daily-stable.yml run, the workflow does two things automatically:

  1. Removes @stable from every attributable hard failure (a test that failed all retries with an error that could be its own), regenerates QA-CHECKLIST.md, and commits it to main with [skip ci] — no PR, no approval. Two things hold it back: the mass-failure guard (too many at once → treated as infra, nothing is removed) and the infra-signature exemption (a failure whose error is transport-level is wedge collateral and keeps its tag, whatever the count).
  2. Opens a single triage issue for the run, listing what was removed.

The triage issue is the analyst's inbox and dispatcher — not the tracking issue for each problem. Its only deliverable is the triage itself: read the run, route each occurrence into a dedicated issue (or enrich an existing one), and then close the triage issue. It never carries an investigation or a fix. The triage issue is closed only once the triage is complete — every needed dedicated issue created or enriched (hard failure / flake / skip, per the criteria below) and every test the criteria require quarantined (hard failures: @stable auto-removed by the workflow; recurrent flakes: quarantined via PR at this point — remove @stable and add test.fixme — as prevention). When the mass-failure guard trips (see below), the triage gains one extra deliverable: decide whether the day was environmental and, if not, manually quarantine the real hard failures (only durable cross-day clusters get a dedicated issue on a guard day; the rest is noted rather than filed — see the Mass-failure guard note below). The triage issue still closes at the end of that triage, like on any other day, with the noted collateral listed in its closing comment. The human steps are the fan-out, the manual quarantine for recurrent flakes, and lifting it after the fix.

The runbook — order, dedup, analysis depth, and how the follow-up issue must be written — is in Triage protocol — working the triage issue below.

daily-stable.yml run goes red
      │
      │  AUTOMATIC:
      │   • infra-signature failures set aside as WEDGE COLLATERAL (tag kept)
      │   • @stable removed from each remaining hard failure + committed to
      │     main [skip ci] (unless the mass-failure guard trips → nothing removed)
      │   • one TRIAGE ISSUE opened for the run, listing both groups
      ▼
Analyst works the triage issue (dispatcher) — order: HARD FAILURES → FLAKES → SKIPS:
      │
      ├─► per WEDGE COLLATERAL failure  (tag still in place)
      │        → NO per-spec issue. Triage the backend outage once, for the run
      │
      ├─► per ATTRIBUTABLE HARD FAILURE  (tag already auto-removed)
      │        → one DEDICATED issue per failure
      │          (group failures that share a root cause into one issue)
      │
      ├─► per RECURRENT FLAKE  (same error_signature within a 30-day window,
      │        confirmed in reports/daily-history.jsonl)
      │        → open a DEDICATED issue AND quarantine via PR (manual):
      │          remove @stable AND add test.fixme (together)
      │
      └─► per UNEXPECTED SKIP  (reason not already tracked)
      │        → open a DEDICATED issue
      ▼
Before opening ANY issue: search for an OPEN issue on the same subject
      │        → if one exists, ENRICH it instead of duplicating
      ▼
Problem resolved → quarantine LIFTED via PR (remove test.fixme + restore
      @stable); the dedicated issue is closed

Mass-failure guard. The threshold is the max_auto_remove input of the auto-remove-stable action (default 5, so the guard trips at 6+ hard failures in a run). Above the threshold, the workflow assumes an environment-wide failure and removes nothing, so a bad infra day cannot strip @stable off the whole suite. When it trips, the triage issue says so and the tags are still in place — investigate the environment; if it turns out not to be environmental, manually quarantine the real hard failures (remove @stable + add test.fixme). There is nothing to lift for tests never quarantined.

On a guard-tripped day, split the clusters by durability: a cluster whose same test + error signature also failed on other, non-adjacent dailies is a durable signal (it reproduces off mass-failure days) and gets its own dedicated issue as usual; today-only collateral (no cross-day recurrence) is noted, not filed — a dedicated tracker for what most likely vanishes when the instance recovers is throwaway triage noise (same reason a first-occurrence flake is noted, not filed). The collateral has no dedicated issue, but that is not a reason to leave the umbrella open: close it at the end of triage, as on any other day, listing the noted-not-filed collateral in the closing comment so the thread stays readable. The standing record is reports/daily-history.jsonl, not the issue — every triage recomputes recurrence from that file over a 30-day window, so a collateral cluster that persists is re-detected on the next run and filed then, whether or not an umbrella was left open. Keeping them open instead accumulates stale rows in the daily-failure list that every later triage has to dedup against, and contradicts the rule above that the triage issue is a dispatcher, never a tracker.

Infra-signature exemption (wedge collateral). A failure whose error is transport-level — the harness could not reach or talk to the backend — is not attributable to the spec that reported it. This holds for a hard failure (last error) and, since #1310, for a flake (first failed attempt) alike: the reason is the error, not the outcome, and a backend that stopped answering says nothing about the spec whichever way the retry went. It is collateral of a mid-run backend wedge (#1030/#1048), and @stable is left in place regardless of the mass-failure guard. The guard only ever covered the wide wedge (6+ failures); a wedge costing ≤5 tests used to strip their tags in an unreviewed commit — issue #1031.

The exempting signatures live in scripts/lib/infra-signature-patterns.json — read by infra-signatures.ts for the auto-removal path and by infra-signatures.mjs for the triage path, two accessors over one list because CommonJS and ESM cannot share a code module here (#1310) — and the list is deliberately narrow: only errors that cannot be a product assertion under any reading (apiRequestContext.*: Timeout, the globalSetup [preflight] … is not reachable, ECONNREFUSED/ECONNRESET/socket hang up, net::ERR_CONNECTION_*, DNS failures). Signatures a wedge also produces but a real regression produces too — locator.click: Timeout, page.waitForSelector: Timeout, expect(...).toBeVisible() — are not on it, because exempting them would switch auto-removal off for most genuine breakage. Widening the list is a deliberate change, not a convenience: add a pattern only when it is transport-level, and add a case to scripts/remove-stable-from-failures.test.ts with it.

What this changes for triage. The umbrella issue renders collateral in its own block, ahead of the removals. Do not open a per-spec issue for a collateral failure — triage the outage once for the run (start from the backend liveness section of the same issue, then the Langflow service container log; WORKER TIMEOUT ⇒ #1048). The tag is still in place, so there is nothing to restore. If the same spec keeps appearing as collateral across days while other specs do not, that is a signal about the spec (it hammers the backend hardest) and belongs on a dedicated issue about load, not about the assertion.

The in-run liveness verdict (#1030) is corroboration, not the criterion: the exemption is decided on the failure's own error, so it still applies on a run where the recorder measured nothing — which is exactly the run whose backend state is least known. The liveness section itself says overlap with an outage window is a lead, not a verdict, because at a 33-73% down-share a failure lands inside a window by chance.

Trade-off. Auto-removal does not distinguish a product regression from a test bug before removing — a hard failure quarantines the test either way, and the classification happens afterwards on the dedicated issue. Accepted trade-off: a genuine regression stops being tracked by the stable suite until a human restores the tag, but the daily stops going red immediately.

Triage protocol — working the triage issue

The triage is dispatch, not solution. Its analysis is preliminary and descriptive: it says what happened, it does not conclude why. The deep investigation always happens on the dedicated issues, so that we keep tight control over what gets analysed and where. Explicit beats implicit — every member and contributor triages the same way by following the steps below, in this order of severity.

1. Hard failures (first).

  • Open one dedicated issue per failure.
  • Group two or more failures into a single issue when they share an equal or related root cause — a shared root cause is one problem.
  • The @stable tag was already auto-removed by the workflow; there is no manual removal at this step.

2. Flakes (second). Consult reports/daily-history.jsonl.

  • Open an issue only for a recurrent flake. A first occurrence is only noted in the triage — the retry budget absorbs single-run noise.
  • Recurrence window: 30 days. The criterion is the cause, not the count: it is not enough that the test flaked 2+ times in the window — the occurrences must share the same error_signature (the cheap, descriptive proxy for "same cause" at triage time). Same signature within the window → recurrent → open a dedicated issue and quarantine the test via PR as prevention (flake quarantine is always manual; the workflow never auto-removes flakes) — so the flaky test stops running until it is worked. Quarantine = remove @stable AND wrap the test as test.fixme(<reason + issue #>), together: @stable removal alone only stops the daily, so the test keeps going red on the pr-validation.yml impacted-specs gate (which selects specs by file diff, not by tag — the quarantine PR itself would go red; #871). test.fixme skips it in every context so the quarantine PR merges green. Lifting the quarantine (remove test.fixme + restore @stable) is a deliverable of that issue, done after the fix. (30 days is a revisable convention, not an absolute.)
  • Recurrence is necessary, not sufficient — the infra-signature exemption applies here too (#1310). A flake whose error is transport-level is wedge collateral, and filing plus quarantining it attributes a backend outage to a spec. The triage dataset enforces this rather than leaving it to judgement: such a flake comes back actionable: false with an infra_excluded block naming the signature, so it is noted against the run's backend outage and neither filed nor quarantined. It is demoted, never dropped — an excluded flake stays visible in the dataset and in the proposal (#1012). The measured case that exposed the hole: agent-context-id-isolation.spec.ts:512 on run 30997773754, a 20 s timeout on GET /api/v1/auto_login whose retry spent 108 of its 119 seconds inside measured backend downtime, recurrent 2× and therefore proposed for quarantine by the letter of the old rule.

3. Skips (third).

  • Analyse the reason for each skip.
  • Open an issue only for an unexpected skip, or one whose reason is not yet tracked. An intentional/known skip already linked to an open issue is just noted — no new issue.

Before opening any issue — deduplicate. Search for an open issue on the same subject first. Another issue may already be attacking the same problem; if so, do not duplicate — enrich the existing issue with the new information (run id, error signature, new occurrence).

Depth of the triage analysis — keep it shallow and descriptive.

  • Do not assume the failure was caused by a test bug, a product regression, or the environment. Not even the environment — usually the easiest to spot — may be asserted without caution.
  • Environment signals (guard tripped, many unrelated tests failing together, a network/provider error in the signature) may be noted descriptively, as an observation — never as a verdict.
  • Advancing root-cause leads is welcome, as long as nothing closes the analysis. The definitive classification (product broke × test wrong) happens on the dedicated issue, not here.

Writing the dedicated (follow-up) issue

The dedicated issue spun out of the triage must be clear and agnostic:

  • Never assert that the problem is the test, a product regression, or the environment.
  • State explicitly that all paths must be investigated independently.
  • Burden of proof — suspect the product first. The analyst who picks up the issue is directed to treat the product as the prime suspect at the first moment: investigate whether the test failure is in fact pointing at a regression. A "test bug" explanation may be dismissed only after confirming the failure does not mask a regression. Once the regression hypothesis is ruled out, proceed to the other paths (test, then environment). If the evidence already points clearly at upstream UI drift, record that — but the confirm-it-is-not-a-regression step remains mandatory.

Classifying "product broke" vs "test wrong" — on the dedicated issue, not at triage:

Observation Classification
Assertion fails because the UI element moved or was renamed Test wrong — update selector
Assertion fails because the feature no longer exists or changed flow Behavior change — update test and spec doc
Assertion fails but the UI works correctly in manual testing Test wrong — fix assertion logic
Assertion fails and manual testing confirms the same failure Product broke — flag upstream

Restoration is a deliverable of the dedicated issue (upon fixing the problem):

  1. Fix the test following the validation guide (all 5 steps), or confirm the Langflow regression is resolved.
  2. Re-add the @stable tag in the correction PR — restoration is always manual, for both hard failures and flakes. The tag was removed as prevention (so the broken test stopped running in the daily); putting it back is an explicit deliverable of the dedicated issue, not an optional follow-up.
  3. Reference the dedicated issue in the PR body; close it upon merge.

When the root cause is a product (Langflow) regression: record it explicitly in the dedicated issue and do not close the issue on a test-side workaround. The issue stays open until the upstream fix has landed in the langflowai/langflow-nightly:latest image (or the corresponding release-1.x.x branch), the behavior is re-validated against it, and @stable is restored. A product regression is only "done" when the product is fixed where the suite runs — not when the test is muted.

The spec doc is not updated during this cycle — the auto-removal commit, the dedicated issue, and the restoration PR are the traceability record.

Regression Ledger — record every confirmed regression

REGRESSIONS.md (repo root) is the curated registry of the real Langflow regressions the suite has caught, and the source of the ROI indicator at its top. It is the suite's headline evidence that it finds product breakage a green run never proves — so keeping it current is not optional.

When a resolution confirms a langflow-regression verdict, has it been adversarially validated (refute-first across source/API/UI where applicable), and a filed upstream ticket exists (DataStax Jira LE-#### or a langflow-ai/langflow issue), adding a row to REGRESSIONS.md is a mandatory step of the report/resolution — not a follow-up.

Only regressions the suite caught earn a row: every row traces to a spec failure or a spec-validation run recorded in a repo issue. A regression the team confirms by hand — a manual Desktop session, an API investigation outside the suite — is filed upstream but stays off the ledger.

  • Confirmed but not yet ticketed → add it under Candidates, not the ledger.
  • A finding that adversarial validation downgrades to a non-user-facing robustness gap → do not list it (see the SQLite-lock counter-example in REGRESSIONS.md).
  • After editing the table, run npm run regressions:summary to regenerate the indicator block, then commit. Never hand-edit the block between the <!-- REGRESSIONS:START --> / <!-- REGRESSIONS:END --> markers. The pr-validation.yml typecheck job runs npm run regressions:check and fails the PR if the committed block disagrees with the table.

Monitoring rules driven by run history

Hard failures are now handled automatically (removed on the first red day — see the lifecycle above), so these history-driven rules are mostly about flakes, which are never auto-removed. Triage decisions should be informed by reports/daily-history.jsonl rather than by gut feel — the file makes recurrence visible. Before deciding whether to act on a flake, list every occurrence of the same test with its date and error signature (weekday dailies over 30 days are ≈22 runs):

jq -r --arg t "<full test title>" \
  '. as $row | (.failures + .flaky)[] | select(.test == $t) | "\($row.date)  \(.error_signature // "flaky")"' \
  reports/daily-history.jsonl | tail -25

Then compare the signatures — the action depends on what recurred (same cause) and whether it recurred within the 30-day window:

Symptom in the latest daily run History context Action
Hard failure, infra signature (transport-level error — the umbrella lists it as wedge collateral) any Not attributable to the spec. @stable was kept (#1031). No per-spec issue — triage the backend outage once for the run. Nothing to restore.
Hard failure (all retries failed) any @stable was already auto-removed (or the mass-failure guard tripped and left it in place). No manual removal — classify on the dedicated issue and restore the tag when the test is fixed.
Flake (passed on retry) No matching signature in the last 30 days Note in the triage; do not open an issue yet. The retry budget absorbs single-run noise.
Flake (recurrent) Same error_signature seen before within 30 days Open a dedicated issue (daily-failure + area:<...>, cite the run ids) and quarantine via PR as prevention — remove @stable and add test.fixme (tag removal alone leaves the test red on the impacted-specs gate; #871). Quarantine is manual — the workflow never auto-removes flakes. Lifting it (remove test.fixme + restore @stable) is a deliverable of that dedicated issue.
Flake with a different signature Prior flake on the same test, but a different signature Treat as a first occurrence of a new cause — note it; the raw count alone does not trigger an issue.
Mix (hard-failed one day, flaky later) 2+ days The hard-fail day already auto-removed @stable; once restored, apply the flake rows above if the flakiness persists.

Why a 30-day window keyed on the signature, and not "2 consecutive runs":

A flake that appears every other day never trips a "2 consecutive dailies" rule, yet it is a real, sustained problem — a 30-day window catches it. Keying on the error_signature (not on the raw count) is what separates a sustained cause from two unrelated blips that happen to hit the same test: two flakes with different signatures are two different (probably transient) events, not one recurrence. The history file is the source of truth: read the signatures out of the JSONL, do not rely on memory of past runs.

The history file is the signal source. The issue tracker remains the workflow. Do not skip opening issues just because the history shows a recurrence — issues carry the investigation and the fix; history only tells you when to open one.