Skip to content

The fixture's HTTP error monitor cannot fail a test, and covers 4 status codes rather than 4xx/5xx #1084

Description

@Victor-w-Madeira

Raised from @rafaelgiln's review of #1052, where every test in a 20-test slice logged
500 /api/v1/store/tags[Errno -2] Name or service not known (the container could not
resolve the Langflow Store) and no test failed. The question asked there was whether that is
a deliberate allowlist. It is not — there is no allowlist to be deliberate about.

What the code actually does

tests/fixtures/fixtures.ts monitors responses and collects two error kinds, then:

// after use(page)
if (flowErrors.length > 0 && !allowFlowErrors) {
  throw new Error(...);
}

Only flow_error can fail a test. An http_error is pushed into the array, counted in the
⚠️ N HTTP error(s) detected line, and then nothing happens to it — on any path, for any
endpoint. So the 🚨 Backend Error: line is purely advisory, and the only thing standing
between a real backend 500 and a green test is a human reading the terminal.

That human step is written down as a validation requirement — CONTRIBUTING.mdTest
validation checklist
step 5 ("Check the terminal logs") and CLAUDE.md → step 4 ("Confirm no
backend errors logged"). It is the load-bearing part of the mechanism, and the Store noise
above degrades it: a log that always carries three 500s is a log nobody reads carefully.

Second, narrower finding

The monitor is not the 4xx/5xx filter both docs describe:

status === 400 || status === 404 || status === 422 || status === 500

Four exact codes. 401, 403, 405, 409, 502, 503 and 504 are invisible to it — including the
403 — No authentication credentials provided shape that shows up under contention (seen in
the same #1052 review, and in #773), and the 502/503 a wedged or restarting backend
produces (#1030/#1048). CLAUDE.md:86 claims "automatic backend HTTP error monitoring
(4xx/5xx)"; that sentence is wrong today whichever way this issue is resolved.

The decision to make

Not obvious, which is why this is an issue rather than a PR:

  1. Make http_error fail the test — matches what both docs imply, but needs an allowlist
    first, or the Store 500 alone turns every spec red. Also needs a page.allowHttpErrors()
    escape hatch for specs that assert error responses on purpose (the API specs do), mirroring
    allowFlowErrors().
  2. Keep it advisory and make the log trustworthy — allowlist the known-benign endpoints
    (/api/v1/store/* when the Store is unreachable) so a logged error means something, and fix
    the two docs to say "logged, never fails".
  3. Widen the status filter — orthogonal to 1 vs 2, and cheap, but widening it while errors
    stay advisory only adds noise to a signal nobody is required to act on.

Whichever way it goes, CLAUDE.md:86 and the two checklist steps have to end up describing
the real behaviour.

Not in scope

The store/tags 500 itself is environmental (no outbound DNS in the container), not a Langflow
defect. It is the example that exposed the gap, not the thing to fix.

Evidence

Metadata

Metadata

Assignees

Labels

follow-upApproved exception: follow-up of merged work (ROADMAP Intake)qa-infraQA testing infrastructure: workflows, automation, evidence, tracking

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions