chore(issue-pipeline): void infra aborts, require a pre-fix flake rate, gate the quarantine lift (#1082) (#1085)

Seven gaps surfaced while working #1060 with the prose orchestrator; the
deterministic pipeline could not represent any of them.

The environment can now answer for the spec and be recognised as such:
classifyRun splits a run into clean / infra-void / real-failure from its
failure messages, so a wedged backend dropping /api/v1/auto_login is re-run
instead of counted. A failure the classifier cannot read stays real — the
classification can never silence a genuine red.

DEBUG gained the evidence it was missing: a pre-fix repro rate for flake
issues (three clean VALIDATE runs do not distinguish a fix from luck at
single-digit flake rates), one verdict per symptom row (an issue's table can
list failures with different causes, and one may belong to another issue), and
an artifacts command that reads the failing run's own JSON blob.

VALIDATE now gates the quarantine lift, and the PR gate checks branch purity
against origin/main plus a CI verdict that makes an ambient-red merge carry a
written justification.

Also: FORCE_FAIL requires red runs only for runnable titles — a test.fixme
never executes, so demanding one deadlocked the phase.

Co-authored-by: Rafael <rafael@oriontech.me>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: 3 people

.claude/skills/langflow-e2e-issue-deterministic/DESIGN.md

@@ -70,6 +70,8 @@ Type variations (same 6 classes as the prose skill):
 npx tsx pipeline/cli.ts next 493                    # validate state, run mechanical work, emit exact next instruction
 npx tsx pipeline/cli.ts complete 493 <step> --evidence <args>
 npx tsx pipeline/cli.ts escalate 493 debug --reason "…"   # park current phase, enter DEBUG from anywhere
+npx tsx pipeline/cli.ts artifacts 493 --run <run-id> [--filter "<title>"]   # per-attempt status/error from the failing run's own JSON artifact
+npx tsx pipeline/cli.ts repro-run 493 --spec <path> [--grep "<title>"] [--runs 10]   # DEBUG only: PRE-fix flake rate on the unmodified spec
 npx tsx pipeline/cli.ts status 493
 npx tsx pipeline/cli.ts abort 493 --reason "…"
 npx tsx pipeline/cli.ts metrics 493                 # benchmark summary (JSON)
@@ -150,6 +152,29 @@ prose skill / `langflow-e2e` references instead.
    `Closes #NNN` and the correct template; `roadmap` label present for wave issues.
    Post-merge instruction: verify the issue actually closed (the edited-`Fixes`
    GitHub quirk) and delete the branch.
+7. **Environment aborts are void, not verdicts** (#1082): `classifyRun` splits a
+   run into `clean | infra-void | real-failure` from its failure messages. A run
+   whose every failure carries an infra signature (`/api/v1/auto_login` timeout,
+   `socket hang up`, connection refused) is re-run and counted as neither; past
+   `PIPELINE_MAX_INFRA_VOIDS` (3) the phase stops naming the instance. A failure
+   the classifier cannot read stays a real failure — it can never silence a red.
+8. **Pre-fix flake rate** (#1082): for a flake-shaped issue, DEBUG completes only
+   with a `repro-run` baseline (≥5 runs on the unmodified spec) — or, when the
+   defect never reproduced, an explicit `evidence.mechanismProof`. Three clean
+   VALIDATE runs do not distinguish a fix from luck at single-digit flake rates.
+9. **One verdict per symptom row** (#1082): every `spec.ts:line` row in the
+   issue's table needs its own verdict; a row owned by another issue carries
+   `ownedBy:"#NNNN"` and must be referenced in the PR body.
+10. **Quarantine lift** (#1082): when the issue quarantined a test, VALIDATE
+    fails while a `test.fixme` survives in a touched spec, and (when the issue
+    asks for it) while a quarantined title lacks `@stable`. Correspondingly,
+    FORCE_FAIL requires red runs only for *runnable* titles — a muted test
+    cannot be force-failed, and demanding it would deadlock the phase.
+11. **Branch purity + CI verdict** (#1082): the PR gate diffs
+    `origin/main..HEAD` against the files SPECIFY/IMPLEMENT recorded (plus
+    `QA-CHECKLIST.md`), failing closed when the base ref cannot be resolved; and
+    it requires `ciVerdict: green | ambient-red`, where `ambient-red` must carry
+    the URL of a justification comment that actually exists on the PR.
 
 ## State file
 

.claude/skills/langflow-e2e-issue-deterministic/SKILL.md

@@ -42,12 +42,19 @@ Other commands, only when the situation calls for them:
 ```bash
 $PIPE escalate <NNN> debug --reason "…"   # behavior looks broken, any phase
 $PIPE ff-run <NNN> --file <spec> --test "<title>" --mutation "<desc>"
+$PIPE artifacts <NNN> --run <workflow-run-id> [--filter "<test title>"]
+$PIPE repro-run <NNN> --spec <path> [--grep "<title>"] [--runs 10]   # DEBUG only
 $PIPE status <NNN>
 $PIPE metrics <NNN>                        # benchmark data
 $PIPE abort <NNN> --reason "…"
 $PIPE authorize-pr <NNN> --quote "<user's exact words>"   # ONLY after explicit user authorization
 ```
 
+`artifacts` downloads the failing run's `playwright-json-daily-<run>` blob and
+prints each attempt's status, duration and error — the first thing to read on a
+daily-failure issue, before any theory. `repro-run` measures the **pre-fix**
+rate of a flake on the unmodified spec (it refuses a dirty spec file).
+
 ## Hard rules
 
 - **Tests exist to catch real Langflow regressions, never to go green**: the
@@ -69,8 +76,37 @@ $PIPE authorize-pr <NNN> --quote "<user's exact words>"   # ONLY after explicit
   slow spec for zero added signal (a real session ran a spec ~10× this way).
   Scout runs to design/debug are fine; redundant confirmation bursts are not.
   Need more/fewer runs? Set `PIPELINE_BURST`, don't loop manually.
+- **An environment abort is not a spec result.** A run whose every failure
+  carries an infra signature (`/api/v1/auto_login` timeout, `socket hang up`,
+  connection refused — the wedged-backend class measured in #1074) is recorded
+  **void** and re-run; it counts neither as a failure nor as one of the clean
+  burst runs. Never "fix" a spec against one, and never report it as a defect.
+  Repeated voids stop the phase naming the **instance** as the blocker: restart
+  it, don't loosen the test. `PIPELINE_MAX_INFRA_VOIDS` (default 3) tunes the cap.
+- **A flake issue needs its PRE-fix rate.** Run `repro-run` on the unmodified
+  spec BEFORE changing anything; DEBUG will not complete without it. VALIDATE's
+  clean burst is not evidence on its own — at an 8 %-per-run flake, three green
+  runs is the expected outcome of doing nothing (#1060). If the baseline never
+  reproduces, prove the mechanism another way and say how in
+  `evidence.mechanismProof`.
+- **One verdict per symptom row.** A dedicated issue's table can list failures
+  with different causes; #1060's second row was another issue's `auto_login`
+  timeout, not the defect being fixed. Each row gets its own verdict, and a row
+  another issue owns carries `ownedBy:"#NNNN"` and must be named in the PR body.
+- **Lifting a quarantine is a deliverable, and it is gated.** If the issue
+  quarantined a test, VALIDATE fails while a `test.fixme` survives in a touched
+  spec — and, when the issue asks for the tag back, while a quarantined title
+  lacks `@stable`.
 - **Never bypass the CLI**: no `gh pr create`, no commit/push, no phase
   skipping, no editing files under `.claude/issue-pipeline/`.
+- **The branch carries this issue's files and nothing else.** The PR gate diffs
+  against `origin/main`; a rebase onto a local `main` that a parallel session
+  already committed to drags their work into your PR (#1060 — caught by hand).
+  Fix it with `git rebase --onto origin/main <their-commit> <your-branch>`.
+- **A red CI check is fixed or justified in writing, never ignored.** The PR
+  gate takes `ciVerdict: green | ambient-red`; `ambient-red` requires the URL of
+  a PR comment naming the cause, the evidence that it is ambient, and why
+  merging is still the right call.
 - **Never fabricate evidence**: `userConfirmed: true` only after the user
   actually confirmed in chat; `--quote` only with words the user actually said.
   Runner-written keys (runs/typecheck/lint/nightly/qaDiff/ff/finalGreen) are

.claude/skills/langflow-e2e-issue-deterministic/pipeline/artifacts.test.ts

@@ -0,0 +1,82 @@
+import { test } from 'node:test'
+import assert from 'node:assert/strict'
+import { summarizeRunArtifact, firstLines } from './artifacts.ts'
+
+// Shape of a real playwright-json-daily-<run> artifact, trimmed to what the
+// summarizer reads: three provider projects of the same spec line, one flaky
+// (passed then failed on a serial-group retry) and one hard failure.
+const ARTIFACT = JSON.stringify({
+  suites: [{
+    title: 'agent-context-id-isolation.spec.ts',
+    suites: [
+      {
+        title: 'Agent Context ID Isolation [openai / gpt-4o-mini]',
+        specs: [{
+          title: "switching the agent's context_id re-tags new turns",
+          file: 'llm-agents/agent-context-id-isolation.spec.ts',
+          line: 570,
+          tests: [{
+            projectName: 'chromium',
+            status: 'flaky',
+            results: [
+              { retry: 0, status: 'passed', duration: 25000 },
+              { retry: 2, status: 'failed', duration: 56000, error: { message: 'Error: expect(received).toBe(expected)\n\nExpected: "turns-tagged"\nReceived: "turn-2 message(s) with wrong context_id"' } },
+            ],
+          }],
+        }],
+      },
+      {
+        title: 'Agent Context ID Isolation [google / gemini-2.5-flash]',
+        specs: [{
+          title: "switching the agent's context_id re-tags new turns",
+          file: 'llm-agents/agent-context-id-isolation.spec.ts',
+          line: 570,
+          tests: [{
+            projectName: 'chromium',
+            status: 'unexpected',
+            results: [{ retry: 0, status: 'failed', duration: 58000, error: { message: 'TimeoutError: apiRequestContext.get: Timeout 20000ms exceeded.\nCall log:\n  - → GET http://localhost:7860/api/v1/auto_login' } }],
+          }],
+        }],
+      },
+      {
+        title: 'some other spec',
+        specs: [{
+          title: 'an unrelated test',
+          file: 'other.spec.ts',
+          line: 10,
+          tests: [{ projectName: 'chromium', status: 'expected', results: [{ retry: 0, status: 'passed', duration: 1000 }] }],
+        }],
+      },
+    ],
+  }],
+})
+
+test('summarizeRunArtifact reports every attempt of every matching test', () => {
+  const lines = summarizeRunArtifact(ARTIFACT, 'context_id')
+  const text = lines.join('\n')
+  assert.equal(lines.filter(l => l.startsWith('llm-agents/')).length, 2)
+  assert.match(text, /\[chromium\].*→ flaky/)
+  assert.match(text, /attempt 0: passed \(25s\)/)
+  assert.match(text, /attempt 2: failed \(56s\)/)
+  assert.match(text, /wrong context_id/)
+  // The two rows had different causes — the summary must show both verbatim.
+  assert.match(text, /auto_login/)
+  assert.ok(!text.includes('an unrelated test'))
+})
+
+test('summarizeRunArtifact without a filter covers the whole run', () => {
+  const text = summarizeRunArtifact(ARTIFACT).join('\n')
+  assert.match(text, /an unrelated test/)
+})
+
+test('summarizeRunArtifact says so when nothing matches or the blob is broken', () => {
+  assert.match(summarizeRunArtifact(ARTIFACT, 'nope')[0], /no test matching/)
+  assert.match(summarizeRunArtifact('not json')[0], /not valid JSON/)
+})
+
+test('firstLines strips ANSI colouring and caps the error head', () => {
+  const msg = 'Error: boom\n\nline two\nline three\nline four'
+  const out = firstLines(msg)
+  assert.ok(!out.includes('['))
+  assert.equal(out.split(' ⏎ ').length, 3)
+})

.claude/skills/langflow-e2e-issue-deterministic/pipeline/artifacts.ts

@@ -0,0 +1,84 @@
+/**
+ * Reading a daily run's own artifact is the highest-signal step of a
+ * daily-failure investigation: the `playwright-json-daily-<run>` blob carries
+ * every attempt's status, duration and error, which is what separates a real
+ * defect from an environment abort (#1060: the `unknown` signature on one row
+ * turned out to be #1030's auto_login timeout, and the flaky row's error named
+ * the stale context_id outright).
+ *
+ * The summarizer is pure so the shape of a run report can be unit-tested; the
+ * download lives in runners.ts.
+ */
+
+interface AttemptLike {
+  retry?: number
+  status?: string
+  duration?: number
+  error?: { message?: string }
+}
+
+interface TestLike {
+  projectName?: string
+  status?: string
+  results?: AttemptLike[]
+}
+
+interface SpecLike {
+  title?: string
+  file?: string
+  line?: number
+  tests?: TestLike[]
+  specs?: SpecLike[]
+  suites?: SpecLike[]
+}
+
+export function firstLines(message: string, max = 3): string {
+  return message
+    // eslint-disable-next-line no-control-regex
+    .replace(/\[[0-9;]*m/g, '')
+    .split('\n')
+    .filter(l => l.trim() !== '')
+    .slice(0, max)
+    .join(' ⏎ ')
+}
+
+function walk(node: SpecLike | undefined, out: SpecLike[]): void {
+  if (!node) return
+  for (const s of node.specs ?? []) {
+    if (s.tests) out.push(s)
+    walk(s, out)
+  }
+  for (const s of node.suites ?? []) walk(s, out)
+}
+
+/**
+ * One block per matching test: its file:line, then one line per attempt with
+ * status, duration and the head of the error. `filter` is a case-insensitive
+ * substring matched against the test title — pass the failing title from the
+ * issue to cut the report down to the rows under investigation.
+ */
+export function summarizeRunArtifact(raw: string, filter?: string): string[] {
+  let data: { suites?: SpecLike[] }
+  try { data = JSON.parse(raw) } catch { return ['✖ artifact is not valid JSON'] }
+  const specs: SpecLike[] = []
+  for (const s of data.suites ?? []) walk(s, specs)
+
+  const needle = filter?.toLowerCase()
+  const matched = specs.filter(s => !needle || (s.title ?? '').toLowerCase().includes(needle))
+  if (matched.length === 0) {
+    return [filter ? `no test matching "${filter}" in the artifact` : 'artifact contains no tests']
+  }
+
+  const lines: string[] = []
+  for (const spec of matched) {
+    for (const t of spec.tests ?? []) {
+      lines.push(`${spec.file ?? '?'}:${spec.line ?? '?'} [${t.projectName ?? 'default'}] ${spec.title ?? '?'} → ${t.status ?? '?'}`)
+      for (const r of t.results ?? []) {
+        const secs = Math.round((r.duration ?? 0) / 1000)
+        lines.push(`  attempt ${r.retry ?? 0}: ${r.status ?? '?'} (${secs}s)`)
+        if (r.error?.message) lines.push(`    ${firstLines(r.error.message)}`)
+      }
+    }
+  }
+  return lines
+}

.claude/skills/langflow-e2e-issue-deterministic/pipeline/cli.test.ts

@@ -1,6 +1,6 @@
 import { test } from 'node:test'
 import assert from 'node:assert/strict'
-import { parseArgs, sanitizeEvidence, releaseCycleOf } from './cli.ts'
+import { parseArgs, sanitizeEvidence, releaseCycleOf, allowedBranchFiles } from './cli.ts'
 
 test('parseArgs extracts command, issue, and flags', () => {
   const a = parseArgs(['next', '493', '--spec', 'tests/x.spec.ts', '--evidence-json', '{"a":1}'])
@@ -24,3 +24,26 @@ test('releaseCycleOf extracts major.minor', () => {
   assert.equal(releaseCycleOf('1.11.2.dev3'), '1.11')
   assert.equal(releaseCycleOf('1.5.1'), '1.5')
 })
+
+test('sanitizeEvidence also strips the new runner-written keys', () => {
+  const out = sanitizeEvidence({
+    reproRate: { runs: 99 }, artifactRuns: ['1'], verdict: 'test-defect',
+  })
+  assert.deepEqual(out, { verdict: 'test-defect' })
+})
+
+test('allowedBranchFiles unions what SPECIFY and IMPLEMENT recorded', () => {
+  const state = {
+    steps: {
+      SPECIFY: { evidence: { specDoc: 'docs/a.md', existingSpec: 'tests/a.spec.ts' } },
+      IMPLEMENT: { evidence: { files: ['tests/a.spec.ts', 'tests/helpers/h.ts'] } },
+    },
+  } as never
+  assert.deepEqual(allowedBranchFiles(state), [
+    'docs/a.md', 'tests/a.spec.ts', 'tests/helpers/h.ts', 'QA-CHECKLIST.md',
+  ])
+})
+
+test('allowedBranchFiles still allows the checklist bullet on a bare state', () => {
+  assert.deepEqual(allowedBranchFiles({ steps: {} } as never), ['QA-CHECKLIST.md'])
+})