docs: truth-sync G1 PASS + T3-harness-built/experiment-deferred across AGENTS.md, CHANGELOG, eval README, roadmap, work log

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: ChanMeng666

AGENTS.md

@@ -66,7 +66,17 @@ measured on its own ledger:
   below statistical resolution at n≈160/arm. All 3 unfaithful assertions were room-count/
   topology derivations on under-determined briefs — T4's schema docs must make the band
   conventions and "assert a count only when the brief enumerates it" normative.
-  Next on the roadmap spine: T3 (L2 loop vs equal-budget resampling).
+- **T3: harness shipped, live experiment NOT run (2026-07-12).** The L2 tier is fully
+  implemented and offline-tested (`eval/l2.ts` pure protocol engine — diagnostic feedback
+  ≤2 rounds, oracle-isolated to compile/lint/`fix --dry-run`/trimmed-describe only;
+  `eval/l2-run.ts` guarded CLI with retrying author + per-brief error isolation; Olausson
+  **equal-token-budget i.i.d. resampling control** with round-up-favors-control accounting;
+  per-metric best-of, mean±σ over trials, `pass@n`/`pass^n`; `eval-l2.yml` workflow). The
+  ~440-call live run (est. $70–95) was **declined on cost** — so the decisive
+  loop-vs-resampling question **remains open**: do not claim a net model-loop gain (or its
+  absence) anywhere. To answer it later: dispatch **"Eval (L2 loop vs resampling)"** in
+  Actions (defaults = full run; smoke: `max 2, trials 1`). L3/L4/L5 stay unbuilt (gated on
+  an L2 net win).
 
 **Latest release — v1.13.0 (2026-07-11; AI-native authoring). Six tranches
 (see `CHANGELOG.md` for detail):**

CHANGELOG.md

@@ -15,6 +15,22 @@ point of the round was to **fix the ruler before measuring capability**: rebuild
 brief-grounded intent assertions, measure the free deterministic-tool gains on their own ledger, and
 write down a calibrated baseline honest enough to trust.
 
+### Added — Gate G1 verdict + the L2 experiment harness (2026-07-12; still repo-internal)
+
+- **Gate G1: PASS** (`eval/g1/` — generator harness, generated intents, double-blind scores,
+  report). NL→intent-JSON per-assertion faithfulness on all 26 briefs: **154/157 (98.1%)** vs
+  93.4% per-assertion accuracy of direct `.arch` generation (one-tailed z = 2.08, p = .019;
+  valid-only sensitivity variant below resolution — recorded). The intent channel (roadmap T4:
+  `src/intent.ts`, `arch validate --intent`, `intent.schema.json`) is **cleared** for a future
+  release. The generation prompt is oracle-isolated and `test/g1.test.ts` enforces it.
+- **T3 harness: the L2 tier** (`eval/l2.ts` + `eval/l2-run.ts` + `.github/workflows/eval-l2.yml`,
+  `npm run eval:l2`). Diagnostic feedback loop (≤2 rounds, fed only compile/lint diagnostics +
+  `fix --dry-run` previews + trimmed `describe()` — oracle-isolated) against an **equal-token-budget
+  i.i.d. resampling control** (Olausson accounting, round-up favours the control), per-metric
+  best-of, mean±σ across trials, `pass@n`/`pass^n`, retrying author + per-brief error isolation.
+  Offline-tested (14 tests). **The live experiment has not been run** (cost declined) — the
+  loop-vs-resampling question remains open and no loop-gain claim is made.
+
 ### Added — judge v2: brief-grounded intent scoring
 
 - **Intent-assertion scoring core** (`eval/assertions.ts`, `JUDGE_VERSION = "2"`). `scoreSource` no

docs/WORK-LOG.md

@@ -402,3 +402,39 @@ as history in `eval/live-baseline.json`'s notes.
 decisive experiment: does an L2 diagnostic feedback loop beat equal-budget resampling? Everything
 downstream (the intent CLI channel, constraint syntax, the repair-trajectory dataset) stays gated on
 what T3's number says.
+
+---
+
+## 2026-07-12 — Gate G1 (PASS) + T3 harness; the live L2 experiment deferred
+
+**Gate G1 — the intent channel's go/no-go: PASS** (`eval/g1/report.md`). A guarded,
+oracle-isolated harness (`eval/g1/generate.ts`, "Eval (G1 intent generation)" workflow) had
+gpt-5.5 write intent JSON (the `Expect` shape, lowered by `compileExpect`) from each of the 26
+briefs — 26/26 parsed, 157 assertions. Double-blind grading, with one honest process amendment:
+the human rater could not judge faithfulness cold, so rater A = three blind opus subagents
+(156/157 faithful), rater B = fable, pre-registered before reading A (154/157), agreement 98.7%
+(κ 0.50), and the human adjudicated the 2 disagreements (both ruled unfaithful). **Final:
+154/157 = 98.1%**, vs **93.4%** per-assertion accuracy of direct `.arch` generation
+(reconstructed reproducibly from the frozen calibrated-baseline scorecard by
+`eval/g1/baseline-accuracy.ts`, cross-checked row-by-row against its failure notes). Gate met:
+≥85% and one-tailed z = 2.08 (p = .019) above the primary control; recorded caveat — against the
+valid-only control variant (95.7%) the margin is below resolution at n≈160/arm. All three
+unfaithful assertions are room-count/topology derivations on under-determined briefs; the band
+conventions ("~N" → ±10% here vs the oracle's ±15%) must become normative schema documentation
+in T4. **T4 is cleared.**
+
+**T3 — harness shipped, experiment deferred.** The full L2 tier landed
+(`eval/l2.ts` pure protocol + `eval/l2-run.ts` guarded CLI + `eval-l2.yml`): diagnostic feedback
+≤2 rounds (compile/lint diagnostics + `fix --dry-run` previews + trimmed `describe()` only —
+oracle-isolated, statically tested), an **equal-token-budget i.i.d. resampling control**
+(Olausson `k = np + np·nfr` accounting in token form; the control's crossing sample is kept, so
+rounding favours the control — conservative toward the loop), per-metric best-of, mean±σ across
+trials, `pass@n`/`pass^n`, a retrying author (429/5xx/network backoff) and per-brief error
+isolation so one blip cannot sink a paid run. 14 offline tests; `eval:ci`/`results.md`
+byte-identical through the `run.ts` export refactor. **The ~440-call live run (est. $70–95 face
+value) was declined by the owner — the loop-vs-resampling question therefore REMAINS OPEN.** No
+net-loop-gain claim may be made in any doc until "Eval (L2 loop vs resampling)" is dispatched and
+scored. L3/L4/L5 stay unbuilt.
+
+**Gates.** 842 tests passing (93 files); typecheck + Biome + `eval:ci` green throughout; every
+commit ran the four-gate set.

docs/research/2026-07-roadmap-proposal.md

@@ -3,7 +3,10 @@
 **Date:** 2026-07-11 · **Status:** **T1 + T2 implemented 2026-07-11** (commits
 `60f5a87`…`83dc0cc`); **Gate G1 PASSED 2026-07-12** (`eval/g1/report.md` — faithfulness
 154/157 = 98.1% vs direct-generation 93.4%, z = 2.08 p = .019; T4 cleared, with a
-valid-only-control sensitivity caveat recorded); T3 in flight, T4–T6 remain open ·
+valid-only-control sensitivity caveat recorded); **T3 harness implemented 2026-07-12,
+live experiment deferred** (the ~440-call spend was declined — the loop-vs-resampling
+question stays open; dispatch the "Eval (L2 loop vs resampling)" workflow to answer it);
+T4–T6 remain open ·
 **Evidence base:** [`2026-07-ai-first-deep-dive.md`](./2026-07-ai-first-deep-dive.md) — every
 tranche below cites its verdicts (H1–H5) and the underlying sources. Written to be executable
 by a fresh implementation session with no other context beyond `AGENTS.md`.
@@ -117,6 +120,12 @@ honestly in AGENTS.md and skip T4.
 
 ## Tranche 3 — L2: the decisive experiment (PR3)
 
+**Status: ⚙ harness implemented 2026-07-12** (`eval/l2.ts` + `eval/l2-run.ts` +
+`eval-l2.yml`, offline-tested in CI, spec below followed incl. the equal-budget control
+and oracle isolation); **the live run itself was deferred** — the user declined the
+~440-call cost, so no L2 number exists yet and the drivability question below is still
+unanswered. Run it any time via the "Eval (L2 loop vs resampling)" workflow_dispatch.
+
 **Motivation:** H3 verdict — the one artifact that can confirm, refute, or bound the
 AGENTS.md drivability narrative, and that gates T4's headline status, T6's scope, and H4's
 reversal triggers. Designed to be able to say "the loop does not pay."

eval/README.md

@@ -20,9 +20,16 @@ model loop (deep-dive H3).
   edits, ADR 0011) then `arch repair` (the geometric corrector, ADR 0006) — **no model in the loop,
   zero extra API calls.** ΔL0→L1 is the free tool dividend. Implemented in `l1.ts` (`l1Pipeline`),
   gated offline in CI by `test/fault-injection.test.ts`, and overlaid on a live run with `--l1`.
-- **L2 — a diagnostic feedback loop (planned, not built).** Roadmap Tranche 3: does feeding
-  diagnostics back to the model beat equal-budget resampling? Until that is measured, `adjacent` /
-  `reachable` stay subscore-only (the T4 hook) and no L2 code exists here.
+- **L2 — a diagnostic feedback loop (built, not yet measured).** Roadmap Tranche 3: does feeding
+  diagnostics back to the model beat **equal-token-budget i.i.d. resampling** (Olausson,
+  arXiv:2306.09896)? The harness is implemented and offline-tested — `l2.ts` (pure protocol: ≤2
+  feedback rounds fed only compile/lint diagnostics with their `fix --dry-run` previews + a trimmed
+  `describe()`, oracle-isolated; the control arm matches the loop's *measured* token spend,
+  rounding its sample count up, which favours the control) and `l2-run.ts` (guarded CLI:
+  `npm run eval:l2 -- --yes [--trials N] [--max N] [--budget …] [--concurrency N]`, or the
+  "Eval (L2 loop vs resampling)" workflow). **The live experiment has not been run** (cost was
+  declined 2026-07-12), so the loop-vs-resampling question is still open; until it is measured,
+  `adjacent`/`reachable` stay subscore-only (the T4 hook) and no net-loop-gain claim may be made.
 
 Calibrated L0 baseline (`gpt-5.5-2026-04-23`, seed `20260711`, 26 briefs, judge v2, 2026-07-11):
 **valid 25/26 (96%) · intent 13/26 (50%) · sound 4/26 (15%)**. Same run's `--l1` overlay: **intent