feat(evidence): gather source paper/refs as cached Markdown to lift the [source-missing] ceiling

Running ntt-fpga to v23 showed the loop working but stuck: reproducibility
oscillated 62→74→70 and never hit 80 even though the self-check now PASSES. The
cause is an information ceiling, not a bug — the scene holds only the CFNTT
paper's URL + a summary, so its signature structures (conflict-free bank map,
crossbar permutation, twiddle addressing, radix-4 schedule, FSM) are never in
the loop. Two engineers each invent a different valid reconstruction → permanent
divergence; amend rightly won't fabricate a [source-missing] value.

- evidence: `visually evidence <scene>` fetches metadata.info.sources via web
  tools, transcribes the technical sections to Markdown, caches under
  ~/.visually-3d/evidence/<id>/, falling back to a checked-in examples/<id>/
  seed. --refs also searches GitHub for reference impls (secondary, [ref-impl]).
- runner: add a `tools` option so a run can opt into web access. This is the
  ONLY tool-enabled step; the core reproduce/judge/amend loop stays tool-less.
- amend: buildAmendPrompt injects the gathered evidence and a provenance scheme
  ([paper] authoritative, [ref-impl] secondary); promotes a [source-missing]
  note to [paper] when evidence now supplies the value. Evidence enters amend
  ONLY — reproduce keeps grading the SPEC, not the paper.
- examples/ntt-fpga: distil the v1–v23 trial-and-error honestly — what's settled,
  what's still [source-missing], and which resource/ATP/timing claims python-smt
  can never verify (out-of-scope, not failures).
- paths: evidence/ workspace dir + packaged examples/ resolution; ship examples/.
- docs/CLAUDE.md: document the evidence substrate and the evidence→amend-only
  invariant. Tests: evidence store/loader/prompt + amend evidence injection.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: grandchildrice

.gitignore

@@ -28,6 +28,7 @@ __pycache__/
 /lib/create.js
 /lib/reproduce.js
 /lib/amend.js
+/lib/evidence.js
 /lib/arith-audit.js
 /bin/visually.js
 /lib/tui/app.js

CLAUDE.md

@@ -65,8 +65,8 @@ When converting/adding a CLI file: write `.ts`, add its generated `.js` to
 
 - `bin/visually.ts` — command dispatcher. Bare `visually` in a TTY launches the
   Ink TUI; otherwise/with args it routes to subcommands (`serve`, `create`,
-  `improve`, `reproduce`, `check`, `upload`). Keeps a `#!/usr/bin/env node`
-  shebang (preserved through tsc emit).
+  `improve`, `reproduce`, `amend`, `evidence`, `refine`, `check`, `upload`).
+  Keeps a `#!/usr/bin/env node` shebang (preserved through tsc emit).
 - `lib/tui/app.ts` — Ink + htm control panel (JSX without a build step). htm
   template markup is opaque to the type-checker; put real types on component
   props, hooks, and effects.
@@ -80,11 +80,19 @@ When converting/adding a CLI file: write `.ts`, add its generated `.js` to
   `defaultBackendFor(mode)` picks one (algorithm→python-smt, else→sim).
 - `lib/impls.ts` — canonical per-scene impl store under
   `~/.visually-3d/impls/<id>/` (`impl.<ext>` + `verify.txt` + `meta.json`).
+- `lib/evidence.ts` — `visually evidence <scene>`: fetch the scene's source
+  (paper/datasheet URLs in `metadata.info.sources`) via the runner's web tools
+  and cache a Markdown transcription under `~/.visually-3d/evidence/<id>/`,
+  falling back to the checked-in `examples/<id>/` seed. **Invariant: evidence
+  feeds `amend` ONLY, never reproduce's reverse-implementers** — reproduce must
+  keep grading the SPEC, not the paper. This is the only tool-enabled step
+  (`runClaudeStreaming({ tools: [...] })`); the rest of the loop is tool-less.
 - `lib/serve.ts` — static GUI server + SSE bridge to the local CLI. Endpoints:
   `/api/health`, `/api/analyze/stream`, `/samples/...`, `/api/impl/<id>`,
   `POST /api/impl/<id>/verify` (streams a live backend run).
 - `lib/paths.ts` — package vs workspace paths. Workspace is `$VISUALLY_HOME`
-  (default `~/.visually-3d`): `scenes/`, `runs/`, `impls/`.
+  (default `~/.visually-3d`): `scenes/`, `runs/`, `impls/`, `evidence/`. Curated
+  seed evidence ships checked-in under the package's `examples/<id>/`.
 
 ## Web app (`src/`)
 

bin/visually.ts

@@ -27,9 +27,13 @@ Usage:
   visually amend <scene>                  fold reproduce's findings back into the
        [--n 2] [--model <m>]              scene's functional spec (parts[].spec /
        [--backend <id>] [--no-verify]     metadata.spec) so it becomes reproducible
+  visually evidence <scene>               fetch the scene's source (paper/datasheet)
+       [--refs] [--model <m>]             via web tools, cache it as Markdown so
+                                          amend can QUOTE it (--refs: also search
+                                          GitHub for reference implementations)
   visually refine <scene>                 closed 3D ⇄ implementation loop: each
        [--rounds 3] [--visual 90]         round runs improve → reproduce → amend
-       [--repro 80] [--iters 2]           until the visual score and reproducibility
+       [--repro 80] [--iters 1]           until the visual score and reproducibility
        [--backend <id>] [--no-amend]      both clear their thresholds (or max rounds)
   visually check <scene> [--png]          inspect a scene (browser, or PNG contact sheet)
        [--out <file.png>] [--no-open]
@@ -73,6 +77,8 @@ async function main() {
       return (await import('../lib/reproduce.js')).reproduce(rest);
     case 'amend':
       return (await import('../lib/amend.js')).amend(rest);
+    case 'evidence':
+      return (await import('../lib/evidence.js')).gatherEvidence(rest);
     case 'refine':
       return (await import('../lib/refine.js')).refine(rest);
     case 'check':

docs/visioned-self-improvement-changelog.md

@@ -261,6 +261,45 @@ work that is still open. The same reproduce-findings seed that already steered t
 visual pass (`seedFromReport`) is what makes the reactive pass *feedback-driven*
 rather than a blind re-render.
 
+## 11. Source evidence — fixing the information ceiling, not the loop
+
+Running `ntt-fpga` to v23 showed the loop working correctly yet stuck:
+`reproducibility` oscillated 62 → 74 → 70 and never reached 80, while the
+self-check finally PASSED (the timeout was gone). The cause was not a loop bug —
+it was an **information ceiling**. The scene held only the CFNTT paper's URL and a
+summary, so the paper's signature structures (the conflict-free bank/offset map,
+the per-stage crossbar permutation, the twiddle-ROM addressing, the radix-4
+schedule and FSM) were never in the loop. Two reverse-implementers each invented a
+different valid reconstruction → permanent divergence → reproducibility capped;
+`amend` could not help because it (correctly) will not fabricate a
+`[source-missing]` value.
+
+The fix is to give the loop the missing information instead of asking it to invent
+more:
+
+- **`visually evidence <scene>` (`lib/evidence.ts`).** Fetches the scene's
+  `metadata.info.sources` with web tools, transcribes the technical sections to
+  Markdown, and caches them under `~/.visually-3d/evidence/<id>/`. `--refs` also
+  searches GitHub for reference implementations, kept strictly secondary
+  (`[ref-impl]`). This is the **only tool-enabled step**; `runClaudeStreaming` now
+  takes a `tools` option, and the rest of the loop stays tool-less and
+  deterministic.
+- **Checked-in seed (`examples/<id>/`).** Curated learnings ship with the package
+  and are the fallback when no evidence has been fetched. `examples/ntt-fpga/`
+  distills the v1–v23 trial-and-error honestly: what is settled and matches the
+  source, what is still `[source-missing]`, and which resource/ATP/timing claims
+  a functional backend can *never* verify.
+- **`amend` quotes the evidence.** `buildAmendPrompt` injects the gathered
+  evidence and a provenance scheme (`[paper]` authoritative, `[ref-impl]`
+  secondary, `[src]`/`[conv]`/`[calc]` as before); a value the evidence supplies
+  for a prior `[source-missing]` note is promoted to `[paper]`.
+
+**Invariant preserved.** Evidence enters `amend` ONLY. reproduce's
+reverse-implementers still see the spec alone, so reproducibility keeps measuring
+the spec's completeness — evidence enriches the spec, then reproduce grades the
+richer spec. What evidence cannot lift (synthesis-only resource/ATP/timing claims)
+is documented as out-of-scope for `python-smt` rather than counted as failure.
+
 ## References
 
 The hardening borrows directly from prior recursive-self-improvement work:

docs/visioned-self-improvement.md

@@ -156,6 +156,38 @@ are just one instantiation of the pattern; the backend prompt
 (`lib/backends/python-smt.ts`) states it for any subject so a timed-out heroic
 check never masquerades as "implementation broken".
 
+### 3.3a Source evidence — lifting the `[source-missing]` ceiling (`lib/evidence.ts`)
+
+A scene carries only a paper URL plus a short summary, so the source's *signature*
+structures (an exact memory map, an addressing function, an FSM) are absent.
+`reproduce` then finds two engineers each inventing a different valid version, and
+`amend` correctly refuses to fabricate, tagging the gap `[source-missing]`. No
+number of rounds can climb past this — the information genuinely isn't in the
+loop.
+
+`visually evidence <scene>` closes that gap: it fetches the scene's
+`metadata.info.sources` with **web tools** (the one tool-enabled step;
+`runClaudeStreaming({ tools: ['WebFetch','WebSearch'] })`), transcribes the
+technical sections to Markdown, and caches them under
+`~/.visually-3d/evidence/<id>/` — falling back to a checked-in `examples/<id>/`
+seed. `--refs` also searches GitHub for reference implementations, tagged
+secondary (`[ref-impl]`, never authoritative over the paper).
+
+**Invariant: evidence flows into `amend` ONLY.** reproduce's reverse-implementers
+never see it, because reproduce measures *"can you rebuild from the SPEC alone"* —
+handing them the paper would measure paper-completeness instead. The flow is:
+
+```
+evidence(paper.md) → amend QUOTES it ([paper]) → spec gains the real values
+                   → next reproduce grades the richer spec → divergence falls
+                   → reproducibility & fidelity rise
+```
+
+What evidence *cannot* fix: claims that need synthesis (resource/area/ATP/timing,
+e.g. "~50% hardware saved", LUT/FF ratios). A functional `python-smt` model
+cannot verify those at all; they stay `unverifiable` by construction and must not
+be counted as fidelity failures.
+
 ### 3.4 Automatic backend selection (`lib/backends/index.ts`)
 
 The verification substrate is chosen from **what the subject is**, so no manual

examples/ntt-fpga/index.json

@@ -0,0 +1,20 @@
+{
+  "id": "ntt-fpga",
+  "sources": [
+    {
+      "title": "CFNTT: Scalable Conflict-Free NTT Multiplication Architecture (TCHES 2022)",
+      "url": "https://tches.iacr.org/index.php/TCHES/article/view/9291"
+    },
+    {
+      "title": "CFNTT PDF (IACR TCHES)",
+      "url": "https://tches.iacr.org/index.php/TCHES/article/download/9291/8857"
+    },
+    {
+      "title": "DOI 10.46586/tches.v2022.i1.94-126",
+      "url": "https://doi.org/10.46586/tches.v2022.i1.94-126"
+    }
+  ],
+  "items": [
+    { "kind": "notes", "title": "Curated learnings (v1–v23)", "file": "notes.md" }
+  ]
+}

examples/ntt-fpga/notes.md

@@ -0,0 +1,53 @@
+# Curated learnings — ntt-fpga (CFNTT, TCHES 2022)
+
+Distilled from many `refine` rounds (v1–v23). This is honest about what is
+settled, what is still missing, and what this backend simply cannot grade —
+it is NOT a transcription of the paper. To fill the gaps below with the paper's
+real values, run `visually evidence ntt-fpga` (it fetches the source and writes
+`paper.md` next to this file), then `visually amend` will quote it.
+
+Source: "CFNTT: A Scalable Conflict-Free NTT Multiplication Architecture"
+(IACR TCHES 2022). See `index.json` for URLs.
+
+## Settled — matches the source, keep pinned in the spec
+The numeric/algebraic core is reproducible and the two independent reverse
+implementations agree on it. These belong in `metadata.spec` / `parts[].spec`
+with `[src]`/`[calc]` provenance and should not regress:
+- transform size N, modulus q, coefficient bit-width, number of butterfly units;
+- Barrett reduction constants (mu, k) and the negacyclic (psi-twisted) flavor;
+- psi / omega / N^-1 and the generator choice (derivable, machine-checked by the
+  arithmetic guard);
+- round-trip identity NTT∘INTT = id (proven on small N by the self-check).
+
+## Still missing — the paper's SIGNATURE structures (the real ceiling)
+These are the contributions a short summary omits, so two engineers each invent a
+different valid version → permanent divergence → reproducibility caps in the
+mid-70s. They are `[source-missing]` until the paper text is gathered:
+- the **conflict-free bank-index and bank-offset functions** (the Sec. III
+  headline result) — a reconstructed XOR-fold is NOT the paper's map and fails
+  per-stage operand-distinctness at large strides;
+- the **per-stage crossbar permutation σ_s** routing table;
+- the **twiddle-ROM address function** f(stage, block) under the reuse loop;
+- the **radix-4 full-transform index/stride schedule** and how the radix-2 lanes
+  reconfigure into radix-4 BUs per stage;
+- the **complete FSM** state list and transition table (the scene has carried two
+  contradictory FSM lists and an unwired DONE state — fix at the source);
+- the exact **Barrett correction** output bound / number of conditional
+  subtractions, and the internal coefficient ordering (natural vs bit-reversed).
+
+## Out of scope for python-smt — do NOT count these against fidelity
+These are RTL/synthesis/area-timing claims a functional Python+Z3 model can never
+verify. Record them in the spec as documentary `[src]` facts, but treat their
+`property_checks` as `unverifiable` by construction, not as failures:
+- "~50% butterfly hardware saved via symmetric operators";
+- ATP / resource ratios (LUT / FF / DSP / BRAM vs radix-2);
+- pipeline latency in cycles/lane and clock frequency;
+- the "33% fewer mults / 20% fewer add-sub" reductions (need a stated naive
+  baseline; verify only the per-BU operation count, which IS finite).
+
+## Process lessons (apply to any paper-derived scene)
+- A timed-out self-check is a HARNESS fault, not a wrong implementation — keep
+  default verification finite (see `lib/backends/python-smt.ts`).
+- `amend` must never fabricate a `[source-missing]` value; gather evidence first.
+- Resource/timing claims need a different backend (RTL synthesis) — until then,
+  reproducibility, not fidelity-on-hardware-claims, is the axis that can move.

lib/amend.ts

@@ -21,6 +21,7 @@ import { runClaudeStreaming } from './runner.js';
 import { extractScene, parseScene, validateScene, specCoverage } from './scene.js';
 import { repairArithmeticClaims } from './arith-audit.js';
 import { reproduce } from './reproduce.js';
+import { loadEvidence, evidenceExcerpt, type LoadedEvidence } from './evidence.js';
 
 interface AmendOpts {
   positional: string[];
@@ -185,9 +186,12 @@ export function hasFindings(report: any): boolean {
 }
 
 // Build the prompt that asks the model to MERGE verification findings into the
-// scene's spec substrate. Pure (no I/O) so it is unit-testable.
+// scene's spec substrate. Pure (no I/O) so it is unit-testable. When `ev` is
+// present, the gathered source evidence is injected so the model can QUOTE the
+// real paper/datasheet instead of guessing — the only way to pin the specific
+// system's structures and raise FIDELITY, not just reproducibility.
 // eslint-disable-next-line @typescript-eslint/no-explicit-any
-export function buildAmendPrompt(scene: any, report: any): string {
+export function buildAmendPrompt(scene: any, report: any, ev?: LoadedEvidence): string {
   type MissingField = { kind?: string; item?: string; where?: string };
   type VerifyFinding = { impl?: number; counterexample?: string };
   type ParamFidelity = { param?: string; reference_value?: string; impl_value?: string; match?: boolean; where?: string };
@@ -232,6 +236,21 @@ export function buildAmendPrompt(scene: any, report: any): string {
     null, 1,
   ).slice(0, 4000);
 
+  // Gathered ground-truth evidence (from `visually evidence`), if any. This is
+  // the authoritative transcription of the SOURCE — quoting it is how a
+  // [source-missing] gap finally gets filled with the paper's real value.
+  const excerpt = ev ? evidenceExcerpt(ev) : '';
+  const evidenceBlock = excerpt
+    ? `\nGATHERED SOURCE EVIDENCE (transcribed from the actual source by \`visually
+evidence\`; origin: ${ev?.origin}). This is AUTHORITATIVE — prefer it over the
+summary above. QUOTE concrete values from it to fill [source-missing] gaps, and
+tag them \`[paper]\` (or \`[ref-impl]\` if from a secondary reference
+implementation). Do NOT exceed what it states:
+\`\`\`markdown
+${excerpt}
+\`\`\`\n`
+    : '';
+
   return `You are closing the loop of a "visioned self-improvement" system. A scene
 descriptor doubles as the SPEC for a real system (a circuit, an algorithm, a
 machine, a building). Independent engineers tried to rebuild the system from
@@ -256,7 +275,7 @@ the SPECIFIC system's real values; do NOT invent values it does not support):
 \`\`\`json
 ${sourceBlock}
 \`\`\`
-
+${evidenceBlock}
 MISSING FIELDS the spec must now carry (each says which part's spec to write):
 ${missing || '(none reported)'}
 
@@ -301,14 +320,19 @@ RULES for choosing values:
    right or leave the operands symbolic. A self-inconsistent counterexample above
    (e.g. "spec N != computed N") MUST be fixed at its numeric source, not annotated.
 5. PROVENANCE — for each fact you commit, mark where it came from in \`spec.notes\`
-   using a tag: \`[src]\` quoted/derived from the SOURCE above, \`[conv]\` a
-   convention you chose to break a tie (not stated by the source), \`[calc]\` a
-   value you derived by computation. \`[conv]\` facts are reproducibility aids, NOT
-   claims about the real system — never present a guessed value as the source's.
-6. CEILING — if a missing field is genuinely NOT determinable from the source and
-   is NOT a free convention (it is specific architectural detail the scene simply
-   lacks), do not fabricate it. Record what is needed in \`spec.notes\` prefixed
-   \`[source-missing]\` so the gap is visible instead of papered over.
+   using a tag: \`[paper]\` quoted/derived from the GATHERED SOURCE EVIDENCE,
+   \`[ref-impl]\` from a secondary reference implementation in that evidence,
+   \`[src]\` from the source metadata above, \`[conv]\` a convention you chose to
+   break a tie (not stated by any source), \`[calc]\` a value you derived by
+   computation. \`[conv]\`/\`[ref-impl]\` facts are reproducibility aids, NOT
+   authoritative claims about the real system — never present a guessed or
+   secondary value as the paper's.
+6. CEILING — if a missing field is genuinely NOT determinable from any source
+   (neither the gathered evidence nor the metadata) and is NOT a free convention,
+   do not fabricate it. Record what is needed in \`spec.notes\` prefixed
+   \`[source-missing]\` so the gap is visible instead of papered over. If the
+   GATHERED EVIDENCE above now supplies a value a previous round marked
+   \`[source-missing]\`, REPLACE that note with the real value tagged \`[paper]\`.
 
 Existing part ids you may target: ${ids}
 
@@ -362,7 +386,14 @@ export async function amendScene(
   writeFileSync(path.join(dir, 'prev.json'), JSON.stringify(scene, null, 2));
   writeFileSync(path.join(dir, 'report.json'), JSON.stringify(report, null, 2));
 
-  const prompt = buildAmendPrompt(scene, report);
+  // Load any gathered source evidence so amend can QUOTE the real source. This
+  // is the ONLY place evidence enters the loop — reproduce's engineers never see
+  // it, so reproducibility keeps measuring the SPEC, not the paper.
+  const ev = loadEvidence(id);
+  if (ev.origin !== 'none') {
+    console.log(`  ↳ using source evidence (${ev.origin}) to ground the spec`);
+  }
+  const prompt = buildAmendPrompt(scene, report, ev);
   writeFileSync(path.join(dir, 'prompt.txt'), prompt);
   const { text } = await runClaudeStreaming({ prompt, model: opts.model, quiet: true });
   writeFileSync(path.join(dir, 'raw.txt'), text);