release: v1.108.140 — first gold corpus: find_implementations channel accuracy, measured

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: jgravelle

CHANGELOG.md

@@ -2,6 +2,41 @@
 
 All notable changes to jcodemunch-mcp are documented here.
 
+## [1.108.140] - 2026-07-18 - the first gold corpus: channel accuracy, measured
+
+### Added
+- **An authored gold corpus for `find_implementations` channel accuracy**
+  (`benchmarks/goldset/`). Every implementation relation — declared
+  subclasses, duck-typed conformers, decorator-registered handlers — and
+  every deliberate false-positive trap (a module-homonym base class, same-
+  name-different-domain methods like electrical/legal `charge`, substring
+  decorator matches like a `/user_created_report` route against a
+  `user_created` event) is labeled in `gold.json` with a per-pair rationale,
+  so ground truth is exact by construction.
+- **A reproducible measurement harness** (`benchmarks/goldset/measure.py`):
+  snapshots the corpus to a temp dir, indexes it, runs
+  `find_implementations` per gold target, joins each surfaced implementation
+  to its label, and writes per-channel precision/recall to
+  `benchmarks/provenance/channel_accuracy.json`. First measured round: AST
+  0.833 precision (the homonym trap), duck 0.6, decorator 0.6 — recall 1.0
+  across all channels. Notably the shipped priors survive contact with
+  measurement: 0.85 declared vs 0.833 measured for AST, 0.65 vs 0.6 for
+  duck, and the 0.45 decorator prior is *more* conservative than measured.
+- **The measurement re-runs in CI** (`tests/test_channel_accuracy.py`): the
+  committed artifact must equal a live re-measurement and the corpus content
+  hash, so the numbers cannot drift from the reproducible run — a corpus or
+  channel change forces a deliberate artifact regeneration.
+- **Registry and responses carry the measured reference.** Each heuristic
+  channel's provenance entry gains `measured_ref` (precision/recall, corpus,
+  artifact path) beside its declared ranking prior, surfaced in
+  `_meta.confidence_provenance`; `MEASURED` gains
+  `implementation_channel_accuracy`. The operating constants deliberately
+  stay `declared` — they are ranking priors, and recalibrating them to a
+  small-n corpus would trade honest stability for false precision; the
+  artifact states its scope (authored-pattern discrimination, not
+  in-the-wild base rates). LSP/SCIP channels carry no `measured_ref` — they
+  are the ground-truth side of the comparison.
+
 ## [1.108.139] - 2026-07-18 - measured provenance rides the reporting surfaces
 
 ### Added

CLAUDE.md

@@ -1,6 +1,7 @@
 # jcodemunch-mcp — Project Brief
 
 ## Current State
+- **Version:** 1.108.140 — **Phase C of the self-attesting contract: first gold corpus + measured channel accuracy.** New `benchmarks/goldset/`: authored Python corpus (14 files: payments protocol + subclasses, duck conformers incl. test-double/adapter, event-bus decorator handlers) with DELIBERATE traps (module-homonym `PaymentProvider` base → AST FP; electrical/legal `charge` → duck FPs; `/user_created_report` route + `cleanup_user_created_exports` task substring-matches → decorator FPs); `gold.json` labels every pair with rationale + expected channel. `measure.py`: snapshot-to-tmp (dodges the git-identity contains-path trap — NEVER index a subtree of the host repo in place), index, run find_implementations per target, join to labels; UNLABELED surfaced pair = harness error (corpus must stay complete). **Measured round 1: ast P=0.833/R=1.0 (tp5 fp1), duck P=0.6/R=1.0 (tp3 fp2), decorator P=0.6/R=1.0 (tp3 fp2)** → `benchmarks/provenance/channel_accuracy.json` (carries corpus_sha256 + scope note). `tests/test_channel_accuracy.py` (7) RE-RUNS the measurement live in CI + hash-pins the corpus + chains registry↔artifact. provenance.py: `measured_ref` per heuristic channel beside the DECLARED operating constant (deliberately NOT recalibrated — small-n would jitter ranking for false precision; declared priors held up: 0.85/0.833, 0.65/0.6, 0.45 more conservative than 0.6); `MEASURED["implementation_channel_accuracy"]`; `channel_provenance()` surfaces measured_ref (schema already permits it — additionalProperties). LSP/SCIP stay declared, no measured_ref (they're the ground-truth side). README provenance section extended. NO INDEX_VERSION / tool-count / schema change. **Corpus growth path: add languages (TS/Go), grow n per channel, then a DELIBERATE recalibration release flips constants to measured values gated by replay. jMRI normative section still LAST.**
 - **Version:** 1.108.139 — **Phase B of the self-attesting contract: measured provenance rides the REPORTING surfaces (never the hot path).** New `provenance.measured_provenance()` (fresh-copy of `MEASURED` + contract note) attached at 4 read-the-numbers surfaces: `receipt --export json` (`provenance` block in `render_json`), `receipt` text methodology footer (cites `benchmarks/provenance/measured.json` + "drift-guarded"), `get_session_stats` (`savings_provenance` — tool wrapper in `tools/get_session_stats.py`, storage layer untouched), `jcodemunch_guide` (`provenance` key beside the snippet; verified the guide response stays JSON so the block survives — no custom encoder). NO CLI-flag change (the duplicate receipt subparser surface in server.py NOT touched — output-only). Strengthens the Tim/QPoint verifiability story: an exported ledger now carries its own receipts. New `tests/test_v1_108_139.py` (6). NO INDEX_VERSION / tool-count / schema change. **OPEN PARITY ITEM ([[feedback_suite_parity_for_agent_contracts]]): jdoc/jdata session-stats should ship the same `savings_provenance` shape (needs sibling provenance registries + their own measured artifacts — do NOT copy jcm's numbers into them).** Phase C next: gold corpus per channel (Loc-Bench feeds it); jMRI normative section LAST.
 - **Version:** 1.108.138 — **Confidence provenance, Phase A of the self-attesting retrieval contract (F-15 #1 from the 2026-07-18 sweep; competitor unnamed in shipped artifacts).** New `retrieval/provenance.py`: `CONFIDENCE_PROVENANCE` registry — every confidence constant carries `basis` ∈ {`declared` (engineering prior, honestly labeled), `measured` (backed by committed artifact `benchmarks/provenance/measured.json`)}. Load-bearing rule: **a prior is NEVER presented as a measurement** — Phase B/C (graduating find_implementations tiers to measured) requires building a gold corpus FIRST (Loc-Bench feeds this); jjg accepted this phased order after my reservations (measurement-validity, fixture-decay drift-guard policy, jMRI-obligation optics). Registry entries: find_implementations 5 channels (all declared), negative_evidence_threshold 0.5, exact_seed_verdict_floor 1.0. `MEASURED` = the two artifacts jcm actually has: token_reduction 99.6%/15 runs/cl100k_base (METHODOLOGY.md) + replay retrieval quality (self_v1_75_0 golden, ndcg/mrr/recall 1.0, CI-gated). ⚠ The 0.992/0.895 Hit@10/MRR figure floated during the sweep is OCTOCODE'S benchmark (versus.php grid cell), NOT ours — never cite it as jcm's. `find_implementations` attaches `_meta.confidence_provenance` (via `channel_provenance()`). Published JSON Schemas in `schemas/` (retrieval-verdict, confidence-provenance, ranked-context-response); `tests/test_provenance.py` (14) drift-guards registry↔code↔artifact↔underlying-benchmark in all directions + jsonschema-validates live responses. README gains "Confidence provenance" subsection. NO new tool / INDEX_VERSION / schema-budget change (JSON-only `_meta` addition; no custom encoder for find_implementations so compact path unaffected). **Phase B next: surface measured munch-bench/methodology numbers as in-band provenance; Phase C: gold corpus per channel; jMRI normative section LAST (SHOULD-with-conformance-path, after suite parity is demonstrable).**
 - **Version:** 1.108.137 — **Source-shaped exact seeding in `get_ranked_context` (clean-room F-15; competitor unnamed in shipped artifacts).** New `retrieval/query_shape.py` classifies query tokens into identifier shapes (qualified `A::B`/`a.b` w/ file-extension-tail exclusion, CamelCase, snake_case, dunders; ≤3 tokens, prose never matches); the DEFAULT ranking path resolves each shaped token via a cached `name_map` (built beside the BM25 corpus) to exact-name symbols (case-sensitive→insensitive; qualified parent narrows via id/parent substring; PageRank ranks ties; include_kinds/scope honored; caps 3/token, 5/query) and PINS them ahead of the ranked tail (`_exact_seed_symbols` + pinning block after `scored.sort`). Items carry `match_channel:"exact_name"`; `_meta.query_shape={source_shaped,exact_seeded}`; verdict best_score floored at 1.0 (raw-BM25 scale vs 0.5 threshold) when seeded, and a seeded result bypasses the negative-evidence early return (`if not raw_scores and not seeded`). **Pure-prose queries byte-identical; `fusion=True` path unchanged (its WRR identity channel already covers exact).** Motivation: the exact-lookup lane is our one measured weak spot (definition MRR 0.70 vs 0.87 in the fair rerun). ALSO: (1) **`openWorldHint` on every tool** — new `_OPEN_WORLD_TOOLS` frozenset (index_repo, index_folder/index_file/summarize_repo [cloud summarizer opt-in], embed_repo/check_embedding_drift/test_summarizer, order/route front door) annotated True, everything else False, in `_apply_readonly_annotations`; machine-checkable no-network claim beside readOnlyHint. (2) **`PRAGMA journal_size_limit=64MB`** in sqlite_store `_PRAGMAS` + embedding_store + parse_cache (WAL starvation under long-lived readers — a failure class three ecosystem peers patched the same month). New `tests/test_v1_108_137.py` (19). NO INDEX_VERSION / tool-count / schema change. Origin: 2026-07-18 top-15 competitor sweep (memory `project_competitor_sweep_2026_07_18`); F-15 #1 (measured-provenance receipts / self-attesting contract) NOT started.

README.md

@@ -128,9 +128,9 @@ is a byte the agent doesn't pay to read.
 <!-- WHATSNEW:START -->
 #### What's new
 
+- **[v1.108.140](https://github.com/jgravelle/jcodemunch-mcp/releases/tag/v1.108.140)** (2026-07-18) — the first gold corpus: channel accuracy, measured
 - **[v1.108.139](https://github.com/jgravelle/jcodemunch-mcp/releases/tag/v1.108.139)** (2026-07-18) — measured provenance rides the reporting surfaces
 - **[v1.108.138](https://github.com/jgravelle/jcodemunch-mcp/releases/tag/v1.108.138)** (2026-07-18) — confidence provenance: every number states its basis
-- **[v1.108.137](https://github.com/jgravelle/jcodemunch-mcp/releases/tag/v1.108.137)** (2026-07-18) — source-shaped exact seeding in ranked context
 <!-- WHATSNEW:END -->
 
 ![License](https://img.shields.io/badge/license-dual--use-blue)
@@ -329,6 +329,8 @@ The `suggest_corrections` tool (and the `reflect` CLI) close the loop: they mine
 
 Every confidence constant the suite emits traces to a stated basis: **`measured`** (backed by a committed, reproducible benchmark artifact — `benchmarks/provenance/measured.json`, drift-guarded in CI so the constants and the artifact can never silently diverge) or **`declared`** (an engineering prior, honestly labeled as exactly that). `find_implementations` responses carry the per-channel basis in `_meta.confidence_provenance`, and the response contracts themselves are published as JSON Schemas in [`schemas/`](schemas/) (`retrieval-verdict`, `confidence-provenance`, `ranked-context-response`) so CI pipelines and agents can validate responses mechanically. A prior is never presented as a measurement: a `declared` value graduates to `measured` only when a gold-labeled corpus backs it, and a build that claims otherwise fails.
 
+The first gold corpus is in: `benchmarks/goldset/` is an authored implementation-pattern corpus (declared subclasses, duck-typed conformers, decorator-registered handlers — plus deliberate false-positive traps: module-homonym base classes, same-name-different-domain methods, substring decorator matches), fully labeled with per-pair rationale. `benchmarks/goldset/measure.py` re-runs `find_implementations` against it and CI asserts the committed results (`benchmarks/provenance/channel_accuracy.json`) match the live measurement — the numbers literally cannot drift from the reproducible run. Each resolution channel's registry entry now carries its `measured_ref` (precision/recall on the corpus) beside the declared ranking prior, and `_meta.confidence_provenance` surfaces both. Scope is stated in the artifact: authored-pattern discrimination, not in-the-wild base rates.
+
 ### Local-first speed
 
 Indexes are stored locally for fast repeated access.

benchmarks/goldset/corpus/events/handlers.py

@@ -0,0 +1,16 @@
+from events.registry import on
+
+
+@on("user_created")
+def send_welcome_email(payload):
+    return {"sent": payload["email"]}
+
+
+@on("user_created")
+def provision_workspace(payload):
+    return {"workspace": payload["user_id"]}
+
+
+@on("user_created")
+def audit_log_user(payload):
+    return {"audited": payload["user_id"]}

benchmarks/goldset/corpus/events/protocol.py

@@ -0,0 +1,5 @@
+def user_created(payload):
+    """Event protocol: fired once when a user account is created.
+
+    Handlers registered for this event receive the payload dict.
+    """

benchmarks/goldset/corpus/events/registry.py

@@ -0,0 +1,25 @@
+"""Tiny event bus used by the handler scenarios."""
+
+_HANDLERS = {}
+
+
+def on(event_name):
+    """Register a handler for a named event."""
+    def _wrap(fn):
+        _HANDLERS.setdefault(event_name, []).append(fn)
+        return fn
+    return _wrap
+
+
+def route(path):
+    """Register a web route (unrelated to the event bus)."""
+    def _wrap(fn):
+        return fn
+    return _wrap
+
+
+def task(name):
+    """Register a background task."""
+    def _wrap(fn):
+        return fn
+    return _wrap

benchmarks/goldset/corpus/events/web.py

@@ -0,0 +1,13 @@
+from events.registry import route, task
+
+
+@route("/admin/user_created_report")
+def show_user_created_report():
+    """Admin page LISTING created users — not an event handler."""
+    return "<table></table>"
+
+
+@task("cleanup_user_created_exports")
+def cleanup_user_created_exports():
+    """Nightly file cleanup — not an event handler."""
+    return 0

benchmarks/goldset/corpus/payments/applepay_adapter.py

@@ -0,0 +1,5 @@
+class ApplePayAdapter:
+    """Adapts the ApplePay SDK to the payment protocol (duck-typed on purpose)."""
+
+    def charge(self, amount_cents, currency):
+        return {"provider": "applepay", "amount": amount_cents, "currency": currency}

benchmarks/goldset/corpus/payments/base.py

@@ -0,0 +1,11 @@
+"""The payment protocol every gateway implements."""
+
+
+class PaymentProvider:
+    """Charge and refund money on a customer's payment method."""
+
+    def charge(self, amount_cents, currency):
+        raise NotImplementedError
+
+    def refund(self, charge_id):
+        raise NotImplementedError

benchmarks/goldset/corpus/payments/legacy.py

@@ -0,0 +1,8 @@
+from payments.stripe_impl import StripeGateway
+
+
+class LegacyBridgeGateway(StripeGateway):
+    """Bridges the pre-2020 billing system onto the Stripe gateway."""
+
+    def refund(self, charge_id):
+        return {"provider": "legacy-bridge", "refunded": charge_id}