release: v1.108.68 - retrieval-regret loop (suggest_corrections / reflect)

The ranking-events telemetry ledger fed only the weight tuner; it also carries
a louder, unread signal: when retrieval failed and the agent had to re-ask.
This mines that regret and suggests config fixes.

- retrieval/regret.py::analyze_regret extracts six regret signals (requery
  churn, low confidence, thin/zero, ambiguous top, stale-at-query, vocabulary
  gap) as severity-ranked clusters. Pure read over the existing ledger loader;
  no new tables.
- tools/suggest_corrections.py fuses clusters + audit_agent_config + a dry-run
  WeightTuner proposal into prioritized, explainable corrections: CLAUDE.md
  routing/glossary lines as unified-diff previews, index-freshness hints, the
  weight proposal. Convergent signals are deduped.
- reflect CLI + suggest_corrections MCP tool (standard tier) + a one-line
  digest summary.

Charter: never writes a user file. Corrections are suggestions with evidence;
applying a patch is the user's keystroke. Only --apply-weights persists state,
and only to the tuning.jsonc sidecar. Honest hint when telemetry is off.

Tool count 83 -> 84; schema_baseline refreshed (core_compact unaffected).
New tests/test_suggest_corrections.py (13) incl. the charter no-write gate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: jgravelle

CHANGELOG.md

@@ -2,6 +2,36 @@
 
 All notable changes to jcodemunch-mcp are documented here.
 
+## [1.108.68] - 2026-06-21 - Retrieval-regret loop: suggest_corrections / reflect
+
+### Added
+
+- **A read-only feedback loop that mines retrieval regret and suggests config
+  fixes.** The ranking-events telemetry ledger already fed exactly one consumer
+  (the weight tuner); it also carries a louder, unread signal — when retrieval
+  failed and the agent had to re-ask. New `suggest_corrections` MCP tool +
+  `reflect` CLI surface it:
+  - **`retrieval/regret.py`** (`analyze_regret`) extracts six regret signals
+    over the ledger — re-query churn, low confidence, thin/zero results,
+    ambiguous top, stale-at-query, and identity-miss-rescued-by-semantic
+    (vocabulary gap) — as severity-ranked clusters. Pure read; no new tables.
+  - **`suggest_corrections`** fuses those clusters with `audit_agent_config`
+    (stale-config findings) and a dry-run `WeightTuner` proposal into prioritized,
+    explainable corrections: CLAUDE.md routing/glossary lines rendered as
+    **unified-diff previews**, index-freshness hints, and the weight proposal.
+  - **`reflect` CLI** (`jcodemunch-mcp reflect [repo] [--all] [--apply-weights]
+    [--json]`) prints the same as a human report.
+  - One-line **digest** integration: "N regret cluster(s) this window; top: …".
+- **Charter: never writes a user file.** Every correction is a *suggestion* with
+  its evidence (event counts + example queries); applying a patch is the user's
+  keystroke. The only state the loop may persist is the ranking-weights sidecar
+  (`tuning.jsonc`), behind an explicit `apply_weights`/`--apply-weights`. A test
+  asserts the targeted CLAUDE.md is byte-unchanged after a run.
+- Degrades honestly when `perf_telemetry_enabled` is off (the ledger only exists
+  then): returns an explicit hint, never fabricated corrections. New tool lives
+  in the `standard` tier (alongside `tune_weights`/`analyze_perf`). Tool count
+  82 -> 83. New `tests/test_suggest_corrections.py` (12).
+
 ## [1.108.67] - 2026-06-21 - Demote the language enum under compact_schemas
 
 ### Changed

CLAUDE.md

@@ -1,6 +1,7 @@
 # jcodemunch-mcp — Project Brief
 
 ## Current State
+- **Version:** 1.108.68 — Retrieval-regret loop (`suggest_corrections` tool + `reflect` CLI). The `ranking_events` telemetry ledger fed only `WeightTuner`; it also carries an unread, louder signal — when retrieval failed and the agent had to re-ask. New `retrieval/regret.py::analyze_regret` mines SIX regret signals over the ledger (re-query churn, low confidence, thin/zero result, ambiguous top, stale-at-query, vocabulary gap = identity-miss-rescued-by-semantic) as severity-ranked clusters (pure read via the existing `token_tracker.ranking_db_query` loader; no new tables). `tools/suggest_corrections.py` fuses clusters + `audit_agent_config` stale-config findings + a dry-run `WeightTuner` proposal into prioritized, explainable corrections: CLAUDE.md routing/glossary lines as **unified-diff previews** (difflib), index-freshness hints (reads `get_watch_status` service.active), the weight proposal. **Charter: NEVER writes a user file** — suggestions only, applying a patch is the user's keystroke; the only persistable state is the `tuning.jsonc` weights sidecar behind explicit `apply_weights`. Honest no-telemetry hint when `perf_telemetry_enabled` off (ledger only exists then). Clean-room "suggest, don't write" is the differentiator ([[feedback_jcm_read_only_charter]]). Surfaces: `suggest_corrections` MCP tool (standard tier, alongside tune_weights/analyze_perf), `reflect` CLI (`reflect [repo] [--all] [--apply-weights] [--json]`), one-line digest integration. Tool count 83→84 (full) / standard +1. `schema_baseline.json` refreshed (core_compact still 3956, suggest_corrections is standard not core). New `tests/test_suggest_corrections.py` (12) incl. the charter no-write gate. See memory `project_retrieval_regret_prd`.
 - **Version:** 1.108.67 — Demote the `language` enum under `compact_schemas`. The `language` filter's full LANGUAGE_REGISTRY enum (~76 values, ~200 tokens of mechanical names an agent already knows) is dropped to a plain `type:string` filter under compact (capability preserved — the tool accepts any language string; unknown → empty match); the full surface keeps the enum. Brings `core_compact` back under the §10 ≤4000 line (4162→**3956**, ~44 margin) after 1.108.55-66 tool additions pushed it to 4146; also trims `standard_compact` (16339→16149). New `_COMPACT_DEMOTE_ENUM_PARAMS` frozenset (keyed by PARAM name, so search_symbols/search_ast/etc. all benefit) + an enum-demotion pass alongside the existing `_COMPACT_STRIP_PARAMS` loop in `_build_tools_list`. `schema_baseline.json` refreshed to honest live counts (the wholesale `capture_schema_baseline.py` run is now CORRECT — it writes 3956, the very thing that made it wrong at v1.108.66 when it wrote 4146). New `test_compact_demotes_language_enum_keeps_capability` pins it (enum gone under compact, present in full). Closes the open follow-up from the Counter release. No behavior change on the full tier.
 - **Version:** 1.108.66 — The Counter: adaptive tool surface (`order` / `menu` / `route`). Collapses the ~83-tool footprint behind a 3-tool front door to cut the per-turn schema tax + dispatch dilution, without removing any capability. New `JCODEMUNCH_TOOL_SURFACE` env / `tool_surface` config: `counter` makes `list_tools` return only the front door + `_ALWAYS_PRESENT_TOOLS`; any other value (default `full`) is **byte-identical** to prior behavior (front-door tools hidden, still callable). **`order(action,args)`** re-enters the normal `call_tool` pipeline after a charter gate (read-only by default; index/session state-changers need `allow_state_change=true`; exec/file-write verbs refused unconditionally — a forward-looking tripwire, the suite ships none). **`menu(query?,limit?)`** = idf-weighted catalog search. **`route(task,repo?,execute?)`** = curated intent map (`_INTENT_RULES`) + catalog-search fallback → ranked actions w/ arg templates; `execute=true` recommends AND dispatches in one call; recommends `assemble_task_context`/`plan_turn` for context intents (honest mapper, NOT a plan_turn reskin — plan_turn recommends symbols, not tools). New `counter.py` (pure logic, no server import) + 4 surgical `server.py` edits (front-door globals after `_UNDISABLEABLE_TOOLS`; surface-collapse in `_build_tools_list`; handlers before `call_tool`; early front-door branch inside `call_tool`). `tests/test_counter.py` (19) incl. a charter CI gate (no catalog action trips the exec/write tripwire) + a STATE_CHANGING-in-catalog drift guard. Full surface stays 83 tools; front door appears only under `tool_surface=counter`. Pre-existing schema-budget drift (unrelated, proven via stash) fixed by a surgical `schema_baseline.json` refresh of the two breached keys (`core_full`→5097, `standard_full`→17360); core_compact LIVE 4146 is over the 4000 v2 design intent (prior-release bloat, separate trim-or-relax follow-up). See `docs/prd-adaptive-tool-surface.md` + memory `project_tool_router_prd`.
 - **Version:** 1.108.65 — Surface the composition tools (`assemble_task_context` / `plan_turn`) in agent-facing docs. Answers the "these code-index tools work in isolation" critique (a competitor's launch claim, see memory `reference_competitor_atelier`): the suite already HAS an orchestration layer — `assemble_task_context` (one-call, intent-classified, single-budget capsule) and `plan_turn` (confidence-routed opening move) — it just wasn't legible. `assemble_task_context` was absent from every agent-facing surface; now added to `_CLAUDE_MD_POLICY` (init.py — the injected CLAUDE.md/Cursor/Windsurf policy) "Opening move" as the one-call shortcut, plus a new README "What you get → One-call task orchestration" subsection + "Why agents need this" bullet. `skills.py` SKILL.md already had a "Task orchestration" section (unchanged). Docs + one policy constant; **no tool behavior change**. `test_claude_md_policy.py` REQUIRED_POLICY_TERMS pins `assemble_task_context` so visibility can't regress (the line is full-tier only, so `_filter_policy_for_tools` drops it under core/standard exactly like `plan_turn` — existing filter tests unaffected, 22 pass). Legibility fix, not capability chase ([[feedback_clean_room_innovation]]).
@@ -65,7 +66,7 @@
 - **INDEX_VERSION:** 16
 - **Tests:** 4432 passed, 10 skipped (1.108.25 — full count varies by optional-dep availability)
 - **Python:** >=3.10
-- **Tool count:** 83 in `full` (front door hidden); `tool_surface=counter` exposes a 3-tool front door (`order`/`menu`/`route`) instead
+- **Tool count:** 84 in `full` (front door hidden; +1 v1.108.68 `suggest_corrections`); `tool_surface=counter` exposes a 3-tool front door (`order`/`menu`/`route`) instead
 
 ## Key Files
 ```
@@ -108,6 +109,7 @@ src/jcodemunch_mcp/
     lsp_bridge.py      # LSP bridge — opt-in compiler-grade call graph resolution via pyright/gopls/ts-language-server/rust-analyzer; LSPServer lifecycle, LSPBridge multi-server manager, enrich_call_graph_with_lsp() + enrich_dispatch_edges() (interface/trait dispatch resolution)
   retrieval/
     signal_fusion.py   # Weighted Reciprocal Rank (WRR) fusion: lexical + structural + similarity + identity channels
+    regret.py          # (v1.108.68) analyze_regret: mines the ranking_events ledger for SIX retrieval-regret signals (requery_churn/low_confidence/thin_result/ambiguous_top/stale_at_query/vocabulary_gap) as severity-ranked clusters. Pure read via token_tracker.ranking_db_query; no new tables. Consumed by suggest_corrections + the digest one-liner
   summarizer/
     batch_summarize.py # 3-tier: Anthropic > Gemini > OpenAI-compat > signature fallback
   tools/
@@ -151,7 +153,8 @@ src/jcodemunch_mcp/
     get_untested_symbols.py   # get_untested_symbols: find functions with no test-file reachability (import graph + name matching)
     search_ast.py             # search_ast: cross-language AST pattern matching; 10 preset anti-patterns + custom mini-DSL (call:, string:, comment:, nesting:, loops:, lines:); enriched with symbol context
     winnow_symbols.py         # winnow_symbols: multi-axis constraint-chain query; AND-intersects kind/language/name/file/complexity/decorator/calls/summary/churn in one round trip; ranks by importance/complexity/churn/name
-    audit_agent_config.py    # audit_agent_config: token waste audit for CLAUDE.md, .cursorrules, etc.; cross-refs against index
+    audit_agent_config.py    # audit_agent_config: token waste audit for CLAUDE.md, .cursorrules, etc.; cross-refs against index. Reused by suggest_corrections (_discover_files / _fuzzy_suggest / stale-config findings)
+    suggest_corrections.py   # (v1.108.68) Retrieval-regret synthesis: fuses regret.analyze_regret clusters + audit_agent_config + WeightTuner dry-run into SUGGESTED corrections (routing/vocabulary/index-freshness/stale-config) with difflib unified-diff CLAUDE.md previews. Read-only charter — never writes a user file; apply_weights touches only tuning.jsonc. Honest no-telemetry hint
     analyze_perf.py          # analyze_perf: per-tool latency telemetry (p50/p95/max/error_rate) + cache hit-rate; reads in-memory session ring or persistent telemetry.db (opt-in via perf_telemetry_enabled); compare_release="X" loads benchmarks/token_baselines/vX.json and adds baseline_diff
   runtime/
     __init__.py          # Trace ingestion package (Phases 0-5): re-exports redact_trace_record, resolve_to_symbol_id, parse_otel_file, ingest_otel_file, OtelSpan, parse_sql_log_file, ingest_sql_log_file, SqlQueryRecord, parse_stack_log_file, ingest_stack_log_file, StackEvent, StackFrame, VALID_SOURCES = {'otel','sql_log','stack_log','apm'}
@@ -208,7 +211,8 @@ src/jcodemunch_mcp/
 | `hook-subagent-start` | SubagentStart hook: inject condensed repo orientation for spawned agents (reads JSON stdin) |
 | `whatsnew` | Refresh README recency block + write `whatsnew.json` from `CHANGELOG.md` (release flow) |
 | `receipt` | Token-economy ledger from Claude transcripts — modeled tokens-saved + dollar value at Sonnet/Opus/Haiku rates; `--explain`, `--export csv\|json`, `--days`, `--model` |
-| `digest` | Agent stand-up briefing — composes since-last-session delta + risk surface + dead-code candidates; tracks per-repo last-seen SHA at `~/.code-index/digest_state/`; also exposed as MCP tool `digest` |
+| `digest` | Agent stand-up briefing — composes since-last-session delta + risk surface + dead-code candidates; tracks per-repo last-seen SHA at `~/.code-index/digest_state/`; also exposed as MCP tool `digest`. v1.108.68 adds a one-line retrieval-regret summary when the ledger has clusters |
+| `reflect` | (v1.108.68) Surface retrieval regret as SUGGESTED config corrections — `reflect [repo] [--project-path] [--window-days N] [--all] [--apply-weights] [--json]`. Thin CLI over the `suggest_corrections` tool; read-only (only `--apply-weights` writes, and only the tuning.jsonc sidecar) |
 | `health` | Print `get_repo_health` JSON to stdout (includes six-axis radar). For CI/scripting; `--radar-only` for just the radar sub-field. Used by the v1.88.0 health-radar GitHub Action |
 | `file-risk` | Print per-symbol risk JSON for a file (composite score + four-axis breakdown). Used by the v0.2.0 VS Code risk-density gutter |
 | `observatory build\|init` | Public OSS code-health observatory pipeline — clones, indexes, scores a configured repo list; writes static HTML + RSS + JSON to an output dir. v1.90.0; CI repo-id bug fixed in v1.90.1. Live at https://jgravelle.github.io/jcodemunch-observatory/ |

README.md

@@ -299,6 +299,8 @@ Every retrieval result now ships with three machine-readable health signals so a
 
 The `tune_weights` tool reads the persistent ranking ledger and learns per-repo retrieval weights (saved to `~/.code-index/tuning.jsonc`). `check_embedding_drift` pins a 16-string canary to detect silent provider model changes. `benchmarks/replay/` provides a CI-friendly retrieval-quality regression gate (nDCG/MRR/Recall) that every release runs against.
 
+The `suggest_corrections` tool (and the `reflect` CLI) close the loop: they mine the same ranking ledger for **retrieval regret** — where retrieval failed and the agent had to re-ask — and return a prioritized, explainable set of *suggested* fixes (a CLAUDE.md routing or glossary line as a unified-diff preview, an index-freshness hint, a stale-config finding, a dry-run weight proposal). It is read-only by design: it suggests a patch and shows you the diff; applying it is your keystroke, never the server's. Requires `perf_telemetry_enabled` (it has a ledger to read only then) and returns an honest hint when off.
+
 ### Local-first speed
 
 Indexes are stored locally for fast repeated access.

benchmarks/schema_baseline.json

@@ -1,8 +1,8 @@
 {
   "core_compact": 3956,
   "core_full": 5097,
-  "standard_compact": 16149,
-  "standard_full": 17360,
-  "full_compact": 17459,
-  "full_full": 18670
+  "standard_compact": 16407,
+  "standard_full": 17618,
+  "full_compact": 17717,
+  "full_full": 18928
 }

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "jcodemunch-mcp"
-version = "1.108.67"
+version = "1.108.68"
 description = "Token-efficient MCP server for source code exploration via tree-sitter AST parsing"
 readme = "README.md"
 requires-python = ">=3.10"