feat(resolver): Phase A — cross-provider displacement (dedup + blend-rank)

Turn the context resolver from a concatenator into a displacement engine: it now
de-duplicates context ACROSS providers and ranks by a blended confidence+priority
score, so adding providers can only SHRINK the packet, never grow it. The honest
"spine" win — less context, ranked + deduped — in the regimes that matter
(multi-provider overlap, sub-agent fan-out). No cost claims (W1.9): counters
attest "redundancy eliminated" / "tokens displaced", never "saved".

- src/providers/dedup.ts (NEW): FNV-1a shingle Jaccard dedup (section + line),
  monotonic (only removes content), + applyDisplacement() wiring helper. Pure, DI-tested.
- src/providers/ranking.ts (NEW): blendScore = 0.6*confidence + 0.4*priorityWeight;
  a high-confidence low-priority result can now out-rank a weak high-priority one.
- src/providers/resolver.ts: blendSort -> applyDisplacement -> never-worse gate;
  RichPacket gains redundantTokensEliminated + tokensDisplaced (candidate-set
  redundancy removed pre-budget, NOT final-packet delta — documented inline).
- docs/design/phaseA-displacement-resolver.md: design spec (14 cases + 3 adversarial).

Audit: tsc 0 . 1205/1205 tests (45 new) . 2 independent supervisors
(honesty=SHIP/0 P0; correctness fixes applied: pure applyDisplacement +
resolver-wiring tests + streaming-doc).

Notes:
- AST/structure special-case KEPT as a deterministic fast path (spec proposed
  folding it in; keeping it avoids a regression if their text is <0.85 similar).
  General dedup covers every other provider pair.
- Streaming (resolveRichPacketStreaming / SSE) is deliberately NON-displacement
  (incremental emission can't see the full set); the hook path IS fully displaced.
  Buffer-then-displace for streaming is a tracked follow-up.
- .gitignore: added *.pem/*.key after gitleaks caught an ed25519 private key
  accidentally generated in this repo during a go-live runbook; key moved out of
  the repo, never committed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: NickCirv

.gitignore

@@ -24,3 +24,9 @@ docs/demos/work-*/
 dist-extensions/
 bench/results/session-level-2*.md
 .claude/
+
+# Private keys / secrets — NEVER commit. (An ed25519 .pem was once generated
+# here by accident during a go-live runbook; gitleaks caught it pre-commit.)
+*.pem
+*.key
+.engram-secrets/

docs/design/phaseA-displacement-resolver.md

@@ -0,0 +1,102 @@
+# Phase A — Resolver: Concatenator → Displacement Engine
+
+**Status:** Design spec (read-only research) · **Date:** 2026-06-20 · **Author:** engram-phaseA-researcher
+**Scope:** `src/providers/resolver.ts` only. No code edited. Honesty: DISPLACEMENT (aggregate→rank→DISCARD), never "aggregate more"; counter attests *displaced*/*redundant*, never "saved". W1.9: net-of-caching ≈ 0 — no cost claims.
+
+## 0. Premise corrections (file:line evidence)
+
+- **Paths wrong in brief.** Files are `src/providers/resolver.ts`, `src/providers/mcp-client.ts`, `src/providers/types.ts` — NOT `src/intercept/providers/`. Line numbers below are the real ones.
+- **`confidence` default 0.75** is real: `mcp-client.ts:293` `tool.confidence ?? 0.75`.
+- **Priority array** is real: `types.ts:163` `PROVIDER_PRIORITY` (9 entries, `engram:ast` … `engram:lsp`).
+- **PARTIAL DEDUP ALREADY EXISTS** (brief said "ZERO" — half-wrong): `resolver.ts:174-179` drops `engram:structure` when `engram:ast` succeeded. It is a single hard-coded provider-pair special case, NOT general semantic dedup. Phase A generalises it; do not claim it's net-new.
+- **`ProviderResult` shape** (`types.ts:33-42`): `{ provider, content, confidence, cached }`. **No symbol id, no path, no line range** — only free-text `content`. This is load-bearing: dedup can only operate on normalized `content`, not structured keys. Symbol-level dedup would require a provider-contract change (out of Phase A scope; noted as risk).
+
+## 1. Where duplication arises
+
+The Read handler resolves enrichment providers (`read.ts:182-188`): `engram:mistakes, engram:git, mempalace, context7, obsidian` (structure already in `fileCtx.summary`). MCP code-graph plugins join via `getAllProviders()` (`resolver.ts:145`). Cross-provider dupes:
+- **structure ↔ MCP code-graph** — same function signature/callers text from `engram:structure` and an MCP graph server (only AST↔structure is currently handled, 174-179).
+- **mistakes ↔ git** — same commit/line cited as "broke here" (mistakes) and "churned here" (git).
+- **mempalace ↔ obsidian** — same decision note synced to both stores.
+- **context7 ↔ MCP docs plugin** — same library doc snippet.
+
+Each arrives as a *section blob* of `content` (`mcp-client.ts:248` joins multiple tool outputs with `\n\n`). Dedup granularity is therefore the **section** and, finer, the **line/shingle** within a section.
+
+## 2. Design — three changes
+
+All three slot into `resolveRichPacket` between collection (`resolver.ts:172`) and assembly (`resolver.ts:212`), in this order: **(b) blend rank → (a) dedup → (c) size gate**. Rank first so dedup's keep-decision honours blended score; gate last over the final packet.
+
+### (a) Dedup pass — content-normalized shingles (cheap first)
+New `dedupResults(results: ProviderResult[]): { kept; redundantTokens }` called at `resolver.ts:185` (replacing the AST/structure special-case 174-179, which becomes one general rule).
+
+- **Normalize:** lowercase, collapse whitespace runs→single space, NFKC unicode fold, strip section header/label noise. → `norm(content)`.
+- **Shingle:** w=5 word k-shingles → `Set<hash>` (FNV-1a 32-bit, reuse `estimateTokens` neighbour util style; no new dep).
+- **Compare:** Jaccard over shingle sets. `J ≥ 0.85` ⇒ duplicate (tune via tests; 0.85 tolerates reformatting, rejects merely-same-topic).
+- **Sub-section dedup:** within the *kept* result, drop individual lines whose 5-shingle set has `J ≥ 0.9` against a line already emitted by a higher-ranked result. This is where most real displacement happens (partial overlap, not whole-section).
+- **Keep rule (tiebreak):** on a dup cluster keep the member with **highest blended score** (from (b)); on score tie keep **highest raw `confidence`**; on confidence tie keep **first by priority index**. Deterministic.
+- **Accounting:** sum `estimateTokens` of every discarded section + discarded line ⇒ `redundantTokens` (for §4).
+- **Embeddings: deferred, NOT used.** Justification gate (must hold to adopt later): only if a stress corpus shows shingle-Jaccard misses ≥15% of true semantic dupes (paraphrase with <0.85 lexical overlap) AND the false-keep tokens exceed the embedding compute cost. No data yet ⇒ hashing only. Recording this so a future session doesn't silently add a model dependency.
+
+### (b) Blend confidence across providers (invert the comparator)
+Replace the comparator at `resolver.ts:197-204` (priority-primary, confidence-tiebreak) with a blended score:
+
+```
+priorityWeight(p) = (P - idx(p)) / P        // idx via PROVIDER_PRIORITY; unknown ⇒ idx=99 ⇒ ~0
+score(r) = 0.6 * r.confidence + 0.4 * priorityWeight(r.provider)
+sort desc by score; tie → higher confidence; tie → lower priority idx (deterministic)
+```
+
+- `P = PROVIDER_PRIORITY.length`. Weights chosen so a **low-priority high-confidence** result (e.g. MCP graph, idx 8, conf 0.95 → 0.6·0.95+0.4·0.11=0.61) **beats** a **high-priority low-confidence** result (engram:git idx 6, conf 0.3 → 0.6·0.3+0.4·0.33=0.31). Priority still contributes (it's the 0.4 term) so it remains a real signal, not discarded.
+- `boostByMistakes` (`resolver.ts:192`, 458-477) stays — it adjusts `confidence` *before* scoring, so mistake-touching results rank up across the blend, not just within a tier. This is an intended upgrade over today's tie-only effect.
+- `0.6/0.4` weights are config-exposed (`config.rankConfidenceWeight`, default 0.6) so they're tunable without a code change.
+
+### (c) Never-worse size gate (reuse ADR-0007 pattern)
+ADR-0007's grep gate (`grep.ts:93 rawGrepFloorTokens`, gate at `read.ts:166-172`) compares `estimateTokens(packet)` vs a conservative raw floor and PASSTHROUGHs if not strictly smaller. Reuse the *shape*, not the rg floor.
+
+- **Baseline = pre-displacement packet tokens.** Compute `baselineTokens = sum(estimateTokens(r.content))` over the budgeted-but-not-yet-deduped results (the tokens engram would emit today). Compute `finalTokens` after dedup+rank+assembly.
+- **Gate:** `if (finalTokens >= baselineTokens) ` → emit the **smaller** of {deduped packet, today's packet}; never emit a packet larger than today's. Because dedup only removes, `finalTokens ≤ baselineTokens` always holds for the *content*; the gate guards against header/label churn re-inflating it (see §5 risk). Mirrors `read.ts:170 if (summaryTokens >= fileTokens) return PASSTHROUGH`.
+- Honest framing: gate proves packet is **smaller-or-equal** while rank keeps top-relevance first.
+
+## 3. Hook points (resolver.ts)
+| Step | Line today | Change |
+|---|---|---|
+| collect results | 165-172 | unchanged |
+| AST/structure special-case | 174-179 | **remove** — folded into general `dedupResults` |
+| per-provider budget | 185 | unchanged (runs before dedup) |
+| boostByMistakes | 192 | unchanged (feeds confidence into blend) |
+| **blend sort (b)** | 197-204 | **replace comparator** with blended `score` |
+| **dedup (a)** | new, after sort (~205) | `{kept, redundantTokens} = dedupResults(sorted)` |
+| assembly loop | 212-226 | iterate `kept` |
+| **size gate (c)** | new, after 228 | compare final vs baseline; emit smaller |
+| return RichPacket | 241-247 | add `displacedTokens`, `redundantTokens` |
+
+## 4. engram-counter instrumentation
+Extend `RichPacket` (`resolver.ts:112-123`) + counter audit block:
+- `redundantTokensEliminated` — tokens removed by dedup (§2a accounting). Attest "redundancy eliminated".
+- `tokensDisplaced` — `baselineTokens − finalTokens` (gate's measured delta). Attest "tokens displaced".
+- **Never** a `saved`/`cost` field. Counter consumes these two integers; aggregate is sum, never a % cost claim.
+
+## 5. Where this risks INCREASING tokens (flagged)
+1. **Header/label re-inflation** — if dedup drops a whole section, `providerCount` shrinks but per-section labels stay; net could rise if dedup removes tiny sections and keeps big ones. Mitigation: gate (c) is the backstop; refuse any packet ≥ baseline.
+2. **Sub-section line-dedup leaving orphans** — removing interior lines can leave a dangling header with no body. Mitigation: drop a section whose body falls to 0 lines post-dedup.
+3. **Blend promoting a verbose low-priority result** that pushes a terse high-priority one past budget at `resolver.ts:214`. Ranking changes *order*, not size — but order changes *which* sections survive the budget cut. Net tokens unchanged (budget caps it), but relevance could drop; covered by the "low-priority-high-confidence beats" test asserting the displaced result is genuinely more relevant, not just longer.
+
+## 6. TDD test list (ordered) — `tests/providers/resolver.test.ts`
+1. **dedup collapses identical sections** — two providers, same `content` → one section, `redundantTokensEliminated == estimateTokens(dup)`.
+2. **dedup collapses whitespace/unicode variants** — same text, NFKC + collapsed spaces differ → still deduped (J≥0.85).
+3. **dedup keeps highest blended score on a cluster** — dup pair, assert the kept one is the higher-score member, not first-arrival.
+4. **sub-section line dedup** — partial overlap (3 of 5 shared lines) → shared lines removed once, both sections survive minus overlap.
+5. **blend: low-priority high-confidence beats high-priority low-confidence** — MCP graph(idx8,0.95) ordered above engram:git(idx6,0.30).
+6. **blend: priority still contributes** — equal confidence → higher-priority first (the 0.4 term decides).
+7. **boostByMistakes feeds blend** — mistake-touching low-priority result outranks a non-touching equal-confidence one.
+8. **size gate refuses larger packet** — synthetic case where labels inflate → assert emitted packet ≤ baseline (never larger).
+9. **size gate: deduped is strictly smaller** → emits deduped, `tokensDisplaced > 0`.
+10. **empty results** → returns null (parity with `resolver.ts:172`).
+11. **malformed result (empty content / non-string)** → skipped, no throw, others survive.
+12. **never-worse under N providers** — property test, 2..8 random providers w/ random overlap → `finalTokens ≤ baselineTokens` ALWAYS.
+13. **AST/structure regression** — old special-case behaviour (drop structure when AST present) still holds via general dedup (guard against regressing 174-179).
+14. **counter fields populated** — `redundantTokensEliminated` + `tokensDisplaced` present and consistent (`displaced == baseline − final`).
+
+### 3 adversarial scenarios
+- **A1 — near-dup-not-dup (J just under threshold):** two results J≈0.84 (same topic, different specifics). Assert BOTH kept (dedup must not eat distinct context). Tunes the 0.85 boundary.
+- **A2 — hostile MCP plugin returns 50KB single section, confidence 1.0:** per-provider budget (185) truncates first; blend can't let it starve others; gate proves no net increase. Assert other providers still present + packet ≤ baseline.
+- **A3 — adversarial label collision:** plugin sets its `content` to mimic another provider's section verbatim to get itself kept. Assert keep-rule uses blended score+confidence (not content identity) so the impersonator doesn't displace the genuine higher-confidence source.

src/providers/dedup.ts

@@ -0,0 +1,196 @@
+/**
+ * dedup.ts — Phase A displacement: collapse redundant context ACROSS providers.
+ *
+ * The resolver fans out to many providers (built-in + plugin + MCP). Two of them
+ * routinely return the SAME thing (structure ↔ an MCP code-graph; mistakes ↔ git;
+ * mempalace ↔ obsidian). Concatenating both INFLATES the packet — the opposite of
+ * what a context spine should do. This module turns concatenation into
+ * DISPLACEMENT: given results already ranked highest-first, it keeps the
+ * highest-ranked member of each duplicate cluster and discards the rest, reporting
+ * how many tokens it eliminated.
+ *
+ * Honesty (W1.9): this only ever REMOVES content — it can never grow the packet.
+ * The accounting is "redundancy eliminated", never "cost saved".
+ *
+ * Pure + dependency-free: shingled Jaccard over normalized content (FNV-1a hashing,
+ * no embeddings — see docs/design/phaseA-displacement-resolver.md §2a for the
+ * data-gated justification on why hashing first). The token estimator is injected
+ * so this stays unit-testable without the resolver's internals.
+ */
+
+import type { ProviderResult } from "./types.js";
+
+/** k-shingle width (words). 5 balances reformatting-tolerance vs topic-collision. */
+export const SHINGLE_W = 5;
+/** Section-level Jaccard ≥ this ⇒ duplicate section. Tolerates reformatting. */
+export const SECTION_THRESHOLD = 0.85;
+/** Line-level Jaccard ≥ this ⇒ duplicate line (stricter — partial overlap). */
+export const LINE_THRESHOLD = 0.9;
+
+/** FNV-1a 32-bit. Fast, no deps, good enough for shingle identity. */
+export function fnv1a(s: string): number {
+  let h = 0x811c9dc5;
+  for (let i = 0; i < s.length; i++) {
+    h ^= s.charCodeAt(i);
+    h = Math.imul(h, 0x01000193);
+  }
+  return h >>> 0;
+}
+
+/**
+ * Normalize for comparison: NFKC-fold, lowercase, strip markdown headers +
+ * bracketed labels (so `[engram:git]`-style section noise doesn't drive matches),
+ * collapse all whitespace. Comparison-only — the original content is preserved.
+ */
+export function normalize(content: string): string {
+  return content
+    .normalize("NFKC")
+    .toLowerCase()
+    .replace(/^#+\s*/gm, "") // markdown headers
+    .replace(/\[[^\]\n]*\]/g, " ") // [label] noise
+    .replace(/\s+/g, " ")
+    .trim();
+}
+
+/** Word-level k-shingle set of normalized text. Short text → one whole-text shingle. */
+export function shingles(normText: string, w: number = SHINGLE_W): Set<number> {
+  const set = new Set<number>();
+  const words = normText.split(" ").filter(Boolean);
+  if (words.length === 0) return set;
+  if (words.length < w) {
+    set.add(fnv1a(words.join(" ")));
+    return set;
+  }
+  for (let i = 0; i + w <= words.length; i++) {
+    set.add(fnv1a(words.slice(i, i + w).join(" ")));
+  }
+  return set;
+}
+
+/** Jaccard similarity of two shingle sets. Both empty ⇒ 1; one empty ⇒ 0. */
+export function jaccard(a: Set<number>, b: Set<number>): number {
+  if (a.size === 0 && b.size === 0) return 1;
+  if (a.size === 0 || b.size === 0) return 0;
+  const [small, large] = a.size < b.size ? [a, b] : [b, a];
+  let inter = 0;
+  for (const x of small) if (large.has(x)) inter++;
+  return inter / (a.size + b.size - inter);
+}
+
+export interface DedupOutput {
+  /** Survivors, in the input (ranked) order, with line-level dupes stripped. */
+  readonly kept: ProviderResult[];
+  /** Tokens removed by section + line dedup (for engram-counter attestation). */
+  readonly redundantTokens: number;
+}
+
+/**
+ * Collapse cross-provider redundancy. `ranked` MUST be pre-sorted highest-first
+ * (the resolver blends confidence+priority before calling this) so the kept member
+ * of every cluster is the most relevant. Two passes:
+ *   1. SECTION dedup — drop a whole result whose shingles ≥ SECTION_THRESHOLD vs
+ *      any already-kept result.
+ *   2. LINE dedup — within survivors (in rank order), drop individual lines whose
+ *      shingles ≥ LINE_THRESHOLD vs a line already emitted by a higher-ranked
+ *      result. A result whose body falls to nothing is dropped entirely.
+ * Total-monotonic: output content ⊆ input content, so it can never grow the packet.
+ */
+export function dedupResults(
+  ranked: readonly ProviderResult[],
+  estimateTokens: (s: string) => number,
+  opts: { sectionThreshold?: number; lineThreshold?: number } = {}
+): DedupOutput {
+  const sectionT = opts.sectionThreshold ?? SECTION_THRESHOLD;
+  const lineT = opts.lineThreshold ?? LINE_THRESHOLD;
+
+  const kept: ProviderResult[] = [];
+  const keptShingles: Set<number>[] = [];
+  const emittedLineShingles: Set<number>[] = [];
+  let redundantTokens = 0;
+
+  for (const r of ranked) {
+    // Defensive: a malformed provider result must never break dedup.
+    if (!r || typeof r.content !== "string") continue;
+    const norm = normalize(r.content);
+    if (norm.length === 0) {
+      kept.push(r); // nothing to compare; pass through untouched
+      continue;
+    }
+
+    const sh = shingles(norm);
+    if (keptShingles.some((ks) => jaccard(sh, ks) >= sectionT)) {
+      redundantTokens += estimateTokens(r.content); // whole section is redundant
+      continue;
+    }
+
+    // Line-level dedup within this (kept) result.
+    const lines = r.content.split("\n");
+    const survivors: string[] = [];
+    let droppedLineTokens = 0;
+    for (const line of lines) {
+      const ln = normalize(line);
+      if (ln.length === 0) {
+        survivors.push(line); // preserve blank/structural lines
+        continue;
+      }
+      const lsh = shingles(ln);
+      if (emittedLineShingles.some((es) => jaccard(lsh, es) >= lineT)) {
+        droppedLineTokens += estimateTokens(line);
+        continue;
+      }
+      survivors.push(line);
+      emittedLineShingles.push(lsh);
+    }
+
+    const newContent = survivors.join("\n");
+    if (normalize(newContent).length === 0) {
+      // Body fully dedup'd away → drop the orphaned section entirely.
+      redundantTokens += estimateTokens(r.content);
+      continue;
+    }
+
+    redundantTokens += droppedLineTokens;
+    keptShingles.push(sh);
+    kept.push(newContent === r.content ? r : { ...r, content: newContent });
+  }
+
+  return { kept, redundantTokens };
+}
+
+export interface Displacement {
+  /** Results to assemble (deduped), or the original ranked set if the
+   *  never-worse backstop tripped. */
+  readonly kept: ProviderResult[];
+  /** Tokens removed by dedup (redundancy eliminated). */
+  readonly redundantTokens: number;
+  /** Content tokens displaced before budgeting (baseline − final). Always ≥ 0. */
+  readonly tokensDisplaced: number;
+  /** Candidate-set content tokens BEFORE dedup (what engram would emit today). */
+  readonly baselineTokens: number;
+}
+
+/**
+ * The resolver's displacement step as a pure, testable unit: measure baseline,
+ * dedup, measure final, apply a never-worse backstop, and report honest counters.
+ * `ranked` MUST be pre-sorted highest-first. Because dedup is monotonic the gate
+ * never trips in practice — it's a belt-and-braces guard against any future
+ * regression that could inflate content. Returns counters labelled "displaced" /
+ * "redundancy eliminated" — never "saved" (W1.9).
+ */
+export function applyDisplacement(
+  ranked: readonly ProviderResult[],
+  estimateTokens: (s: string) => number
+): Displacement {
+  const baselineTokens = ranked.reduce(
+    (sum, r) => sum + (typeof r?.content === "string" ? estimateTokens(r.content) : 0),
+    0
+  );
+  const { kept, redundantTokens } = dedupResults(ranked, estimateTokens);
+  const finalTokens = kept.reduce((sum, r) => sum + estimateTokens(r.content), 0);
+
+  if (finalTokens > baselineTokens) {
+    // Should be impossible (dedup only removes) — fall back to the original set.
+    return { kept: [...ranked], redundantTokens: 0, tokensDisplaced: 0, baselineTokens };
+  }
+  return { kept, redundantTokens, tokensDisplaced: baselineTokens - finalTokens, baselineTokens };
+}