Commit 28c1a5c
feat(bench): session-level token bench with honest recall-recovery model
Adds bench/session-level.ts (ADR-0002) — the proof instrument for the
research-loop-elimination thesis. Replays machine-generated investigation
traces (Grep(symbol) → Read(caller files), top-K most-referenced symbols, no
hand-picking) through the REAL shipped handlers, counting total tool-call
tokens with vs without engram over the same fixed sequence.
CRITICAL honesty correction (an adversarial methodology audit caught this and
it was right): engram's packet is a SUBSET of the raw tool output — the grep
packet is a file list, the read packet is a structural summary that provably
does NOT contain the investigated call site. So a fixed-sequence bench is an
OPTIMISTIC UPPER BOUND, not the 'conservative lower bound' the first ADR draft
claimed. When the packet is insufficient the agent re-fetches raw content ON
TOP of the packet.
So the bench does NOT report one number. It reports the session reduction as a
function of P (fraction of intercepted calls where the agent still needs raw
content): engram_cost = intercepted ? packet + P*baseline : baseline. On
engram's own repo: P=0 → 87.3% (ceiling), P=0.5 → 41.3%, P=1 → -4.7%
(overhead), BREAK-EVEN P = 94.9% (engram nets positive unless the agent falls
back to raw on >95% of calls). The high break-even is the defensible headline.
Word-boundary symbol baseline (apples-to-apples); grep-only stopword
control proves 0% (no harm/no padding); leads with median + range, not the
weighted aggregate.
ADR-0002 rewritten to call it an optimistic upper bound + document the P-model
and break-even. The recall-sufficiency is workload-dependent (high on
structural/discovery, low when exact lines needed) — the same bimodal split as
the live W1.9 bench.
Audit evidence: tsc clean; full suite 1082/1082 (bench not imported by src);
e2e run verified (break-even math hand-checked); adversarial methodology audit
(verdict INFLATED → fixed) + code-review (clean). Leak-scan 0 hits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent 7cb6ef6 commit 28c1a5c
3 files changed
Lines changed: 541 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| 25 | + | |
0 commit comments