Skip to content

Commit 28c1a5c

Browse files
NickCirvclaude
andcommitted
feat(bench): session-level token bench with honest recall-recovery model
Adds bench/session-level.ts (ADR-0002) — the proof instrument for the research-loop-elimination thesis. Replays machine-generated investigation traces (Grep(symbol) → Read(caller files), top-K most-referenced symbols, no hand-picking) through the REAL shipped handlers, counting total tool-call tokens with vs without engram over the same fixed sequence. CRITICAL honesty correction (an adversarial methodology audit caught this and it was right): engram's packet is a SUBSET of the raw tool output — the grep packet is a file list, the read packet is a structural summary that provably does NOT contain the investigated call site. So a fixed-sequence bench is an OPTIMISTIC UPPER BOUND, not the 'conservative lower bound' the first ADR draft claimed. When the packet is insufficient the agent re-fetches raw content ON TOP of the packet. So the bench does NOT report one number. It reports the session reduction as a function of P (fraction of intercepted calls where the agent still needs raw content): engram_cost = intercepted ? packet + P*baseline : baseline. On engram's own repo: P=0 → 87.3% (ceiling), P=0.5 → 41.3%, P=1 → -4.7% (overhead), BREAK-EVEN P = 94.9% (engram nets positive unless the agent falls back to raw on >95% of calls). The high break-even is the defensible headline. Word-boundary symbol baseline (apples-to-apples); grep-only stopword control proves 0% (no harm/no padding); leads with median + range, not the weighted aggregate. ADR-0002 rewritten to call it an optimistic upper bound + document the P-model and break-even. The recall-sufficiency is workload-dependent (high on structural/discovery, low when exact lines needed) — the same bimodal split as the live W1.9 bench. Audit evidence: tsc clean; full suite 1082/1082 (bench not imported by src); e2e run verified (break-even math hand-checked); adversarial methodology audit (verdict INFLATED → fixed) + code-review (clean). Leak-scan 0 hits. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 7cb6ef6 commit 28c1a5c

3 files changed

Lines changed: 541 additions & 0 deletions

File tree

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,3 +22,4 @@ bench/results/real-world-2*.md
2222
# Hyperframes scratch/render directories (per-run intermediates)
2323
docs/demos/work-*/
2424
dist-extensions/
25+
bench/results/session-level-2*.md

0 commit comments

Comments
 (0)