A Claude Code benchmark of tools that promise to cut the token consumption of agentic AI coding: serena vs graphify vs CodeGraph vs archex vs rtk vs Pare vs caveman vs Ponytail vs lazy-cat vs a no-tool baseline.
👉 Read the results: REPORT.md
Nine tools, four token layers, one rule: match the tool to the layer the task stresses.
Measured over an 8-task comprehension suite (+1 code-gen task) on a realistic ~30-file Python app,
totals vs the no-tool baseline (7 tools run for real, Ponytail + lazy-cat modeled; tokenizer
tiktoken o200k_base):
| Tool | Layer it targets | Δ tokens vs baseline |
|---|---|---|
| serena | code-read input (LSP symbols) | −66.1% |
| graphify | code-read input (code graph) | −65.6% |
| CodeGraph | code-read input (indexed graph) | −54.1% |
| archex | code-read input (retrieval bundles) | +15.5% (bundle overhead beats a small repo) |
| rtk | command-output input | −4.6% (but −36% on command tasks) |
| Pare | command-output input (structured MCP) | +0.2% (but −72% on the test run — the best command cell) |
| caveman | generated prose output | −0.7% (output-only; understated here) |
| Ponytail | generated code output | ~0% on comprehension; ~40% on code-gen* |
| lazy-cat | generated code output | ~0% on comprehension; ~45% on code-gen* |
| serena + rtk + caveman (stacked) | three layers at once | −69.6% |
| serena + Pare + caveman (stacked-pare) | three layers at once | −66.8% |
The tools target different layers and are complementary — except within a layer, where they
overlap (pick one per layer). The code-read incumbents split the wins by question type: serena on
broad comprehension, graphify on call-path tracing, CodeGraph on pinpoint caller lookups; archex
(built for large repos) never breaks even on this small one. On the command layer rtk wins overall
while Pare's structured pytest is the single best command result. On the code-output layer
Ponytail and lazy-cat land within noise of each other.
* Ponytail and lazy-cat have no headless CLI (behavioural plugins), so their figures are modeled and illustrative — see the caveat in REPORT.md. For methodology, per-task tables, the combination recommendations, and limitations, read the full report.
taskflow/ the dummy codebase under test (REST API + CLI, 13 passing tests)
benchmark/ the measurement harness
tasks.py 8 comprehension tasks + 1 code-gen task (D1) + per-tool specs
tools.py real artifact producers (read/rtk/graphify/codegraph/archex/caveman/...)
serena_client.py stdio MCP client driving serena
pare_client.py stdio MCP client driving Pare's servers (@paretools/*)
run_benchmark.py orchestrator -> results/
make_report.py renders REPORT.md from results
fixtures/caveman/ committed real caveman outputs (reproducible)
fixtures/ponytail/ vendored Ponytail rule text + committed modeled output
fixtures/lazycat/ vendored lazy-cat rule texts + committed modeled output
results/ environment.md, results.json, results.csv
token-saviour/ SKILL.md routing agents to the right tool by scenario (+ references/)
docs/superpowers/ design spec + implementation plan
REPORT.md the shareable write-up
uv venv .venv && uv pip install --python .venv/bin/python tiktoken pytest
uv tool install graphifyy
uv tool install --python 3.13 serena-agent
uv tool install archex && archex init && archex index
npm i -g @colbymchenry/codegraph
brew install fd ripgrep # Pare's search server shells out to fd/rg
# Pare's MCP servers (@paretools/python, @paretools/search) start via npx -y.
curl -fsSL https://raw.githubusercontent.com/rtk-ai/rtk/refs/heads/master/install.sh | sh
curl -fsSL https://raw.githubusercontent.com/JuliusBrussee/caveman/main/install.sh | bash
# Ponytail and lazy-cat are behavioural plugins (no CLI); their rule texts are vendored at
# benchmark/fixtures/ponytail/rules.md and benchmark/fixtures/lazycat/*.md.
PATH="$HOME/.local/bin:$PATH" .venv/bin/python -m benchmark.run_benchmark
.venv/bin/python -m benchmark.make_report