Skip to content

vagkaratzas/token-consumption-benchmark

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

22 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

token-consumption-benchmark

A Claude Code benchmark of tools that promise to cut the token consumption of agentic AI coding: serena vs graphify vs CodeGraph vs archex vs rtk vs Pare vs caveman vs Ponytail vs lazy-cat vs a no-tool baseline.

👉 Read the results: REPORT.md

TL;DR

Nine tools, four token layers, one rule: match the tool to the layer the task stresses. Measured over an 8-task comprehension suite (+1 code-gen task) on a realistic ~30-file Python app, totals vs the no-tool baseline (7 tools run for real, Ponytail + lazy-cat modeled; tokenizer tiktoken o200k_base):

Tool Layer it targets Δ tokens vs baseline
serena code-read input (LSP symbols) −66.1%
graphify code-read input (code graph) −65.6%
CodeGraph code-read input (indexed graph) −54.1%
archex code-read input (retrieval bundles) +15.5% (bundle overhead beats a small repo)
rtk command-output input −4.6% (but −36% on command tasks)
Pare command-output input (structured MCP) +0.2% (but −72% on the test run — the best command cell)
caveman generated prose output −0.7% (output-only; understated here)
Ponytail generated code output ~0% on comprehension; ~40% on code-gen*
lazy-cat generated code output ~0% on comprehension; ~45% on code-gen*
serena + rtk + caveman (stacked) three layers at once −69.6%
serena + Pare + caveman (stacked-pare) three layers at once −66.8%

The tools target different layers and are complementary — except within a layer, where they overlap (pick one per layer). The code-read incumbents split the wins by question type: serena on broad comprehension, graphify on call-path tracing, CodeGraph on pinpoint caller lookups; archex (built for large repos) never breaks even on this small one. On the command layer rtk wins overall while Pare's structured pytest is the single best command result. On the code-output layer Ponytail and lazy-cat land within noise of each other.

* Ponytail and lazy-cat have no headless CLI (behavioural plugins), so their figures are modeled and illustrative — see the caveat in REPORT.md. For methodology, per-task tables, the combination recommendations, and limitations, read the full report.

Repository layout

taskflow/            the dummy codebase under test (REST API + CLI, 13 passing tests)
benchmark/           the measurement harness
  tasks.py             8 comprehension tasks + 1 code-gen task (D1) + per-tool specs
  tools.py             real artifact producers (read/rtk/graphify/codegraph/archex/caveman/...)
  serena_client.py     stdio MCP client driving serena
  pare_client.py       stdio MCP client driving Pare's servers (@paretools/*)
  run_benchmark.py     orchestrator -> results/
  make_report.py       renders REPORT.md from results
  fixtures/caveman/    committed real caveman outputs (reproducible)
  fixtures/ponytail/   vendored Ponytail rule text + committed modeled output
  fixtures/lazycat/    vendored lazy-cat rule texts + committed modeled output
results/             environment.md, results.json, results.csv
token-saviour/       SKILL.md routing agents to the right tool by scenario (+ references/)
docs/superpowers/    design spec + implementation plan
REPORT.md            the shareable write-up

Reproduce

uv venv .venv && uv pip install --python .venv/bin/python tiktoken pytest
uv tool install graphifyy
uv tool install --python 3.13 serena-agent
uv tool install archex && archex init && archex index
npm i -g @colbymchenry/codegraph
brew install fd ripgrep   # Pare's search server shells out to fd/rg
# Pare's MCP servers (@paretools/python, @paretools/search) start via npx -y.
curl -fsSL https://raw.githubusercontent.com/rtk-ai/rtk/refs/heads/master/install.sh | sh
curl -fsSL https://raw.githubusercontent.com/JuliusBrussee/caveman/main/install.sh | bash
# Ponytail and lazy-cat are behavioural plugins (no CLI); their rule texts are vendored at
# benchmark/fixtures/ponytail/rules.md and benchmark/fixtures/lazycat/*.md.

PATH="$HOME/.local/bin:$PATH" .venv/bin/python -m benchmark.run_benchmark
.venv/bin/python -m benchmark.make_report

About

Benchmark different tools that promise token consumption decrease for agentic ai prompts

Resources

License

Stars

4 stars

Watchers

0 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors

Languages