|
1 | 1 | # jcodemunch-mcp -- Token Efficiency Benchmark |
2 | 2 |
|
3 | 3 | **Tokenizer:** `cl100k_base` (tiktoken) |
4 | | -**Workflow:** `search_symbols` (top 5) + `get_symbol_source` x 3 |
| 4 | +**Workflow:** `search_symbols` (top 5) + `get_symbol` x 3 |
5 | 5 | **Baseline:** all source files concatenated (minimum for "open every file" agent) |
| 6 | +**Run:** 2026-07-23, jcodemunch-mcp v1.108.163 (repos re-indexed same day) |
6 | 7 |
|
7 | 8 | ## expressjs/express |
8 | 9 |
|
9 | 10 | | Metric | Value | |
10 | 11 | |--------|-------| |
11 | | -| Files indexed | **165** | |
12 | | -| Symbols extracted | **181** | |
13 | | -| Baseline tokens (all files) | **137,978** | |
| 12 | +| Files indexed | **172** | |
| 13 | +| Symbols extracted | **182** | |
| 14 | +| Baseline tokens (all files) | **143,355** | |
14 | 15 |
|
15 | 16 | | Query | Baseline tokens | jMunch tokens | Reduction | Ratio | |
16 | 17 | |-------|---------------------:|-------------------:|----------:|------:| |
17 | | -| `router route handler` | 137,978 | 886 | **99.4%** | 155.7x | |
18 | | -| `middleware` | 137,978 | 1,008 | **99.3%** | 136.9x | |
19 | | -| `error exception` | 137,978 | 859 | **99.4%** | 160.6x | |
20 | | -| `request response` | 137,978 | 872 | **99.4%** | 158.2x | |
21 | | -| `context bind` | 137,978 | 993 | **99.3%** | 139.0x | |
22 | | -| **Average** | — | — | **99.4%** | **150.1x** | |
| 18 | +| `router route handler` | 143,355 | 1,342 | **99.1%** | 106.8x | |
| 19 | +| `middleware` | 143,355 | 1,284 | **99.1%** | 111.6x | |
| 20 | +| `error exception` | 143,355 | 1,167 | **99.2%** | 122.8x | |
| 21 | +| `request response` | 143,355 | 1,193 | **99.2%** | 120.2x | |
| 22 | +| `context bind` | 143,355 | 209 | **99.9%** | 685.9x | |
| 23 | +| **Average** | — | — | **99.3%** | **229.5x** | |
23 | 24 |
|
24 | 25 | <details><summary>Query detail (search + fetch tokens, latency)</summary> |
25 | 26 |
|
26 | 27 | | Query | Search tokens | Fetch tokens | Hits fetched | Search ms | |
27 | 28 | |-------|-----------------:|------------------:|------------------:|---------------:| |
28 | | -| `router route handler` | 381 | 505 | 3 | 6.2 | |
29 | | -| `middleware` | 370 | 638 | 3 | 0.2 | |
30 | | -| `error exception` | 362 | 497 | 3 | 7.1 | |
31 | | -| `request response` | 372 | 500 | 3 | 0.5 | |
32 | | -| `context bind` | 372 | 621 | 3 | 0.3 | |
| 29 | +| `router route handler` | 543 | 799 | 3 | 33.8 | |
| 30 | +| `middleware` | 495 | 789 | 3 | 1.2 | |
| 31 | +| `error exception` | 493 | 674 | 3 | 17.3 | |
| 32 | +| `request response` | 527 | 666 | 3 | 2.3 | |
| 33 | +| `context bind` | 209 | 0 | 0 | 5.6 | |
33 | 34 |
|
34 | 35 | </details> |
35 | 36 |
|
36 | 37 | ## fastapi/fastapi |
37 | 38 |
|
38 | 39 | | Metric | Value | |
39 | 40 | |--------|-------| |
40 | | -| Files indexed | **951** | |
41 | | -| Symbols extracted | **5,325** | |
42 | | -| Baseline tokens (all files) | **699,425** | |
| 41 | +| Files indexed | **1,000** | |
| 42 | +| Symbols extracted | **6,722** | |
| 43 | +| Baseline tokens (all files) | **823,784** | |
43 | 44 |
|
44 | 45 | | Query | Baseline tokens | jMunch tokens | Reduction | Ratio | |
45 | 46 | |-------|---------------------:|-------------------:|----------:|------:| |
46 | | -| `router route handler` | 699,425 | 1,199 | **99.8%** | 583.3x | |
47 | | -| `middleware` | 699,425 | 1,643 | **99.8%** | 425.7x | |
48 | | -| `error exception` | 699,425 | 873 | **99.9%** | 801.2x | |
49 | | -| `request response` | 699,425 | 4,439 | **99.4%** | 157.6x | |
50 | | -| `context bind` | 699,425 | 1,016 | **99.9%** | 688.4x | |
51 | | -| **Average** | — | — | **99.8%** | **531.2x** | |
| 47 | +| `router route handler` | 823,784 | 1,608 | **99.8%** | 512.3x | |
| 48 | +| `middleware` | 823,784 | 1,837 | **99.8%** | 448.4x | |
| 49 | +| `error exception` | 823,784 | 1,231 | **99.9%** | 669.2x | |
| 50 | +| `request response` | 823,784 | 4,689 | **99.4%** | 175.7x | |
| 51 | +| `context bind` | 823,784 | 3,103 | **99.6%** | 265.5x | |
| 52 | +| **Average** | — | — | **99.7%** | **414.2x** | |
52 | 53 |
|
53 | 54 | <details><summary>Query detail (search + fetch tokens, latency)</summary> |
54 | 55 |
|
55 | 56 | | Query | Search tokens | Fetch tokens | Hits fetched | Search ms | |
56 | 57 | |-------|-----------------:|------------------:|------------------:|---------------:| |
57 | | -| `router route handler` | 464 | 735 | 3 | 131.4 | |
58 | | -| `middleware` | 460 | 1,183 | 3 | 0.3 | |
59 | | -| `error exception` | 383 | 490 | 3 | 0.7 | |
60 | | -| `request response` | 430 | 4,009 | 3 | 12.0 | |
61 | | -| `context bind` | 402 | 614 | 3 | 0.4 | |
| 58 | +| `router route handler` | 605 | 1,003 | 3 | 776.0 | |
| 59 | +| `middleware` | 556 | 1,281 | 3 | 1.5 | |
| 60 | +| `error exception` | 514 | 717 | 3 | 1.8 | |
| 61 | +| `request response` | 548 | 4,141 | 3 | 16.8 | |
| 62 | +| `context bind` | 556 | 2,547 | 3 | 1.9 | |
62 | 63 |
|
63 | 64 | </details> |
64 | 65 |
|
65 | 66 | ## gin-gonic/gin |
66 | 67 |
|
67 | 68 | | Metric | Value | |
68 | 69 | |--------|-------| |
69 | | -| Files indexed | **98** | |
70 | | -| Symbols extracted | **1,489** | |
71 | | -| Baseline tokens (all files) | **187,018** | |
| 70 | +| Files indexed | **109** | |
| 71 | +| Symbols extracted | **1,502** | |
| 72 | +| Baseline tokens (all files) | **192,800** | |
72 | 73 |
|
73 | 74 | | Query | Baseline tokens | jMunch tokens | Reduction | Ratio | |
74 | 75 | |-------|---------------------:|-------------------:|----------:|------:| |
75 | | -| `router route handler` | 187,018 | 1,151 | **99.4%** | 162.5x | |
76 | | -| `middleware` | 187,018 | 1,130 | **99.4%** | 165.5x | |
77 | | -| `error exception` | 187,018 | 818 | **99.6%** | 228.6x | |
78 | | -| `request response` | 187,018 | 1,083 | **99.4%** | 172.7x | |
79 | | -| `context bind` | 187,018 | 1,436 | **99.2%** | 130.2x | |
80 | | -| **Average** | — | — | **99.4%** | **171.9x** | |
| 76 | +| `router route handler` | 192,800 | 1,568 | **99.2%** | 123.0x | |
| 77 | +| `middleware` | 192,800 | 1,726 | **99.1%** | 111.7x | |
| 78 | +| `error exception` | 192,800 | 1,099 | **99.4%** | 175.4x | |
| 79 | +| `request response` | 192,800 | 1,530 | **99.2%** | 126.0x | |
| 80 | +| `context bind` | 192,800 | 1,634 | **99.2%** | 118.0x | |
| 81 | +| **Average** | — | — | **99.2%** | **130.8x** | |
81 | 82 |
|
82 | 83 | <details><summary>Query detail (search + fetch tokens, latency)</summary> |
83 | 84 |
|
84 | 85 | | Query | Search tokens | Fetch tokens | Hits fetched | Search ms | |
85 | 86 | |-------|-----------------:|------------------:|------------------:|---------------:| |
86 | | -| `router route handler` | 440 | 711 | 3 | 43.5 | |
87 | | -| `middleware` | 371 | 759 | 3 | 5.7 | |
88 | | -| `error exception` | 345 | 473 | 3 | 0.8 | |
89 | | -| `request response` | 387 | 696 | 3 | 0.8 | |
90 | | -| `context bind` | 391 | 1,045 | 3 | 7.9 | |
| 87 | +| `router route handler` | 522 | 1,046 | 3 | 155.2 | |
| 88 | +| `middleware` | 441 | 1,285 | 3 | 15.4 | |
| 89 | +| `error exception` | 464 | 635 | 3 | 1.8 | |
| 90 | +| `request response` | 671 | 859 | 3 | 1.7 | |
| 91 | +| `context bind` | 490 | 1,144 | 3 | 18.1 | |
91 | 92 |
|
92 | 93 | </details> |
93 | 94 |
|
94 | | ---- |
95 | | - |
96 | 95 | ## Real-world A/B test: naming audit task (2026-03-18) |
97 | 96 |
|
98 | 97 | 50-iteration test by @Mharbulous comparing JCodeMunch vs native tools (Grep/Glob/Read) on a real Vue 3 + Firebase production codebase. Full report: [ab-test-naming-audit-2026-03-18.md](ab-test-naming-audit-2026-03-18.md) |
@@ -132,9 +131,9 @@ Raw data: https://gist.github.com/Mharbulous/bb097396fa92ef1d34d03a72b56b2c61 |
132 | 131 |
|
133 | 132 | | | Tokens | |
134 | 133 | |--|-------:| |
135 | | -| Baseline total (15 task-runs) | 5,122,105 | |
136 | | -| jMunch total | 19,406 | |
| 134 | +| Baseline total (15 task-runs) | 5,799,695 | |
| 135 | +| jMunch total | 25,220 | |
137 | 136 | | **Reduction** | **99.6%** | |
138 | | -| **Ratio** | **263.9x** | |
| 137 | +| **Ratio** | **230.0x** | |
139 | 138 |
|
140 | | -> Measured with tiktoken `cl100k_base`. Baseline = all indexed source files. jMunch = search_symbols (top 5) + get_symbol_source x 3 per query. |
| 139 | +> Measured with tiktoken `cl100k_base`. Baseline = all indexed source files. jMunch = search_symbols (top 5) + get_symbol x 3 per query. |
0 commit comments