Skip to content

TODO.md

Latest commit

 

History

History
14 lines (9 loc) · 2.78 KB

TODO.md

File metadata and controls

14 lines (9 loc) · 2.78 KB
Raw

TODO

Open work on the benchmark harness and docs. Surfaced 2026-06-10 during a token-efficiency review of the harness, the generated code, and the published numbers.

Harness Bugs

  • CLAUDE.md grows a stale duplicate context block every chained round. run-chain.sh:272 (and run-feature.sh:215) append a fresh ## Project-Specific Context section with printf ... >> CLAUDE.md without removing the previous one. By F5 the Brace work dir's CLAUDE.md contains five such blocks - four stale and contradicting the current code (old entity lists missing endDate, trackId, waitlistCount, etc.). That is a few thousand junk tokens loaded into context on every turn of every later round, for all frameworks, inflating costs and risking the model trusting an outdated entity shape. Fix: delete everything from the first ## Project-Specific Context marker to EOF before appending the regenerated block (the two scripts duplicate this logic - extract it to one shared function while in there).
  • The Brace route listing is broken and misleading. run-chain.sh:267 and run-feature.sh:208 grep only App.java for app.get(...) etc., but the generated code registers routes inside controllers via XController.register(app) - so the "Routes (from App.java)" section lists exactly one route (/) while presenting itself as the full set. Either grep all of src/ like the Hono branch (run-feature.sh:179) does, or drop the section for Java projects and let the source-file list stand alone.
  • Re-run F1-F5 after fixing the above. Absolute costs should drop for all frameworks once the duplicated-context tokens are gone. The relative Brace-vs-Spring conclusion is expected to hold since both bugs hit all three frameworks equally, but the published numbers predate the fix - update the results tables and note the harness change in the provenance section.

Methodology / README

  • State the idiom-vs-necessity caveat. Spring does not force the Controller -> Service -> Repository -> DTO layering; the AI writes it because that is what idiomatic Spring looks like in training data. The benchmark measures "cost of the code an AI actually emits per framework," which is the realistic measure, but the causal claim is "frameworks whose idiom is verbose cost more" - worth one sentence in Caveats so a skeptical reader does not get there first.
  • Keep the Brace README in sync with this repo's reconciled numbers. The brace repo's README still shows pre-reconciliation figures ($5.43 cumulative, F5 $1.36/41%, "the gap widens as the codebase grows") vs this repo's $5.62, F5 $1.54/33%, and "fairly stable around a third, rather than widening monotonically." Tracked on the Brace side in its TODO.md (Documentation section); listed here too since this repo is the source of truth for the data.