Investigate broader CLAUDE.md / AGENTS.md discovery in find_overhead_files #290

Summary

find_overhead_files (crates/relayburn-sdk/src/analyze/overhead.rs:93-112) currently checks a fixed, hard-coded set of three relative paths under the project root and stops there:

This is narrow enough to miss real overhead-bearing files in a non-trivial number of layouts. We should investigate whether to extend discovery and, if so, settle the semantics before changing code.

Motivation

Three concrete gaps observed in real repos:

1. Ancestor CLAUDE.md / AGENTS.md. Claude Code and Codex both walk up to ancestor directories looking for instruction files; running burn overhead from a subpackage (e.g. packages/foo/) misses the repo-root CLAUDE.md even though the harness was paying for it on every turn. The cost attribution silently underreports.
2. .codex/AGENTS.md (and similar hidden subdirs). Codex supports a .codex/AGENTS.md location analogous to .claude/CLAUDE.md. We don't have a candidate for it. Same idea may apply to other harnesses we add later.
3. Non-deterministic ordering once Add Codex session reader and CLI integration #1 is in play. As soon as we start walking ancestors, the result vector needs a defined order so per-file rows in burn overhead and the gather_overhead aggregation in crates/relayburn-sdk/src/query_verbs.rs:650 are stable across invocations and platforms.

Current behavior (verified)

• find_overhead_files joins each candidate's parts to project_path, stats the result, and pushes matches in declaration order. No traversal, no sort. (crates/relayburn-sdk/src/analyze/overhead.rs:93-112.)
• The lone discovery test (find_overhead_files_discovers_all_three_with_correct_applies_to, crates/relayburn-sdk/src/analyze/overhead.rs:230-253) only covers the three-file project-root case.
• attribute_overhead already de-duplicates riding-turns across discovered files via a per-session max_riding_by_session map (crates/relayburn-sdk/src/analyze/overhead.rs:127-151), so total_riding_turns is correct even when CLAUDE.md and .claude/CLAUDE.md both attribute to the same session. Per-file total_cost is still summed verbatim into grand_total, though — overlapping content between two discovered files double-counts in the cost number even when riding-turns don't.
• Caller gather_overhead (crates/relayburn-sdk/src/query_verbs.rs:650-696) iterates whatever order find_overhead_files returns; the CLI's burn overhead table renders that order verbatim, and the "no overhead files found" message hard-codes the candidate list (crates/relayburn-cli/src/commands/overhead.rs:53,103).

Open questions to investigate

1. How far up should we walk? Options:

   • To the nearest VCS root (git rev-parse --show-toplevel).
   • To $HOME (matches how Claude Code resolves ~/.claude/CLAUDE.md).
   • To filesystem root (matches Codex's documented walk).
   • Capped at N levels.

   Pick one; document it; stop bouncing between.

2. What additional candidates belong in CANDIDATES?

   • .codex/AGENTS.md (agents-md, appliesTo: ['codex']?).
   • User-level: ~/.claude/CLAUDE.md, ~/.codex/AGENTS.md — these are real overhead but not project-scoped; should they live in a separate "global overhead" surface instead of being mixed in?
   • Anything we should add for OpenCode that I'm missing?

3. appliesTo for ancestor files. Same as project-root, or do we need to narrow (e.g. an ancestor .claude/CLAUDE.md only applies to claude-code sessions whose CWD is at or below the file)? The current cost math doesn't have a CWD filter — adding one is a separate, larger change.

4. Cost double-counting. Riding-turns are already deduped (max-per-session), but grand_total is the additive sum of per-file total_cost. If both <repo>/CLAUDE.md and <repo>/sub/CLAUDE.md exist and we're invoked from <repo>/sub, both are read by Claude Code and both contribute to the cached prefix — so the cost number double-counts even though the rides don't. Decide: report per-file cost as-is and let grand_total overstate, switch grand_total to the same max-per-session model as rides, or introduce a "merged virtual file" so totals don't double up. Worth pulling in someone who's read the Claude Code prefix-caching code.

5. Ordering. Proposal: sort by (depth from project_path ascending, then path string). That makes the project-root file first (matching today's output for the common case) and is platform-stable. Confirm this matches what burn overhead should display.

6. Performance. A bounded ancestor walk is cheap; an unbounded walk on a deep mount could stat() dozens of paths per invocation. Probably fine, but worth measuring on a realistic monorepo.

Suggested investigation steps

1. Read the Claude Code and Codex docs (or source) for the actual ancestor-walk rule each harness uses. Match ours to whatever they do — fabricating our own discovery rule will diverge from the cost the harness is actually paying.
2. Audit real ~/.relayburn/ ledgers to see how often turns reference CLAUDE.md content that wouldn't be discovered by the current rule (heuristic: file paths in tool-use blocks, or sessions whose cache_read is suspiciously high relative to discovered overhead).
3. Sketch the new CANDIDATES shape — likely the static parts slice becomes "path template + ancestor-walk strategy" rather than a fixed prefix.
4. Spec the ordering and double-counting rules in writing, then implement with tests covering: ancestor walk hit, hidden-dir candidate hit, multiple matches sorted, empty project, file vs directory at candidate path. Update the hard-coded candidate list in the CLI's "no overhead files found" message (crates/relayburn-cli/src/commands/overhead.rs:53,103) to match.

Out of scope (for this issue)

• Walking ancestors when loading a specific overhead file. load_overhead_file takes an explicit path; only discovery changes.
• The CWD-filtered appliesTo semantics in question (3) above. If we go there, it's a follow-up.
• Skill-level overhead (SKILL.md, etc.) — separate surface.

References

• Discovery + attribution: crates/relayburn-sdk/src/analyze/overhead.rs (find_overhead_files at :93, attribute_overhead at :119, tests at :230-).
• SDK caller: crates/relayburn-sdk/src/query_verbs.rs:650 (gather_overhead), wired into LedgerHandle::overhead and LedgerHandle::overhead_trim.
• CLI presenter: crates/relayburn-cli/src/commands/overhead.rs — burn overhead and burn overhead trim, including the hard-coded candidate list in the empty-state message.
• Related: Rust perf: stop cloning whole turn streams in analyze hot paths #325 (analyze hot-path clones), mcp: expand tool catalog to cover the read surface (summary, hotspots, overhead, overhead trim) #237 (MCP tool catalog including overhead surface).