Skip to content

Commit 26a585e

Browse files
NickCirvclaude
andcommitted
feat(reach): git co-change as a complementary reach tier (#143)
Adds "files that historically changed together" as Tier 2 of the sub-agent broker's related-files (graph-adjacency → co-change → path-reach). Validated by the #141 temporal-holdout (non-circular): co-change ALONE is weak (≤ a popularity baseline), but COMBINED with path-reach it beats popularity on all 3 repos, CI excludes 0 — and it's the only source that reaches CROSS-DIRECTORY / NON-CODE targets (e.g. a route co-changing with schema.prisma) that graph edges and path-reach structurally cannot. PATH-KEYED on purpose. The git-miner already computed a path-keyed coChangeMap; the existing depends_on edges stem-collapse it (merging distinct files) and bind it to node ids, which drops exactly the non-code cross-dir pairs that matter. This exposes the raw pairs and persists them keyed by real file path. - git-miner: expose `coChange` (pairs with count ≥ 2; zero extra git work). - migrate v10: `co_change(file_a,file_b,count)` table (safe on fresh + v9→v10, idempotent) + reverse-lookup index. - store: replaceCoChange (full replace — co-change is global, recomputed from full history each mine, correct on incremental too) + getCoChangeNeighbors (bidirectional, count-ranked, never returns the focal). - relatedFiles: Tier 2 between graph and path-reach. Never-worse by construction — co-change only fills slots graph-adjacency left empty; the `seen` set dedups across tiers; focal can never appear. NEVER-WORSE / honesty held: co-change framed as weak-alone/complementary, not a standalone claim. Real-repo e2e: for engram's own src/core.ts it surfaces CHANGELOG.md and package.json — cross-dir, non-code files with no graph node. Independent correctness review: SAFE TO COMMIT (never-worse by construction, migration safe both paths, params bound, full-replace correct). Its 2 nits (redundant index, bidirectional-lookup test gap) both applied/closed here. Audit: tsc clean; 1245/1245 vitest (+6: pure tier + e2e cross-dir + direct bidirectional/tie-break); real-repo init migrates v9→v10 + populates 259 pairs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 154f0f9 commit 26a585e

7 files changed

Lines changed: 250 additions & 30 deletions

File tree

src/core.ts

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -172,6 +172,10 @@ export async function init(
172172
store.clearAll();
173173
}
174174
store.bulkUpsert(allNodes, allEdges);
175+
// #143 — persist path-keyed co-change for the reach tier. Recomputed from
176+
// the full git history each mine (it's global, not per-changed-file), so a
177+
// full replace is correct on both full and incremental init.
178+
store.replaceCoChange(gitResult.coChange);
175179
store.setStat("last_mined", String(Date.now()));
176180
store.setStat("project_root", root);
177181
// Persist mtimes for next incremental run
@@ -330,7 +334,10 @@ export async function relatedFilesFor(
330334
const graphAdjacent = [...adjCount.entries()]
331335
.sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]))
332336
.map(([f]) => f);
333-
return relatedFiles(focal, graphAdjacent, [...allFilesSet], limit);
337+
// Tier 2 (#143) — git co-change neighbours, count-ranked. Path-keyed, so it
338+
// reaches cross-dir / non-code files (config, schema) the graph tier can't.
339+
const coChange = store.getCoChangeNeighbors(focal, limit);
340+
return relatedFiles(focal, graphAdjacent, coChange, [...allFilesSet], limit);
334341
} finally {
335342
store.close();
336343
}

src/db/migrate.ts

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ export interface MigrationResult {
1515
}
1616

1717
/** Current schema version — bump this when adding new migrations. */
18-
export const CURRENT_SCHEMA_VERSION = 9;
18+
export const CURRENT_SCHEMA_VERSION = 10;
1919

2020
export interface RollbackResult {
2121
readonly fromVersion: number;
@@ -34,6 +34,9 @@ export interface RollbackResult {
3434
* automatic. Forward migrations are append-only and idempotent.
3535
*/
3636
const DOWN_MIGRATIONS: Record<number, string> = {
37+
// v4.5 (#143): path-keyed co-change table. Cleanly droppable.
38+
10: `DROP INDEX IF EXISTS idx_co_change_b;
39+
DROP TABLE IF EXISTS co_change;`,
3740
// v4.0: bi-temporal mistake fields (then_believed, found_false_at,
3841
// truth_now, applies_to). SQLite pre-3.35 cannot DROP COLUMN cleanly;
3942
// leaving the columns in place is safe and we don't depend on their
@@ -206,6 +209,20 @@ CREATE INDEX IF NOT EXISTS idx_query_cache_file ON query_cache(file_path);`,
206209
addColumnIfMissing(db, "nodes", "truth_now", "truth_now TEXT");
207210
addColumnIfMissing(db, "nodes", "applies_to", "applies_to TEXT");
208211
},
212+
213+
// v4.5 (#143): path-keyed co-change table for the reach tier. Keyed by real
214+
// file paths (not node ids), so it reaches cross-dir / non-code targets (e.g.
215+
// a route co-changing with schema.prisma) that the node-bound edges miss. Full
216+
// CREATE TABLE IF NOT EXISTS so it's safe on fresh AND existing databases.
217+
10: `
218+
CREATE TABLE IF NOT EXISTS co_change (
219+
file_a TEXT NOT NULL,
220+
file_b TEXT NOT NULL,
221+
count INTEGER NOT NULL,
222+
PRIMARY KEY (file_a, file_b)
223+
);
224+
CREATE INDEX IF NOT EXISTS idx_co_change_b ON co_change(file_b);
225+
`,
209226
};
210227

211228
type ExecDb = { exec: (sql: string) => Array<{ values: unknown[][] }> };

src/graph/related-files.ts

Lines changed: 20 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -2,15 +2,17 @@
22
* related-files.ts — tiered "you'll likely also need these files" for a focal
33
* file, for the sub-agent context broker (#83/#139).
44
*
5-
* TIERING (bench-validated, #139): graph-adjacent files FIRST (real call/import
6-
* edges — highest signal), then path-reach APPENDED (test↔impl above same-dir
7-
* siblings). Path candidates never displace a graph hit, so a focal file with
8-
* rich graph adjacency is unaffected and path-reach only fills the slots the
9-
* graph left empty — never-worse by construction. On engram's own repo this
10-
* lifted recall@10 +6.1pp and nearly doubled the ranker's lift over random.
5+
* TIERING (bench-validated): graph-adjacent files FIRST (real call/import edges —
6+
* highest precision, #139), then GIT CO-CHANGE (files historically changed
7+
* together — #141 temporal-holdout: weak alone, but combined with path-reach it
8+
* beats a popularity baseline on 3/3 repos, CI excludes 0; it's the only source
9+
* that reaches cross-directory pairs like route↔schema that path-reach can't),
10+
* then PATH-REACH (test↔impl above same-dir siblings, #139). Each lower tier only
11+
* fills the slots the higher tiers left empty, so a focal with rich graph
12+
* adjacency is unaffected — never-worse by construction.
1113
*
12-
* Pure + dependency-free (composes reach.ts). The store-touching adjacency query
13-
* lives in core.ts; this module is the orderable, unit-testable core.
14+
* Pure + dependency-free (composes reach.ts). The store-touching adjacency + the
15+
* co-change lookup live in core.ts; this module is the orderable, unit-testable core.
1416
*/
1517
import { sameDirSiblings, testImplCounterparts } from "./reach.js";
1618

@@ -20,12 +22,16 @@ import { sameDirSiblings, testImplCounterparts } from "./reach.js";
2022
* @param focal the file the sub-agent's work centers on
2123
* @param graphAdjacent files sharing a graph edge with focal, in the caller's
2224
* preferred (e.g. degree-ranked) order — Tier 1
23-
* @param allFiles every source file in the graph (for path-based reach)
25+
* @param coChange files that historically changed together with focal,
26+
* count-ranked (already path-keyed, may include non-code
27+
* files like config/schema) — Tier 2
28+
* @param allFiles every source file in the graph (for path-based reach) — Tier 3
2429
* @param limit max files to return (>0; <=0 yields [])
2530
*/
2631
export function relatedFiles(
2732
focal: string,
2833
graphAdjacent: readonly string[],
34+
coChange: readonly string[],
2935
allFiles: readonly string[],
3036
limit: number
3137
): string[] {
@@ -42,8 +48,11 @@ export function relatedFiles(
4248
for (const f of graphAdjacent) {
4349
if (take(f)) return out;
4450
}
45-
// Tier 2 — path-reach: test↔impl counterparts (strong co-change signal) above
46-
// same-dir siblings (moderate). Appended only after every graph hit.
51+
// Tier 2 — git co-change (count-ranked; reaches cross-dir code↔config pairs).
52+
for (const f of coChange) {
53+
if (take(f)) return out;
54+
}
55+
// Tier 3 — path-reach: test↔impl counterparts above same-dir siblings.
4756
for (const f of testImplCounterparts(focal, allFiles)) {
4857
if (take(f)) return out;
4958
}

src/graph/store.ts

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -342,6 +342,55 @@ export class GraphStore {
342342
return out;
343343
}
344344

345+
/**
346+
* Replace the entire co_change table with `pairs` (#143). Full replace because
347+
* co-change is recomputed from the complete git history on each mine. Pairs are
348+
* stored once under the (file_a, file_b) sorted key the miner produced; the
349+
* neighbour query reads both columns. INSERT OR REPLACE keeps it idempotent.
350+
*/
351+
replaceCoChange(pairs: ReadonlyArray<{ a: string; b: string; count: number }>): void {
352+
this.db.run("BEGIN TRANSACTION");
353+
try {
354+
this.db.run("DELETE FROM co_change");
355+
for (const p of pairs) {
356+
if (p.a && p.b && p.a !== p.b) {
357+
this.db.run(
358+
"INSERT OR REPLACE INTO co_change (file_a, file_b, count) VALUES (?, ?, ?)",
359+
[p.a, p.b, p.count]
360+
);
361+
}
362+
}
363+
this.db.run("COMMIT");
364+
} catch (err) {
365+
this.db.run("ROLLBACK");
366+
throw err;
367+
}
368+
this.save();
369+
}
370+
371+
/**
372+
* Files that historically co-changed with `filePath`, most-frequent first,
373+
* capped to `limit`. Reads both columns since a pair is stored once under the
374+
* sorted key. Returns paths (which may be non-code files like config/schema).
375+
*/
376+
getCoChangeNeighbors(filePath: string, limit = 10): string[] {
377+
const stmt = this.db.prepare(
378+
`SELECT other, count FROM (
379+
SELECT file_b AS other, count FROM co_change WHERE file_a = ?
380+
UNION ALL
381+
SELECT file_a AS other, count FROM co_change WHERE file_b = ?
382+
) ORDER BY count DESC, other ASC LIMIT ?`
383+
);
384+
stmt.bind([filePath, filePath, Math.max(0, limit)]);
385+
const out: string[] = [];
386+
while (stmt.step()) {
387+
const row = stmt.getAsObject() as { other?: string };
388+
if (typeof row.other === "string" && row.other !== filePath) out.push(row.other);
389+
}
390+
stmt.free();
391+
return out;
392+
}
393+
345394
incrementQueryCount(nodeId: string): void {
346395
this.db.run(
347396
"UPDATE nodes SET query_count = query_count + 1 WHERE id = ?",

src/miners/git-miner.ts

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,26 @@ import { execFileSync } from "node:child_process";
66
import { resolve } from "node:path";
77
import type { GraphEdge, GraphNode } from "../graph/schema.js";
88

9+
/** A path-keyed co-change pair: files `a` and `b` changed together `count` times.
10+
* Unlike the stem-collapsed `depends_on` edges, this keeps the real file paths
11+
* (incl. non-code files like config/schema) so it can drive cross-dir reach (#143). */
12+
export interface CoChangePair {
13+
a: string;
14+
b: string;
15+
count: number;
16+
}
17+
918
interface GitMineResult {
1019
nodes: GraphNode[];
1120
edges: GraphEdge[];
21+
/** Path-keyed co-change pairs with count ≥ CO_CHANGE_MIN_COUNT (file→file). */
22+
coChange: CoChangePair[];
1223
}
1324

25+
/** Min commits two files must co-change before the pair is persisted. 1 = noise,
26+
* so require a repeated signal; ranking by count handles the rest at query time. */
27+
const CO_CHANGE_MIN_COUNT = 2;
28+
1429
function runGit(args: string[], cwd: string): string {
1530
try {
1631
return execFileSync("git", args, {
@@ -52,7 +67,7 @@ export function mineGitHistory(
5267

5368
// Check if this is a git repo
5469
const isGit = runGit(["rev-parse", "--git-dir"], root);
55-
if (!isGit) return { nodes, edges };
70+
if (!isGit) return { nodes, edges, coChange: [] };
5671

5772
// Get recent commits with files changed
5873
const log = runGit(
@@ -65,7 +80,7 @@ export function mineGitHistory(
6580
root
6681
);
6782

68-
if (!log) return { nodes, edges };
83+
if (!log) return { nodes, edges, coChange: [] };
6984

7085
// Parse commits into file groups
7186
const coChangeMap = new Map<string, Map<string, number>>();
@@ -170,5 +185,16 @@ export function mineGitHistory(
170185
});
171186
}
172187

173-
return { nodes, edges };
188+
// Path-keyed co-change pairs (#143). coChangeMap stores each unordered pair once
189+
// under the sorted key, so this is already de-duplicated. Unlike the stem-
190+
// collapsed depends_on edges above, these keep the real file paths — including
191+
// non-code files (config, schema, css) — so they can drive cross-dir reach.
192+
const coChange: CoChangePair[] = [];
193+
for (const [a, neighbours] of coChangeMap) {
194+
for (const [b, count] of neighbours) {
195+
if (count >= CO_CHANGE_MIN_COUNT) coChange.push({ a, b, count });
196+
}
197+
}
198+
199+
return { nodes, edges, coChange };
174200
}

tests/core-cochange.test.ts

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
/**
2+
* Co-change reach tier (#143) — end-to-end on a real git repo.
3+
* Proves the cross-directory win path-reach structurally can't reach:
4+
* a src/lib file co-changing with a prisma/ schema in different directories.
5+
*/
6+
import { describe, it, expect, beforeEach, afterEach } from "vitest";
7+
import { init, relatedFilesFor, getStore } from "../src/core.js";
8+
import { mkdtempSync, rmSync, mkdirSync, writeFileSync } from "node:fs";
9+
import { execFileSync } from "node:child_process";
10+
import { join } from "node:path";
11+
import { tmpdir } from "node:os";
12+
13+
function git(cwd: string, ...args: string[]): void {
14+
execFileSync("git", args, { cwd, stdio: "pipe" });
15+
}
16+
17+
describe("co-change reach tier (#143)", () => {
18+
let root: string;
19+
20+
beforeEach(async () => {
21+
root = mkdtempSync(join(tmpdir(), "engram-cc-"));
22+
git(root, "init", "-q");
23+
git(root, "config", "user.email", "t@t");
24+
git(root, "config", "user.name", "t");
25+
mkdirSync(join(root, "src", "lib"), { recursive: true });
26+
mkdirSync(join(root, "prisma"), { recursive: true });
27+
// 3 commits that change a src/lib file AND a prisma schema together — a
28+
// cross-directory co-change pair (count 3 ≥ CO_CHANGE_MIN_COUNT=2).
29+
for (let i = 0; i < 3; i++) {
30+
writeFileSync(join(root, "src", "lib", "db.ts"), `export const q${i} = ${i};\n`);
31+
writeFileSync(join(root, "prisma", "schema.prisma"), `model M${i} {}\n`);
32+
git(root, "add", "-A");
33+
git(root, "commit", "-q", "-m", `change ${i}`);
34+
}
35+
await init(root);
36+
});
37+
38+
afterEach(() => rmSync(root, { recursive: true, force: true }));
39+
40+
it("getCoChangeNeighbors surfaces the cross-dir co-changed file (either query direction)", async () => {
41+
const store = await getStore(root);
42+
try {
43+
expect(store.getCoChangeNeighbors("src/lib/db.ts", 10)).toContain("prisma/schema.prisma");
44+
expect(store.getCoChangeNeighbors("prisma/schema.prisma", 10)).toContain("src/lib/db.ts");
45+
} finally {
46+
store.close();
47+
}
48+
});
49+
50+
it("relatedFilesFor reaches the co-changed schema (a non-code, cross-dir file with no graph node)", async () => {
51+
const rel = await relatedFilesFor(root, "src/lib/db.ts", 5);
52+
expect(rel).toContain("prisma/schema.prisma");
53+
});
54+
55+
it("co-change is rebuilt as a full replace on re-init (idempotent, no duplicates)", async () => {
56+
await init(root); // second mine
57+
const store = await getStore(root);
58+
try {
59+
const n = store.getCoChangeNeighbors("src/lib/db.ts", 10);
60+
expect(n.filter((f) => f === "prisma/schema.prisma")).toHaveLength(1);
61+
} finally {
62+
store.close();
63+
}
64+
});
65+
66+
it("getCoChangeNeighbors: bidirectional lookup + count ordering + tie-break (direct)", async () => {
67+
const store = await getStore(root);
68+
try {
69+
// Hand-seed known pairs (stored once under the sorted key, a < b).
70+
store.replaceCoChange([
71+
{ a: "a.ts", b: "b.ts", count: 5 }, // query side = file_a
72+
{ a: "b.ts", b: "c.ts", count: 9 }, // query "b.ts" must hit the file_a side
73+
{ a: "b.ts", b: "d.ts", count: 9 }, // tie with c.ts at count 9 → path ASC
74+
]);
75+
// From b.ts: neighbours are c.ts(9), d.ts(9), a.ts(5) — count DESC, tie by path ASC.
76+
expect(store.getCoChangeNeighbors("b.ts", 10)).toEqual(["c.ts", "d.ts", "a.ts"]);
77+
// From a.ts (only on the file_a side of one pair) → b.ts.
78+
expect(store.getCoChangeNeighbors("a.ts", 10)).toEqual(["b.ts"]);
79+
// From c.ts (only on the file_b side) → b.ts. Never returns itself.
80+
expect(store.getCoChangeNeighbors("c.ts", 10)).toEqual(["b.ts"]);
81+
// Limit is respected.
82+
expect(store.getCoChangeNeighbors("b.ts", 1)).toEqual(["c.ts"]);
83+
} finally {
84+
store.close();
85+
}
86+
});
87+
});

0 commit comments

Comments
 (0)