bench(bsim): reproducible corpus + weighted-vs-unweighted at scale

[rdmmf]

Stacks on worktree-bsim-weights (#50). Replaces the 21-function comparison posted there with a corpus that can actually separate the algorithms.

The corpus (doc/bench_corpus.md)

5 open-source C projects — sqlite, zlib, lua, zstd, mbedtls — pinned by version and sha256, cross-compiled to 6 targets × 3 optimisation levels × dynamic/static: 180 binaries, 313,685 defined functions. One direct compiler invocation per binary, no autotools and no container, so it rebuilds identically on any machine with the Debian cross toolchains. Nothing is vendored here; build_corpus.sh fetches and verifies.

Two deliberate choices: binaries are unstripped, because symbol names are the ground truth (same name across builds = known match, no labelling pass); and half are statically linked, so libc — the boilerplate weighting is supposed to suppress — is really in the corpus, and unrelated programs genuinely share code.

The reference side of every query is every other project's binary of the same variant, so a wrong top-1 is a cross-program false positive rather than a same-program near-miss. That is what #50's single-project dataset could not measure at all.

Results (doc/bench_weighted_vs_unweighted.md)

46 binaries extracted so far, 166 build pairs, 18,239 queried functions:

axis / algo	recall@1	MRR	cross-program FP
all jaccard	86.2%	0.896	2.89%
all unweighted_cosine	80.3%	0.841	4.00%
all weighted_cosine	86.4%	0.899	2.59%
opt unweighted_cosine	63.3%	0.696	9.75%
opt weighted_cosine	75.0%	0.810	6.44%

• Optimisation level is the hard axis and where weighting pays: +11.7 points over unweighted. Linkage saturates at ~97.8% for everything — a benchmark reporting only that would call all three equivalent.
• Jaccard is not a weak baseline. It ties weighted overall. Any case for changing the default has to clear jaccard, not just unweighted cosine.
• Cost: 1.16–1.19x unweighted per pair, parity on large vectors, matching the reference compare() to 7.8e-16.
• IDF gate, run against a real collection for the first time: 0/50 of this corpus's most common features are missing from Ghidra's table, 34.5% of occurrences covered. Applicable here; still open for the Go/MIPS production corpus.

Ghidra BSim baseline

bsim_baseline.py + bsim/BSimQueryAll.java build a real BSim H2 database, commit signatures and query it headlessly. 12.9% of true matches never appear in BSim's results at all — eliminated by LSH binning before scoring. oracle_compare.py proves our arithmetic matches Ghidra's; this shows what its candidate-selection stage costs, which that test cannot.

Benchmark-only: it drives support/bsim and support/analyzeHeadless from the Ghidra already in bin/. No BSim dependency, driver or config is added to BSimVis.

Throughput, and one honest gap

jaccard 34.2 s / 6,359 sims vs unweighted_cosine 80.4 s / 34,568 sims on 5,709 functions (cosine admits 5.4x more pairs at the same threshold, so it is cheaper per similarity written). Ingest is ~7 min/binary, all decompiler — scoring changes are noise against it.

weighted_cosine cannot be measured end to end: the build path rejects it, so all quality numbers above come from offline scoring. Landing weighting in the build path is the prerequisite for a real throughput comparison.

CLI fix

--save-json was silently ignored on the --local-analysis chunked path. Fixed, plus --no-upload (offline extraction, no stack) and --save-vectors, which writes compact {function: {hash: tf}} — the full dump of a static binary is several GB, which no offline consumer can load without OOM.

Verification

./scripts/wt-test.sh: 326/326 pass.

Extraction of the remaining 134 binaries is Ghidra-bound and still running; the corpus, scripts and conclusions do not change, the numbers will move slightly.

🤖 Generated with Claude Code