Bridge Sighthouse library corpus into a BSimVis reference collection - #48
Draft
rdmmf wants to merge 3 commits into
Draft
Bridge Sighthouse library corpus into a BSimVis reference collection#48rdmmf wants to merge 3 commits into
rdmmf wants to merge 3 commits into
Conversation
…nce collection Sighthouse compiles library sources under several compiler variants and analyses them with Ghidra. This pulls the resulting build artifacts out of its S3 repo and stages them for upload as a BSimVis reference collection, which is the corpus side of standard-library identification: a pool with only_cross_collection can then match unknown binaries against known library code without computing similarities inside the reference set. Keeps object files as well as shared libraries -- only openssl and glibc emit .so in this corpus, so a .so-only bridge would silently drop zlib, zstd, lz4, upx and mbedtls. Test, fuzz and demo paths are excluded. Uploads pass --skip-sim: a reference collection is only ever compared against other collections, so intra-collection similarities are wasted work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Testing the bridge end-to-end against a live BSimVis turned up four
defects in the previously unexercised --upload path:
- There is no `bsimvis-upload` binary; uploading is a subcommand of the
single `bsimvis` entrypoint.
- bsimvis resolves bsimvis_config.toml relative to the working directory,
which is not necessarily where the bridge runs from. Added --bsimvis-config.
- bsimvis rejects a bare directory as a target ("Target path is not a
file"); its "directory/*" help means a shell-expanded glob. Pass the
extracted files individually.
- bsimvis upload exits 0 even when every file fails, so the exit code was
reporting uploads that never happened. Parse the printed success rate
instead, and report extracted and uploaded counts separately.
Also adds --only to bridge a single group, which is what makes a small
end-to-end run practical.
Verified: zlib v1.2.13 x86_64-O1, 19/19 files uploaded to a live instance,
tags stdlib/lib:zlib/ver:v1.2.13/variant:x86_64-O1 present on each, and 0
similarities built in the collection confirming --skip-sim takes effect.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The usage example described an invocation that cannot work: it used the wrong port and omitted --bsimvis-bin and --bsimvis-config, both of which are needed in practice. Replace it with the form actually verified end-to-end, and document the prerequisite that rustfs must be running and does not publish port 9000 to the host. Drop DEFAULT_REPO and make --repo required. A default pointing at 127.0.0.1:9000 could never connect, so it only produced an obscure failure. Also records the collection the corpus lands in (stdlib-ref), the tags each file carries, and the known limitation that top-level demo programs such as zlib's example.o are not caught by the path-based noise filter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Corpus side of standard-library identification (no issue open yet — discussed as strategy #2: a reference collection matched via a pool with
only_cross_collection).scripts/sighthouse_bridge.pyreads Sighthouse's S3 repo, takes every job that reached the Ghidra Analyzer success state, and stages its compiled artifacts on disk grouped by(library, version, compiler variant). Each group becomes onebsimvis-uploadrun so the group's tags (stdlib,lib:*,ver:*,variant:*) land on every file in it.Notes
.o, not just.so. Only openssl and glibc emit shared objects in this corpus; zlib, zstd, lz4, upx and mbedtls are object-files-only, so a.so-only bridge would silently drop most of the libraries.test/,fuzz/,doc/,demos/,examples/,benchmark/— harness code, not library code. Trims openssl from 2215 to 1894 objects.--skip-simon every upload, since a reference collection is only compared across collections.--uploadruns them.Checks
--selftestcovers the selection filter and variant parsing../scripts/wt-test.sh: 278/278 passed.Not run end-to-end into a live BSimVis — no API instance was up, so the
--uploadpath is unexercised.🤖 Generated with Claude Code