Add -remap-path-prefix: reproducible .bo/.ba output across build directories#1040
Open
nanavati wants to merge 6 commits into
Open
Add -remap-path-prefix: reproducible .bo/.ba output across build directories#1040nanavati wants to merge 6 commits into
nanavati wants to merge 6 commits into
Conversation
The compiler bakes the invocation directory into every output: source positions carry pwd-encoded file names (even for relative invocations, via createEncodedFullFilePath), the importer's .bo embeds the importee's source path and a content hash of its .bo (so path nondeterminism cascades transitively through the package graph), and .ba files additionally store the elaboration path, source name, and the path-valued Flags fields. Under remote execution (Bazel), every sandbox has a different absolute path, so identical inputs produce different outputs and nothing caches. Add a repeatable flag, -remap-path-prefix FROM=TO (after gcc's -ffile-prefix-map and rustc's --remap-path-prefix): stored paths whose decoded form starts with FROM have it replaced by TO at serialization time. The longest matching FROM wins; unmatched paths are untouched, and with no flag given the output bytes are unchanged (verified against the previous compiler). Positions are remapped at the single sharing chokepoint in BinData (the cache stays keyed by the original position, so the emitted sharing pattern is unaffected); .ba files additionally remap abmi_path/abmi_src_name (and the schedule-error variants) and normalize the stored Flags copy -- including clearing remapPathPrefix itself, whose FROM values are machine-specific by construction. Diagnostics that print positions from imported .bo files show the remapped (e.g. workspace-relative) form. The .ba format tag advances for the new Flags field; the .bo format is unchanged. A build invokes it as: bsc -remap-path-prefix "$PWD=." ... testsuite/bsc.options/remap-path compiles the same sources from two directories and requires byte-identical, residual-path-free, and repeatable .bo/.ba output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019cqQdWKp7jr73GHQYnvhiq
Keying the cache on the pre-remap position was needless caution: the reader reconstructs sharing from occurrence order, so keying on what is actually written both canonicalizes the stream and lets positions that remap equal share one payload. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019cqQdWKp7jr73GHQYnvhiq
- bump the .ba format tag (bsc-ba-20260712-1): the Bin Flags record gained a field, so same-tag readers would misparse stale .ba files (the original commit message promised this bump; the code now does it) - regenerate the -help and -print-flags-raw goldens for the new flag - add remapPathPrefix to showFlagsRaw's field list - remap dumpAll in the stored .ba Flags copy - remapPathMaybe fast path: no-match returns the original interned value (no per-occurrence FString re-interning) - document the duplicate-FROM tie-break; grep -qF in the test script Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019cqQdWKp7jr73GHQYnvhiq
The BinData/GenBin changes in this PR thread a Position transform through the encoder without altering any serialized bytes when the flag is unused -- verified by byte-comparison against the previous compiler (see the test's no-flag check). The .ba tag was bumped for the real format change (new Flags field); the .bo format is unchanged. Bin-format: bo-unchanged Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019cqQdWKp7jr73GHQYnvhiq
nanavati
added a commit
to nanavati/bsc
that referenced
this pull request
Jul 12, 2026
BOUNDARY-CONTRACT.md is a self-contained handoff to the interface- contract refactor: the measured requirement (parent .bo/.ba byte-stability under child BODY edits), the two mechanisms that break it today (elaboration heap positions leaking child shape into parent def names — 12.6k dumpba diff lines from one leaf constant; and ipkg_depends hashing the import's FULL content instead of its contract), the contract contents (the auto-derived VModInfo/BVI equivalent: types, ports, conflict matrix, clock/reset domains, paths), what is already solved (same-dir determinism passes; PR B-Lang-org#1040 -remap-path-prefix closes path sensitivity), and the acceptance test. tools/ba-stability-audit.sh is that test: A same-dir determinism (PASS), B path insensitivity (PASS with B-Lang-org#1040), C parent stability under leaf body edit (FAIL today — flips green when the contract is load-bearing). CPHASE-PLAN.md: the sharpened dual-flow doctrine (no Bazel rules — flags only; bsc orchestrates trs always, module-specific work behind -c, only link-level work in -e, Bazel needs no knowledge of trs beyond the binary riding the toolchain filegroup; -trs-export-only dropped) and the I1(e) experiment results with the B-Lang-org#1040 flag set. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011boYaaQ9oCqhZ3aCsFNk5L
nanavati
added a commit
to nanavati/bsc
that referenced
this pull request
Jul 12, 2026
I2 MEASURED and CLEARS, class-scoped: per-type exec share is 66-94% on outline-heavy designs (sudoku/conflict_free/grid — the replicated statically-scheduled target shape) and ~0-15% on fused-testbench designs; I4 activates exactly when the outline dial fires. REBASE-PLAYBOOK.md: fleet-verified upstream study — true merge-base d2f996c, ~30 of 38 upstream commits are squashes of this line's own work; conflict map (Bin Flags binder replay, TCMisc/TCheck hotspot, ANoInlineFun named params, SplitVector packaging), the B-Lang-org#1040-last rule, and the narrow post-rebase parity-audit targets (6be62d6, 71226f0). Plus two pre-rebase discoveries: the encExpr tuple fix is unblocked (port model already in-tree), and SimExportIR S.toList sites need an interned-Id-order audit before the -c/-e split. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011boYaaQ9oCqhZ3aCsFNk5L
nanavati
added a commit
to nanavati/bsc
that referenced
this pull request
Jul 12, 2026
I2 MEASURED and CLEARS, class-scoped: per-type exec share is 66-94% on outline-heavy designs (sudoku/conflict_free/grid — the replicated statically-scheduled target shape) and ~0-15% on fused-testbench designs; I4 activates exactly when the outline dial fires. REBASE-PLAYBOOK.md: fleet-verified upstream study — true merge-base d2f996c, ~30 of 38 upstream commits are squashes of this line's own work; conflict map (Bin Flags binder replay, TCMisc/TCheck hotspot, ANoInlineFun named params, SplitVector packaging), the B-Lang-org#1040-last rule, and the narrow post-rebase parity-audit targets (6be62d6, 71226f0). Plus two pre-rebase discoveries: the encExpr tuple fix is unblocked (port model already in-tree), and SimExportIR S.toList sites need an interned-Id-order audit before the -c/-e split. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011boYaaQ9oCqhZ3aCsFNk5L
…er decoder remapPathMaybe applied getFullFilePath to every input, but on a path without the /// pwd marker its recursion appends a spurious '/' to the last component: remapping /build/dir/Foo.bsv with FROM=/build/dir stored "./Foo.bsv/" in the .ba (abmi_src_name and every path-valued Flags field were affected). Decode only paths that actually carry the marker. The remap-path test now also runs an absolute-path invocation and checks the stored strings: no trailing separator, no residual build dir, and byte-identity with the relative invocation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019cqQdWKp7jr73GHQYnvhiq
grep for "./Foo.bsv/" rather than "Foo.bsv/" so a chance 0x2f byte after the serialized string in the binary .ba cannot false-positive; the byte-identity check remains the authoritative assertion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019cqQdWKp7jr73GHQYnvhiq
matx-amy
reviewed
Jul 14, 2026
Comment on lines
+340
to
+341
| -- remapping helper lives in FileNameUtil; it is passed in here to | ||
| -- avoid an import cycle. |
Contributor
There was a problem hiding this comment.
i don't see an import cycle?
Contributor
other than that, reviewed, 👍 don't love the need to manually traverse things to find the paths, but i don't love taking on a dep on syb or something either |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
bsc bakes the invocation directory into every output file, so identical inputs compiled in different directories produce different bytes. Under remote execution (Bazel), every sandbox has a different absolute path, so nothing ever caches and outputs are non-deterministic by construction.
An audit of where machine-specific bytes enter the formats found:
Positioncarries a pwd-encoded file name (createEncodedFullFilePathembeds the pwd even for relative invocations), shared once per file in the.bo/.bastream..boembeds the importee's source path (via positions on importedIds) and a content hash of the imported.bo(ipkg_depends), so one path difference at the leaves ripples through the entire package graph even where the raw string doesn't appear..baextras:abmi_path/abmi_src_name(and the schedule-error variants) store the elaboration directory directly, and the serializedFlagsrecord storesbdir,bluespecDir, the search paths, output paths, and dump targets.abmi_timeis long gone), and the read-time hash is computed over payload bytes, so it becomes deterministic exactly when the bytes do. (Designs that usedate/epochTime/Environmentvalues remain inherently non-reproducible; that is orthogonal to paths.)Fix
A repeatable flag,
-remap-path-prefix FROM=TO(after gcc's-ffile-prefix-mapand rustc's--remap-path-prefix), applied at serialization time:FROMhave it replaced byTO; the longest matchingFROMwins; unmatched paths are stored untouched. Decoding the internal///pwd marker before matching means a match also normalizes away the relative-vs-absolute invocation difference.BinData(share), before sharing, so the cache is keyed on the stored value: positions that remap equal share one payload and the stream is canonical..bafiles additionally remapabmi_path/abmi_src_name(+abmsei_*) and normalize the storedFlagscopy (which is consumed only by bluetcl introspection) — including clearingremapPathPrefixitself, whoseFROMvalues are machine-specific by construction..bofiles show the remapped (e.g. workspace-relative) form, which keeps them readable.Guarantees (all verified empirically;
testsuite/bsc.options/remap-pathpins them):-remap-path-prefix "$PWD=.", the same sources compiled from two different directories produce byte-identical.boand.ba, with no residual build-directory strings, including through an import (-u) and an elaboration (-g).The
.baformat tag advances for the newFlagsfield; the.boformat is unchanged.Usage under Bazel
Pass
-remap-path-prefix "$PWD=."(or=<workspace-relative package dir>) on every compile; if the toolchain itself lives inside the sandbox, add a second mapping for its prefix. Generated Verilog headers additionally want the existing-no-show-timestamps.