perf(tests): hardlink immutable blobs in template copy - #580
Conversation
…ying The complete-portfolio template contains ~7 MB of content-addressed blobs that are immutable and never modified. Copying them for every test with shutil.copytree causes significant I/O overhead across 80+ composition tests. Replace copytree with a hybrid approach: hardlink blobs/ (content-addressed, immutable) and copy everything else (metadata.sqlite3, lock files). This eliminishes ~7 MB of I/O per test while preserving isolation — hardlinks share inodes but content-addressed blobs are never modified. Includes regression tests verifying hardlinks are used for blobs and copies for mutable metadata.
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 26fc5afd50
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if entry.is_dir(follow_symlinks=False): | ||
| _hardlink_tree(s, d) | ||
| elif entry.is_file(follow_symlinks=False): | ||
| os.link(s, d) |
There was a problem hiding this comment.
When xdist workers copy the shared session template while another worker writes content whose digest is already present, each os.link changes that shared inode's st_ctime_ns. The existing-blob path in src/jacobian/storage/blobs.py:213-234 explicitly compares signatures, including ctime, before and after reading and reports any change as ArtifactIntegrityError; therefore a concurrent fixture setup can make a valid idempotent artifact write fail as apparent corruption. Use copies/reflinks that do not mutate the source inode's metadata, or otherwise avoid sharing these inodes between simultaneously active runtimes.
Useful? React with 👍 / 👎.
1 participant
Summary
The complete-portfolio template contains ~7 MB of content-addressed blobs that are immutable and never modified. Copying them with
shutil.copytreefor every test causes significant I/O overhead across 80+ composition tests (83 test files × 7 MB = ~580 MB of unnecessary copies).Fix
Replace
copytreewith a hybrid approach intests/support/state.py:blobs/(content-addressed, immutable files)metadata.sqlite3, lock files)This eliminates ~7 MB of I/O per test while preserving test isolation — hardlinks share inodes but content-addressed blobs are never modified. New blobs created by a test get new hashes and dont collide.
Validation
tests/unit/support/test_copy_template.py— all pass