perf(cli): faster snapshot create with multi-threaded zstd in one container#1061
Merged
Conversation
…ontainer `abi stack snapshot create` archived each stateful volume with a separate `alpine tar czf` (single-thread gzip) container, run sequentially. On a real 3.88GB TDB2 (fuseki) volume this took ~146s and produced a 1.06GB archive, with the wall-clock dominated by single-thread gzip on the one large volume. Two changes: - Compress with multi-threaded zstd instead of gzip. A tiny helper image (alpine + zstd, ~6MB) is baked once via `ensure_snapshot_helper_image()` before the stack is stopped, so its one-time build never lands in the downtime window and there is no per-snapshot package-mirror dependency. Compression runs inside the container, so only the compressed bytes cross the docker VM->host boundary. Measured: ~146s -> ~23s on the fuseki volume, and the archive roughly halves (1.06GB -> 501MB). - Archive every volume in a SINGLE helper container (`archive_volumes`) instead of one container per volume, paying container/image startup once. Volumes are mounted read-only at /from/<key> and streamed `tar cf - | zstd -T0 -3` to /to/<key>.tar.zst. `set -eo pipefail` so a failed tar aborts the run rather than being masked by zstd's exit code. Volume archives are now `.tar.zst` (centralised in `volume_archive_name`); `extract_volume` decompresses with `zstd -dc | tar xf -`. No backward-compat shim is needed (no existing snapshots). Cross-volume parallelism was deliberately not added: one 3.88GB volume is ~81% of the bytes, so parallel archiving caps at ~1.2x, and a per-archive `-T0` already saturates all cores. The clean `docker compose stop` is kept (Fuseki/TDB2 needs a clean checkpoint; `docker pause` only yields a crash-consistent, in-flight-lossy image). `storage/` intentionally stays on host-side gzip to preserve the audited `_safe_extract`/`_assert_safe_members` guard and avoid a host-zstd dependency. Tests: 56 pass; batched archive + per-volume extract verified byte-identical end-to-end on real volumes.
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Deploying abi-docs with
|
| Latest commit: |
6644bc4
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://6acbb8b6.abi-docs-76c.pages.dev |
| Branch Preview URL: | https://singularity-unruffled-hodgki.abi-docs-76c.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Speeds up
abi stack snapshot createby replacing the per-volume single-thread gzip archiving with multi-threaded zstd, all done in a single helper container.Changes
.tar.zst, compressed withzstd -T0(all cores). A tiny helper image (abi-snapshot-helper:zstd1= alpine + zstd, ~6 MB) is baked once viaensure_snapshot_helper_image()before the stack is stopped, so its one-time build never lands in the downtime window and there's no per-snapshot package-mirror dependency. Compression runs inside the container, so only the compressed bytes cross the Docker VM→host boundary.archive_volumes()mounts every present volume read-only at/from/<key>and streamstar cf - | zstd -T0 -3to/to/<key>.tar.zstin a singledocker run, paying container/image startup once instead of 7×.set -eo pipefailso a failedtaraborts the run rather than being masked by zstd's exit code.extract_volume()decompresses withzstd -dc | tar xf -. Filenames centralised involume_archive_name().Why
On a real 3.88 GB TDB2 (
fuseki) volume, gzip took ~146 s and produced a 1.06 GB archive. Benchmarking showed the wall-clock was dominated by single-thread gzip on the one large volume (which is ~81% of the bytes), not I/O.Result: ~6× faster on the dominant volume and roughly half the on-disk/export size.
Notes for reviewers
.tar.gzarchives — there are no existing snapshots to migrate.-T0already saturates all cores; combining both would oversubscribe them.docker compose stopis kept.docker pausewas considered and rejected: the freezer doesn't flush/fsync, so it only yields a crash-consistent, in-flight-lossy image — unsafe for Fuseki/TDB2 (unflushed mmap indexes + journal at an unclean boundary).storage/intentionally stays on host-side gzip to preserve the audited_safe_extract/_assert_safe_membersguard and avoid a host-zstddependency; it's the minority of bytes.Testing
snapshot_runtime_test.py,snapshot_test.py), including new coverage for the helper-image build (present/missing/failure) and the batched archive command.