perf: multi-slot prefetch eliminates parallel pread serialization#22
Closed
XciD wants to merge 5 commits into
Closed
perf: multi-slot prefetch eliminates parallel pread serialization#22XciD wants to merge 5 commits into
XciD wants to merge 5 commits into
Conversation
Contributor
Benchmark Results |
Contributor
POSIX Compliance (pjdfstest) |
XciD
force-pushed
the
perf/reconstruction-cache
branch
from
a9c241d to
b30ca81
Compare
XciD
force-pushed
the
perf/multi-slot-prefetch
branch
2 times, most recently
from
96eb6d8 to
1772c6f
Compare
XciD
force-pushed
the
perf/reconstruction-cache
branch
from
b30ca81 to
98d25dc
Compare
XciD
force-pushed
the
perf/multi-slot-prefetch
branch
from
1772c6f to
f11828d
Compare
XciD
force-pushed
the
perf/multi-slot-prefetch
branch
from
f11828d to
a94e965
Compare
…OS FUSE - Add safetensor concurrent pread benchmark (tests/bench.rs) - Add concurrent_pread_scaling unit test reproducing multi-thread pread - Fix macOS FUSE: disable clone_fd and force n_threads=1 (fuser limitation) - Remove macos-no-mount feature to allow FUSE mount on macOS
Replace the single PrefetchState mutex per file handle with N=8
independent slots. Concurrent pread() calls on a shared fd each
select their own slot via a lock-free 'slot: { ... } pass, so
independent streams never block each other.
Root cause: the old single Mutex<PrefetchState> was held for the
entire network fetch (including sleep/await). All threads on a
shared fd serialized on this one lock, collapsing 8-thread pread
throughput to ~1/30 of sequential (2637 MB/s -> 85 MB/s measured).
Slot selection priority (all via try_lock to avoid blocking):
1. Buffer cache hit (offset already fetched)
2. Near-sequential stream (offset just past buf_start+len)
3. Empty slot (best for a fresh independent stream)
4. Any unlocked slot (evict its buffered data)
5. Blocking fallback by 8 MiB offset bucket
Unit test: inject 50ms first-chunk delay, assert 4-thread wall
time < 150ms (75% of serial 200ms). Without fix: ~28 000ms.
Integration test: tests/pread_scaling.rs uploads 200 MB, opens
one fd, runs 1/2/4-thread strided pread, asserts 2-thread/1-thread
ratio >= 0.3.
…ow on far-seek Non-sequential reads (e.g. safetensors work-stealing across N threads) were issuing one HTTP request per tensor (~3 MiB avg), causing 7x more requests than sequential reads. Each CAS request has ~50ms latency, so 92 requests vs 13 added ~4s of serial latency overhead. Fix: use a 64 MiB minimum fetch window for RangeDownload strategy so that each HTTP request covers the full stride (N x avg_tensor_size) and reduces request count from 92 to ~18 for 8-thread safetensors access. Also stop resetting window_size to INITIAL_WINDOW on far-seek. Slot eviction by another thread should not undo accumulated sequential read headroom. Also adds tests/cas_parallel_download.rs: an integration benchmark that measures raw xet-core CAS throughput for N concurrent download streams without the FUSE layer, isolating xet-core bottlenecks from prefetch logic.
XciD
force-pushed
the
perf/multi-slot-prefetch
branch
from
61b0275 to
9db6d54
Compare
With 4 KiB chunks and an 8 MiB prefetch window, fetch_data runs 2048 non-yielding next() iterations. On 2-core CI runners with 2 tokio worker threads, pairs of tasks (1&2, then 3&4) saturate both workers without yielding, effectively serialising the 4-thread concurrent_pread_scaling test. Bumping chunk_size to 512 KiB reduces the loop to 16 iterations, keeping per-task overhead well under 10 ms and letting all 4 tasks complete their fetches in parallel within the 150 ms threshold.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Concurrent
pread()on a shared file descriptor collapsed to ~1/30 of sequential throughput (2637 MB/s single-thread, 85 MB/s with 2 threads).This matches the exact access pattern used by transformers/vllm when loading safetensors shards: they open one fd per file, then dispatch N worker threads each calling
pread()at different tensor offsets.Root cause:
OpenFile::Lazyheld a singleMutex<PrefetchState>per file handle. The mutex was held for the entire network fetch (including async sleep/await). All concurrent readers on the same fd blocked on this one lock.Fix
Replace the single slot with
N=8independentMutex<PrefetchState>slots per file handle. Each concurrent reader selects its own slot with a non-blockingtry_lockpass, so independent streams never contend.Slot selection priority (all via
try_lock, no blocking until fallback):The slot selection uses a Rust labeled block
'slot: { ... }that holds theMutexGuardfrom the successfultry_lock()call. This eliminates the TOCTOU race that would arise from returning an index and locking separately (multiple tasks could select the same empty slot before any locked it).Results (EC2 m6i.2xlarge, release build)
Unit test with 50ms injected first-chunk delay:
Changes
src/prefetch.rs: addN_PREFETCH_SLOTS = 8constantsrc/virtual_fs/mod.rs:OpenFile::LazyandReadTarget::Remotenow holdArc<Vec<Mutex<PrefetchState>>>, slot selection via labeled blocksrc/test_mocks.rs:MockXetnow usesArc<Vec<u8>>for O(1) clone in concurrent tests, adds injectablefirst_chunk_delay_mssrc/virtual_fs/tests.rs:concurrent_pread_scalingtest with timing assertiontests/pread_scaling.rs: integration test for real FUSE mount (requires HF_TOKEN)