fix(read): bound remote chunk fetch with a timeout to prevent FUSE thread-pool wedge#200
Merged
Conversation
…read-pool wedge Each FUSE read() runs on a worker thread blocked in block_on on a synchronous CAS/CDN fetch. fetch_data awaited stream.next() with no timeout, so a stalled connection (client-aborted media seek, hung CDN socket, killed ffmpeg probe) parked that worker thread forever. The MAX_ATTEMPTS retry loop only caught errors and EOF, never a hang. Once enough stalled reads accumulated, all max_threads (default 16) workers were wedged and the mount silently stopped serving cold reads while readdir kept working (served from cached metadata on a thread that resolves instantly). No error surfaced because the await never returned. Bound each chunk read with a configurable timeout (--read-fetch-timeout-ms, default 30s, 0 disables). On expiry we log an error (breaking the silence), fail the attempt, and let the retry loop eventually return EIO. Dropping the stream on the way out cancels the in-flight request. Note: the sparse-write read path (fill_sparse_holes) has the same unbounded await and should get the same treatment when that branch lands.
Contributor
Benchmark Results |
Contributor
POSIX Compliance (pjdfstest) |
XciD
force-pushed
the
fix/read-fetch-timeout
branch
3 times, most recently
from
c9c4c73 to
ec823c1
Compare
XciD
force-pushed
the
fix/read-fetch-timeout
branch
2 times, most recently
from
1c050f2 to
1b1edab
Compare
XciD
added a commit
that referenced
this pull request
Jun 30, 2026
…204) ## Context Follow-up to #200. A reviewer on the xet-core side ([xet-core#880](huggingface/xet-core#880)) pointed out that the xet client already has the correctly-grained stall guard: `HF_XET_CLIENT_READ_TIMEOUT`, a **per-read inactivity timeout that resets on every byte received** (so slow-but-progressing transfers are unaffected; only a genuine stall trips it). We had it set to **600s**, raised long ago "for shard uploads". But uploads now use a separate client with **no** read_timeout (`shard_upload_http_client` / `build_auth_http_client_no_read_timeout`), so that 600s only governs the **download/reconstruction** path. A stalled CAS/CDN read therefore wouldn't error for 10 minutes, which is what let stalled reads pile up and wedge the mount. ## Change Drop `HF_XET_CLIENT_READ_TIMEOUT` to **30s**. - #200's per-chunk timeout already bounds the streaming **read** path (`fetch_data`). - This additionally bounds the **whole-file download** path (`download_to_file`, used by advanced-write open / flush), which goes straight through xet-core's `download_file` and was bounded only by this value. That makes the separate write-path workaround in #203 unnecessary — closing it. Healthy transfers are unaffected (the timeout resets on progress). Still overridable via the env var. ## Layering note This is the fix at the right layer (the client), as the xet-core reviewer recommended. #200's per-chunk wrapper stays as complementary defense-in-depth: it also catches non-socket stalls (e.g. a stall inside the reconstruction pipeline) that a socket read_timeout would not.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
After a number of reads, all cold reads on the mount freeze silently.
head -c 5MBon a not-yet-read file returns 0 bytes and hangs until timeout, with no error in the daemon log.lsstill responds (metadata, no data fetch). A fresh daemon re-reads everything fine, proving it is accumulated daemon state, not the network/file/token.Observed in production under media streaming (Jellyfin over an hf-mount bucket): ffmpeg probes, seeks (each opens a new range download) and client retries generate many cold fetches that get abandoned.
Root cause
Each FUSE
read()runs on a worker thread blocked inruntime.block_on(virtual_fs.read(...)), and the session runs withn_threads = max_threads(default 16). The read path'sfetch_dataawaitedstream.next()with no timeout; theMAX_ATTEMPTSretry loop only caughtErr/EOF, never a hang. A stalled fetch (client-aborted seek, hung CDN socket) parked its worker forever (block_onis not cancellable). Once enough piled up, all 16 worker threads were wedged and the mount stopped serving cold reads.readdirkept working (cached metadata, resolves instantly). Nothing was logged because theawaitnever returned.This path is not gated on write mode — reads of clean remote files go through the lazy prefetch (
open_lazy→fetch_data) in every mode, so this is the fix for the reported wedge regardless of--advanced-writes.Fix
--read-fetch-timeout-msflag (default30000,0disables), plumbedMountOptions→VfsConfig→VirtualFs.fetch_databounds each chunk read withtokio::time::timeout. On expiry it logs anerror!(breaking the silent hang), fails the attempt, and lets the retry loop returnEIO. Dropping the stream cancels the in-flight request.Test
read_stalled_stream_times_out_with_eio: a mock stream that hangs onnext()is read through a VFS with a 150ms fetch timeout; the read returnsEIOinstead of hanging. An outertokio::time::timeoutguard makes a regression (unbounded wait) fail deterministically. Verified that with the bound disabled (Duration::ZERO, i.e. the pre-fix code path) the test fails withElapsed, confirming it reproduces the wedge.Scope
This PR is just the read-path fix — the reported wedge. Complementary follow-ups, split out so this stays minimal and mergeable:
fill_sparse_holes) has the same unbounded await but does not exist onmainyet.