HTTP: retry transient failures of cat_file and HTTPFile block reads - #2124
Open
lfoppiano wants to merge 2 commits into
Open
HTTP: retry transient failures of cat_file and HTTPFile block reads#2124lfoppiano wants to merge 2 commits into
lfoppiano wants to merge 2 commits into
Conversation
HTTPFileSystem never retried anything: every cat_file and every HTTPFile block read was one session.get plus one body read, so a single 503, a throttling 429 or a connection dropped mid-body killed the whole read. Add a small loop around those two paths that retries 408/425/429/5xx, dropped or reset connections, timeouts and truncated bodies, honouring Retry-After and otherwise waiting retry_wait * 2**n (capped, jittered). 404, 401/403 and other 4xx are never retried, so subclasses that map them in _raise_not_found_for_status keep working. The loop wraps request and body read together on purpose: a mid-body ClientPayloadError surfaces from r.read(), which a request-level retry client never sees. Two silent truncations become (retried) errors: a 206 body shorter than the requested range while the file size says more exists, and a 416 for a range inside the file, the CloudFront behaviour reported in fsspec#1895. New HTTPFileSystem options retries=3 and retry_wait=1.0, overridable per open(); retries=0 restores the previous behaviour. The shared test server gains header-driven fault injection (fail_status, fail_times, retry_after, truncate_body, short_body) and a reset_faults fixture. Refs fsspec#550, fsspec#1895.
Add a ``retry_statuses`` option to HTTPFileSystem (default unchanged: 408/425/429/500/502/503/504) and an overridable ``_is_retryable(exc)`` method, so a caller can opt extra codes in (a CDN answering 403 under load) or opt default codes out without patching the module. Dropped connections, timeouts and truncated bodies are still always retried. HTTPFile picks the predicate up from its filesystem.
lfoppiano
force-pushed
the
http-retry
branch
from
September 9, 2026 14:14
b75b5ad to
495faf0
Compare
lfoppiano
marked this pull request as ready for review
September 9, 2026 14:16
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HTTPFileSystemdoes not seems to have a mechanism for retry: everycat_fileand everyHTTPFileblock read was onesession.getplus one body read, so a single 503, a throttling 429 or a connection dropped mid-body killed the whole read.With this PR we added a small loop around those two paths that retries 408/425/429/5xx, dropped or reset connections, timeouts and truncated bodies, honouring Retry-After and otherwise waiting retry_wait * 2**n (capped, jittered).
404, 401/403 and other 4xx are never retried, so subclasses that map them in _raise_not_found_for_status keep working. The loop wraps request and body read together on purpose: a mid-body ClientPayloadError surfaces from r.read(), which a request-level retry client never sees.
The HTTP error handling the retry can be altered using
retry_statusesoption to HTTPFileSystem (default: 408/425/429/500/502/503/504) and an overridable_is_retryable(exc)method, so a caller can opt extra codes in (a CDN answering 403 under load) or opt default codes out without patching the module.Two silent truncations become (retried) errors: a 206 body shorter than the requested range while the file size says more exists, and a 416 for a range inside the file, the CloudFront behaviour reported in #1895.
New HTTPFileSystem options retries=3 and retry_wait=1.0, overridable per open(); retries=0 restores the previous behaviour. The shared test server gains header-driven fault injection (fail_status, fail_times, retry_after, truncate_body, short_body) and a reset_faults fixture.
Refs #550, #1895.
This PR was derived from one issue raised and fixed here.