libstore: eliminate curl_easy_pause() to fix TCP window collapse on large downloads

[jld-adriano]

Motivation

curl_easy_pause(CURLPAUSE_RECV) collapses the TCP receive window (rcv_wnd drops from ~80 KB to ~7 KB) and, with HTTP/2, adds WINDOW_UPDATE round-trip overhead on every pause/unpause cycle. For large NAR downloads where the sink is slow (e.g. store imports at ~5 MB/s while curl bursts at 300+ MB/s), this throttles effective throughput by ~60x.

This was the root cause of multi-minute CI stalls when downloading from binary caches — observed in production on exa-labs/monorepo CI runners. The stall affects all substituter protocols on nix ≥2.33: both https:// and s3://, since S3BinaryCacheStore now inherits from HttpBinaryCacheStore (NixOS/nix PRs NixOS#14198 + NixOS#14350, merged Oct 2025).

Context

Since nix 2.33, S3BinaryCacheStore inherits from HttpBinaryCacheStore, meaning both s3:// and https:// substituters download NARs through the same FileTransfer::download(request, sink) streaming path. This path calls curl_easy_pause() when the buffer exceeds downloadBufferSize, triggering TCP window collapse on every pause/unpause cycle. With the default 1 MB buffer (or even 64 MB), a 2.9 GB NAR like tensorrt-llm causes hundreds of pause cycles, each throttling the connection.

Note for nix <2.33: On older versions, S3BinaryCacheStore used the AWS SDK directly for data transfer (not curl), so only https:// substituters were affected.

Related monorepo PRs for context: exa-labs/monorepo#29228 (disabled HTTP/2), exa-labs/monorepo#29235 (reverted to s3://), exa-labs/monorepo#29254 (download-buffer-size → 4 GB as a config-level workaround). This PR fixes the underlying nix issue so all substituters work at full speed regardless of download-buffer-size.

Changes

Two commits, read in order:

1. http-binary-cache-store.cc — HttpBinaryCacheStore::getFile(path, Sink&) now downloads the entire file via the non-streaming fileTransfer->download(request) before writing to the sink. This is the targeted fix for the binary cache path (covers both https:// and s3:// on nix ≥2.33).

2. filetransfer.cc — The general-purpose streaming download(request, sink) path no longer calls curl_easy_pause(). The buffer grows unboundedly and the consumer drains concurrently. This benefits all callers of the streaming path, not just binary caches.

Note: commit 1 makes commit 2 partially redundant for HttpBinaryCacheStore (since it no longer uses the streaming path at all), but commit 2 fixes the behavior for any other code that calls download(request, sink).

Review checklist

• Memory usage: Both changes allow peak memory equal to the full download size per concurrent transfer. With max-substitution-jobs = 128 and a 2.9 GB NAR, worst-case peak is significant — verify runners have headroom (our runners have 123+ GB RAM).
• Discarded ItemHandle: Commit 2 discards the return value of enqueueFileTransfer since unpauseTransfer is no longer called. Verify ItemHandle destruction has no side effects (it's a trivial wrapper around std::reference_wrapper<Item>).
• downloadBufferSize setting: Now effectively unused by the streaming path — the buffer is never checked against it. Should it be deprecated or removed in a follow-up?
• Backpressure removal: Commit 2 removes ALL backpressure from streaming downloads globally, not just binary caches. Any caller relying on bounded memory from the streaming path will now see unbounded growth.

---

Add 👍 to pull requests you find important.

Link to Devin session: https://app.devin.ai/sessions/beb12ee3c3c94474ad193926cde267a9
Requested by: @jld-adriano

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 5 additional findings.