fix: use sequential read instead of seek for multipart uploads on Windows

[s-zx]

Summary

File uploads over 2GB get stuck at ~1.98GB on Windows 11 with Python 3.12. The upload speed reduces then stops entirely.

Root Cause

On Windows, file.seek() with offset >= 2GB can fail due to 32-bit signed integer limits in some C runtime APIs. The multipart LFS upload used SliceFileObj, which seeks to chunk_size * part_idx for each part. When that offset exceeds 2^31 bytes, the seek fails and the upload hangs.

Fix

Replace SliceFileObj + seek with sequential reads: file.read(chunk_size) for each chunk. This avoids large-offset seeks entirely and works for files of any size on Windows.

Changes:

• src/huggingface_hub/lfs.py: _upload_parts_iteratively now reads chunks sequentially
• src/huggingface_hub/cli/lfs.py: lfs_multipart_upload (git-lfs custom transfer) uses the same approach

Fixes #3871

---

Note

Low Risk
Small, localized change to multipart upload chunk reading logic; primary risk is behavior differences around EOF/truncated files and memory usage per chunk.

Overview
Fixes multipart LFS uploads hanging on Windows for files >2GB by removing SliceFileObj/seek()-based chunking and switching to sequential read(chunk_size) uploads in both the CLI custom transfer agent (cli/lfs.py) and the library uploader (lfs._upload_parts_iteratively).

Adds a guard that raises a ValueError if a chunk unexpectedly reads empty before all parts are uploaded, making truncated reads fail fast instead of stalling.

^{Written by Cursor Bugbot for commit 062c99b. This will update automatically on new commits. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[Wauplin]

Hi @s-zx thanks for raising this error. I had never thought of this limitation on 32-bits systems. I do understand the need for a fix but I think it should rather be done in SliceFileObj instead with iterative file.seek if needed.

The problem of the current solution is that it introduces a potentially large memory footprint which we want to avoid at all costs (especially since it's to fix an issue happening only in rare cases). Could you rewrite the PR differently? Thanks in advance

[Wauplin]

This will load the entire chunk_data into memory which we absolutely want to avoid on large files.

[s-zx]

@Wauplin Thanks for the feedback — reworked in the latest commit.

The fix is now entirely inside SliceFileObj:

• Added _safe_seek() helper in _lfs.py that breaks large SEEK_SET offsets (>= 2^31) into incremental SEEK_CUR steps of at most 2^31 - 1 bytes
• Updated the three seek call sites in SliceFileObj.__enter__, __exit__, and seek() to use _safe_seek()
• Reverted cli/lfs.py and lfs.py back to original SliceFileObj usage — no more loading chunks into memory

Zero-copy, no memory overhead, and the callers don't need to know about the 32-bit limitation.