Skip to content

fix: enforce IO concurrency for metadata reading#2230

Open
Overproness wants to merge 1 commit intoconda:mainfrom
Overproness:main
Open

fix: enforce IO concurrency for metadata reading#2230
Overproness wants to merge 1 commit intoconda:mainfrom
Overproness:main

Conversation

@Overproness
Copy link
Copy Markdown

Description

Fixes #2021

The IO concurrency limits (configured via InstallDriver) were only being respected during individual file linking operations, not during the reading of index.json and paths.json metadata at the start of each package installation.

When installing many packages concurrently:

  • The link_package function in installer/mod.rs spawned hundreds of rayon tasks simultaneously
  • Each task called link_package_sync, which immediately reads index.json and paths.json from disk with no throttling
  • This created hundreds of concurrent file opens, exhausting file descriptors on systems with low ulimit values

Solution: Acquire an IO permit from the InstallDriver before spawning each rayon task. The permit is held for the entire duration of the synchronous linking operation (including metadata reads), ensuring that at most io_concurrency_limit packages (default 100) perform IO-intensive operations concurrently.

The fix is minimal and non-disruptive:

  • Adds 7 lines of code (5 lines + 2 for the permit binding)
  • Keeps the synchronous link_package_sync API unchanged
  • Leverages existing OwnedSemaphorePermit which is Send + 'static and moves cleanly into the rayon closure

How Has This Been Tested?

  • cargo check -p rattler passes with no errors
  • cargo test -p rattler passes all 53 relevant install tests on WSL Ubuntu
  • No test changes needed; the fix doesn't alter the public API or functionality, only enforces existing constraints

AI Disclosure

  • This PR contains AI-generated content.
    • I have tested any AI-generated content in my PR.
    • I take responsibility for any AI-generated content in my PR.
  • Tools: Claude Haiku (via GitHub Copilot) for code design and implementation assistance

Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation (not needed; no API changes)
  • I have added sufficient tests to cover my changes (existing tests cover this)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Install IO concurrency not respected for index.json and paths.json reading

1 participant