fix: Replace aiofile with aiofiles to prevent caio context leak under concurrent execution - #12525
Conversation
… concurrent execution aiofile uses caio (kernel AIO) which creates contexts in a global dict that are never cleaned up. Under concurrent execution these accumulate until the OS aio-max-nr limit is exhausted, causing SystemError(11, 'Resource temporarily unavailable'). aiofiles uses thread pools instead and does not have this issue. Migrates all aiofile.async_open usages across both backend and lfx packages to aiofiles.open. Based on #12433 by @manav2000, extended to cover all remaining usages. Co-Authored-By: manav2000 <manav2000@users.noreply.github.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Exercises the exact failure pattern from #12414: multiple concurrent save-then-immediately-read operations on the storage service. This would previously trigger SystemError(11, EAGAIN) after ~150-200 runs with the aiofile/caio backend. Co-Authored-By: manav2000 <manav2000@users.noreply.github.com>
Codecov Report❌ Patch coverage is ❌ Your project status has failed because the head coverage (49.26%) is below the target coverage (60.00%). You can increase the head coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## release-1.9.0 #12525 +/- ##
=================================================
+ Coverage 50.37% 50.39% +0.01%
=================================================
Files 1933 1937 +4
Lines 172918 172750 -168
Branches 26641 24219 -2422
=================================================
- Hits 87112 87056 -56
+ Misses 84732 84644 -88
+ Partials 1074 1050 -24
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
… concurrent execution (#12525) * fix: replace aiofile with aiofiles to prevent caio context leak under concurrent execution aiofile uses caio (kernel AIO) which creates contexts in a global dict that are never cleaned up. Under concurrent execution these accumulate until the OS aio-max-nr limit is exhausted, causing SystemError(11, 'Resource temporarily unavailable'). aiofiles uses thread pools instead and does not have this issue. Migrates all aiofile.async_open usages across both backend and lfx packages to aiofiles.open. Based on #12433 by @manav2000, extended to cover all remaining usages. Co-Authored-By: manav2000 <manav2000@users.noreply.github.com> * test: add concurrent write-then-read regression test for caio EAGAIN fix Exercises the exact failure pattern from #12414: multiple concurrent save-then-immediately-read operations on the storage service. This would previously trigger SystemError(11, EAGAIN) after ~150-200 runs with the aiofile/caio backend. Co-Authored-By: manav2000 <manav2000@users.noreply.github.com> * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) --------- Co-authored-by: manav2000 <manav2000@users.noreply.github.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
aiofile uses caio (kernel AIO) which creates contexts in a global dict that are never cleaned up. Under concurrent execution these accumulate until the OS aio-max-nr limit is exhausted, causing SystemError(11, 'Resource temporarily unavailable'). aiofiles uses thread pools instead and does not have this issue.
Migrates all aiofile.async_open usages across both backend and lfx packages to aiofiles.open.
Based on #12433 by @manav2000, extended to cover all remaining usages.