[Perf] Run consecutive read-only built-in tools concurrently (#1750) - #2094
Open
ayaangazali wants to merge 1 commit into
Open
[Perf] Run consecutive read-only built-in tools concurrently (#1750)#2094ayaangazali wants to merge 1 commit into
ayaangazali wants to merge 1 commit into
Conversation
…urrently] (kyegomez#1750) The loop executed every built-in tool call serially, so a model response asking for four file reads paid four sequential round trips. Reads have no ordering constraint against each other, only against writes. Consecutive read-only built-ins are now buffered and flushed together through a thread pool. The buffer is flushed before any mutating or control-flow tool runs, and again at the end of the response, so a read can never be reordered past a write. Replaces kyegomez#1772, which I force-pushed. Opening fresh rather than rewriting that branch again so the diff a reviewer already looked at is not moved under them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1750. Replaces #1772.
What is wrong
The autonomous loop executes every built-in tool call serially. A model response asking for four file reads pays four sequential round trips, even though reads have no ordering constraint against one another.
What this does
Consecutive read-only built-ins are buffered and flushed together through a thread pool. Ordering safety comes from when the buffer is flushed, not from inspecting arguments:
create_file,run_bash, ...) runssubtask_done, ...) runsSo a read can never be reordered past a write, and a tool that is not on the read-only list never joins a batch.
Why this replaces #1772
I force-pushed that branch to rebase it, which moves the diff under anyone already reading it. Opening a fresh PR instead and closing that one. Same change, rebased on current
master, no history rewritten.tests/structs/test_agent.py: 88 passed.