feat: implement async-safe token bucket rate limiting - #19
Open
iron-prog wants to merge 2 commits into
Open
Conversation
Contributor
Author
|
@zas |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Following up on the recent SSE transport addition, we need to ensure the server remains compliant with MusicBrainz's strict 1 request-per-second API limit. This is critical now that the server can handle multiple concurrent connections or aggressive LLM reasoning loops that fire multiple tool calls simultaneously.
Implementation
mcp_musicbrainz/throttling/token_bucket.pyimplementing a standard token bucket (capacity: 3, refill rate: 1.0/s) to allow for small bursts while maintaining the average limit.threading.Lock()to ensure safe token consumption across FastMCP's thread pools, making it robust for multi-user Uvicorn/SSE deployments.time,threading,functools) to avoid bloatingpyproject.toml.Architectural Decisions
@rate_limitedDecorator: Keptserver.pyclean by intercepting tool calls via a decorator rather than repeatingif/elseblocks inside every function.@rate_limitedunderneath@cached_tool(). This ensures that if FastMCP hits the disk cache, the request returns instantly without needlessly draining the token bucket.HTTP 429(which often crashes simple LLM agents), throttled requests gracefully return a semantic string:"Error: MusicBrainz Rate Limit reached. Please retry in 1.0s.". This allows the AI to pause, read the context, or summarize existing data without breaking its reasoning loop.Testing
Verified locally by simulating a "Thundering Herd" of 5 concurrent tool calls to
search_artists. The first 3 requests successfully parsed data (consuming the initial burst capacity), and the subsequent 2 were safely intercepted and returned the agentic error string without hitting the network.