Skip to content

feat: implement async-safe token bucket rate limiting - #19

Open
iron-prog wants to merge 2 commits into
zas:mainfrom
iron-prog:feat/rate-limiting
Open

feat: implement async-safe token bucket rate limiting#19
iron-prog wants to merge 2 commits into
zas:mainfrom
iron-prog:feat/rate-limiting

Conversation

@iron-prog

@iron-prog iron-prog commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

Motivation

Following up on the recent SSE transport addition, we need to ensure the server remains compliant with MusicBrainz's strict 1 request-per-second API limit. This is critical now that the server can handle multiple concurrent connections or aggressive LLM reasoning loops that fire multiple tool calls simultaneously.

Implementation

  • Token Bucket Algorithm: Added mcp_musicbrainz/throttling/token_bucket.py implementing a standard token bucket (capacity: 3, refill rate: 1.0/s) to allow for small bursts while maintaining the average limit.
  • Thread Safety: Utilized threading.Lock() to ensure safe token consumption across FastMCP's thread pools, making it robust for multi-user Uvicorn/SSE deployments.
  • Zero Dependency: Built entirely with the Python standard library (time, threading, functools) to avoid bloating pyproject.toml.

Architectural Decisions

  1. The @rate_limited Decorator: Kept server.py clean by intercepting tool calls via a decorator rather than repeating if/else blocks inside every function.
  2. Cache Awareness: Carefully stacked @rate_limited underneath @cached_tool(). This ensures that if FastMCP hits the disk cache, the request returns instantly without needlessly draining the token bucket.
  3. Agentic Error Handling: Instead of raising a hard Python exception or an HTTP 429 (which often crashes simple LLM agents), throttled requests gracefully return a semantic string: "Error: MusicBrainz Rate Limit reached. Please retry in 1.0s.". This allows the AI to pause, read the context, or summarize existing data without breaking its reasoning loop.

Testing

Verified locally by simulating a "Thundering Herd" of 5 concurrent tool calls to search_artists. The first 3 requests successfully parsed data (consuming the initial burst capacity), and the subsequent 2 were safely intercepted and returned the agentic error string without hitting the network.

@iron-prog iron-prog changed the title feat: implement async-safe token bucket rate limiting (The Sentinel) feat: implement async-safe token bucket rate limiting Apr 7, 2026
@iron-prog

Copy link
Copy Markdown
Contributor Author

@zas
This keeps rate limiting simple at the process level, but with HTTP/ASGI and concurrent users, we may eventually need a shared or per-user rate limiting strategy to avoid cross-user interference.
Happy to improve it further in that direction if it makes sense for the roadmap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant