Description
guard() currently starts one provider request for every text chunk or non-empty PDF page at the same time.
For sufficiently large inputs, a single guard() call can therefore create a large burst of concurrent provider requests. This can increase the likelihood of provider rate limits, connection and memory pressure, and fallback bursts. Callers currently have no per-call way to apply backpressure.
This behavior is currently mirrored across both SDKs:
- TypeScript uses unbounded
Promise.all() for text chunks and PDF pages.
- Python uses unbounded
asyncio.gather() for the same paths.
I believe guard() would benefit from an optional per-call concurrency limit so applications can control the amount of parallel provider work generated by a single request.
Proposed Solution
Add an optional concurrency setting to GuardOptions in both SDKs:
- TypeScript:
maxConcurrency
- Python:
max_concurrency
Use a bounded concurrency mechanism for both text-chunk and PDF-page analysis.
The exact default value, validation rules, and whether an explicit unlimited mode should remain available can be discussed with maintainers.
The implementation should preserve:
- Existing result ordering
- Classification and aggregation semantics
- Violation/CWE aggregation
- Reasoning aggregation
- Token usage accounting
- Existing error propagation behavior
The concurrency behavior should remain consistent between the TypeScript and Python SDKs.
Additional Context
This would primarily improve reliability for large inputs where guard() currently creates many simultaneous provider requests. Limiting concurrency could reduce request bursts and make provider rate-limit behavior more predictable without changing the core guard semantics.
Related work: #1158 addresses honoring Retry-After after a provider returns HTTP 429. That is related to rate-limit resilience, but it does not limit the number of concurrent chunk/page requests created by a single guard() call.
I searched existing issues and PRs for guard concurrency, chunk concurrency, PDF-page concurrency, parallel requests, and related rate-limit work, and did not find an exact existing proposal for this behavior.
I'm leaving the API details and default concurrency value open for maintainer feedback before implementation.
Description
guard()currently starts one provider request for every text chunk or non-empty PDF page at the same time.For sufficiently large inputs, a single
guard()call can therefore create a large burst of concurrent provider requests. This can increase the likelihood of provider rate limits, connection and memory pressure, and fallback bursts. Callers currently have no per-call way to apply backpressure.This behavior is currently mirrored across both SDKs:
Promise.all()for text chunks and PDF pages.asyncio.gather()for the same paths.I believe
guard()would benefit from an optional per-call concurrency limit so applications can control the amount of parallel provider work generated by a single request.Proposed Solution
Add an optional concurrency setting to
GuardOptionsin both SDKs:maxConcurrencymax_concurrencyUse a bounded concurrency mechanism for both text-chunk and PDF-page analysis.
The exact default value, validation rules, and whether an explicit unlimited mode should remain available can be discussed with maintainers.
The implementation should preserve:
The concurrency behavior should remain consistent between the TypeScript and Python SDKs.
Additional Context
This would primarily improve reliability for large inputs where
guard()currently creates many simultaneous provider requests. Limiting concurrency could reduce request bursts and make provider rate-limit behavior more predictable without changing the core guard semantics.Related work: #1158 addresses honoring
Retry-Afterafter a provider returns HTTP 429. That is related to rate-limit resilience, but it does not limit the number of concurrent chunk/page requests created by a singleguard()call.I searched existing issues and PRs for guard concurrency, chunk concurrency, PDF-page concurrency, parallel requests, and related rate-limit work, and did not find an exact existing proposal for this behavior.
I'm leaving the API details and default concurrency value open for maintainer feedback before implementation.