feat(responses): add cancel endpoint for background responses - #5051
Closed
cdoern wants to merge 11 commits into
Closed
feat(responses): add cancel endpoint for background responses#5051cdoern wants to merge 11 commits into
cdoern wants to merge 11 commits into
Conversation
meta-cla
Bot
added
the
CLA Signed
Contributor
✱ Stainless preview buildsThis PR will update the Edit this comment to update it. It will appear in the SDK's changelogs.
|
⚠️ llama-stack-client-go studio · conflict
Your SDK build had at least one new warning diagnostic, which is a regression from the base state.
New diagnostics (1 warning)
⚠️ Endpoint/NotConfigured: `post /v1/responses/{response_id}/cancel` exists in the OpenAPI spec, but isn't specified in the Stainless config, so code will not be generated for it.
✅ llama-stack-client-openapi studio · code · diff
Your SDK build had at least one "warning" diagnostic, but this did not represent a regression.
generate ⚠️New diagnostics (1 note)
💡 Endpoint/NotConfigured: Skipped endpoint because it's not in your Stainless config: `post /v1/responses/{response_id}/cancel`
✅ llama-stack-client-python studio · conflict
Your SDK build had at least one new note diagnostic, which is a regression from the base state.
New diagnostics (1 note)
💡 Endpoint/NotConfigured: Skipped endpoint because it's not in your Stainless config: `post /v1/responses/{response_id}/cancel`
This comment is auto-generated by GitHub Actions and is automatically kept up to date as you push.
If you push custom code to the preview branch, re-run this workflow to update the comment.
Last updated: 2026-03-24 15:20:12 UTC
Contributor
|
This pull request has merge conflicts that must be resolved before it can be merged. @cdoern please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork |
This was referenced Mar 17, 2026
Contributor
|
This pull request has merge conflicts that must be resolved before it can be merged. @cdoern please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork |
Add POST /v1/responses/{response_id}/cancel endpoint to cancel queued
or in-progress background responses. The implementation includes:
- CancelResponseRequest model and protocol method
- Task tracking infrastructure to map response_id to asyncio.Task
- Updated background worker to handle CancelledError gracefully
- Proper status transitions and terminal state protection (409 for
completed/failed/incomplete responses)
- Idempotent cancellation support
- Integration tests covering all cancel scenarios (queued, in-progress,
idempotent, terminal states, not found)
The cancelled status already existed in the schema but had no API to
trigger it. This follows the same pattern as the Batches API cancel
implementation.
Signed-off-by: Charlie Doern <cdoern@redhat.com>
Contributor
|
Recording workflow completed Providers: gpt, azure, watsonx Recordings have been generated and will be committed automatically by the companion workflow. Fork PR: Recordings will be committed if you have "Allow edits from maintainers" enabled. |
WatsonX has a concurrent request limit of 10 requests. The new cancel tests pushed the test suite over this limit, causing CI failures. Skip these tests for WatsonX similar to the existing background test skip. Signed-off-by: Charlie Doern <cdoern@redhat.com>
Contributor
|
✅ Recordings committed successfully Recordings from the integration tests have been committed to this PR. |
Contributor
|
This pull request has merge conflicts that must be resolved before it can be merged. @cdoern please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork |
Signed-off-by: Charlie Doern <cdoern@redhat.com>
Co-Authored-By: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: Charlie Doern <cdoern@redhat.com>
In replay mode, background workers process queued responses immediately. If there's no recording (which is the case for cancel tests since the request is cancelled before processing), the worker fails and marks the response as 'failed'. This creates a race between cancel() and the background worker. The time.sleep(0.5) was giving the worker time to pick up the task, causing it to fail in most deployment modes (server, docker-azure). docker-gpt only passed because its worker was slower to start. Fix: Remove the sleep and cancel immediately, winning the race consistently. Signed-off-by: Charlie Doern <cdoern@redhat.com>
…gx-ai#5134) # What does this PR do? Adds per-request HTTP header forwarding to the `remote::passthrough` inference provider, following the pattern established by the safety passthrough provider (PR ogx-ai#5004, already merged). A `forward_headers` config field maps provider-data keys to outbound HTTP header names. Only explicitly listed keys are forwarded from `X-LlamaStack-Provider-Data` to the downstream service (default-deny). An `extra_blocked_headers` field lets operators add custom blocked names on top of the core security list. The shared utility `providers/utils/forward_headers.py` is used by both the inference and safety passthrough providers, keeping the forwarding logic and blocked-header policy in one place. Closes ogx-ai#5040 Relates ogx-ai#4607 ## Test Plan Unit tests cover the full path — config validation, header extraction, CRLF sanitization, blocked-header enforcement, auth priority chain, and concurrent request isolation: ```bash uv run pytest tests/unit/providers/inference/test_passthrough_forward_headers.py -v ``` Tests cover: - `build_forwarded_headers()` — key mapping, default-deny, CRLF stripping, SecretStr unwrap, case-insensitive dedup - `validate_forward_headers_config()` — blocked header rejection, operator extra blocklist, invalid names - Adapter auth priority — static api_key > passthrough_api_key > forwarded Authorization - Provider data validator — extra fields preserved for forwarding, reserved keys rejected - Concurrent request isolation — contextvars don't leak between parallel requests Also tested end-to-end locally against a mock inference server and a mock `/v1/moderations` server. Headers land on the downstream exactly as configured and blocked headers are rejected at stack startup, not at request time. Example config: ```yaml providers: inference: - provider_id: maas-inference provider_type: remote::passthrough config: base_url: ${env.PASSTHROUGH_URL} forward_headers: maas_api_token: "Authorization" tenant_id: "X-Tenant-ID" ``` Callers pass credentials via `X-LlamaStack-Provider-Data`: ```bash curl http://localhost:8321/v1/chat/completions \ -H 'X-LlamaStack-Provider-Data: {"maas_api_token": "Bearer user-jwt", "tenant_id": "acme"}' \ -d '{"model": "passthrough/my-model", "messages": [{"role": "user", "content": "hello"}]}' ``` The downstream receives `Authorization: Bearer user-jwt` and `X-Tenant-ID: acme`. Only keys explicitly listed in `forward_headers` are forwarded to the downstream service. Any keys in `X-LlamaStack-Provider-Data` that don't have a mapping in `forward_headers` are ignored — they never leave the stack. This is the default-deny policy: if it's not in the config, it doesn't get forwarded.
- Add Bedrock (`bedrock/openai.gpt-oss-20b`) to the responses test suite in CI - Pre-register the Bedrock model in the ci-tests distribution config - Include ~330 newly recorded Bedrock response test recordings (104 passing, 87% coverage) - Update the provider compatibility matrix to reflect Bedrock coverage - [x] CI replay tests pass for Bedrock responses suite - [x] Provider compatibility matrix renders correctly with Bedrock column - [ ] Provider record job works --------- Co-authored-by: Sébastien Han <seb@redhat.com>
Background workers in replay mode start processing very quickly, creating a race between cancel() and the worker picking up the task. Even calling cancel immediately isn't fast enough to guarantee it wins the race. Solution: Poll for 'cancelled' status (similar to test_background_completes) instead of asserting immediately after calling cancel(). This handles the case where the worker has already started processing when cancel() is called. Signed-off-by: Charlie Doern <cdoern@redhat.com>
Signed-off-by: Charlie Doern <cdoern@redhat.com>
Collaborator
Author
|
ugh. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Add POST /v1/responses/{response_id}/cancel endpoint to cancel queued or in-progress background responses. The implementation includes:
The cancelled status already existed in the schema but had no API to trigger it. This follows the same pattern as the Batches API cancel implementation.
Test Plan
new integration tests for canceled requests
Closes #5180