-
Notifications
You must be signed in to change notification settings - Fork 1.3k
test(messages): add live Claude Agent SDK smoke test against /v1/messages #6010
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
ca5c4d0
test(messages): add live Claude Agent SDK smoke test against /v1/mess…
cdoern 8238399
Merge branch 'main' into messages-claude-agent-sdk-smoke
cdoern 7a8e061
test(messages): combine CLI and SDK smoke tests into one workflow
cdoern 5b37343
Merge remote-tracking branch 'origin/messages-claude-agent-sdk-smoke'…
cdoern b48b804
fix(ci): use single-word -k pattern for messages client smoke tests
cdoern 9192105
Merge branch 'main' into messages-claude-agent-sdk-smoke
cdoern 2798974
Merge branch 'main' into messages-claude-agent-sdk-smoke
cdoern c7ae6ec
Merge branch 'main' into messages-claude-agent-sdk-smoke
cdoern File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,108 @@ | ||
| # Copyright (c) The OGX Contributors. | ||
| # All rights reserved. | ||
| # | ||
| # This source code is licensed under the terms described in the LICENSE file in | ||
| # the root directory of this source tree. | ||
|
|
||
| """Smoke test: drive the Claude Agent SDK against the OGX Messages API. | ||
|
|
||
| Points the SDK at a local OGX server and runs a single prompt through the | ||
| upstream `claude-agent-sdk` Python package (github.com/anthropics/ | ||
| claude-agent-sdk-python). The SDK does not speak HTTP itself: it spawns the | ||
| Claude Code CLI as a subprocess and parses its streamed session output. This | ||
| exercises a different client surface than the CLI smoke test -- the SDK's | ||
| session machinery (message streaming, ResultMessage parsing) on top of the same | ||
| end-to-end path through /v1/messages: full system prompt, tool definitions, and | ||
| an inline system-role message that the server must accept and dispatch to the | ||
| backing provider. | ||
|
|
||
| This runs LIVE against a real backend, not in replay mode. The SDK drives the | ||
| real CLI, which bakes the working directory, date, and platform into every | ||
| request body, so the request-body hashes the recording system keys on are not | ||
| reproducible across runs or machines; recording/replay is therefore not viable. | ||
|
|
||
| The test self-skips unless both the `claude-agent-sdk` package and the `claude` | ||
| binary are available, since the SDK requires the CLI at runtime. | ||
| """ | ||
|
|
||
| import asyncio | ||
| import importlib.util | ||
| import shutil | ||
|
|
||
| import pytest | ||
|
|
||
| CLAUDE_CLI = shutil.which("claude") | ||
| HAS_SDK = importlib.util.find_spec("claude_agent_sdk") is not None | ||
|
|
||
| pytestmark = pytest.mark.skipif( | ||
| CLAUDE_CLI is None or not HAS_SDK, | ||
| reason="claude-agent-sdk and the claude CLI must both be installed to run", | ||
| ) | ||
|
|
||
|
|
||
| def _run_query(prompt: str, base_url: str, model: str, cwd: str) -> list: | ||
| """Run a single Agent SDK query() to completion and return all messages.""" | ||
| from claude_agent_sdk import ClaudeAgentOptions, query | ||
|
|
||
| options = ClaudeAgentOptions( | ||
| model=model, | ||
| # Run from an isolated directory so the spawned CLI does not pick up | ||
| # repo-local context, which matters while permissions are bypassed. | ||
| cwd=cwd, | ||
| # Passed to the spawned CLI subprocess so it reaches OGX instead of | ||
| # api.anthropic.com. | ||
| env={ | ||
| "ANTHROPIC_BASE_URL": base_url, | ||
| "ANTHROPIC_API_KEY": "dummy", | ||
| "ANTHROPIC_MODEL": model, | ||
| }, | ||
| # The prompt is pure Q&A and triggers no tools, but bypass permissions | ||
| # so the non-interactive session can never stall on a permission prompt. | ||
| permission_mode="bypassPermissions", | ||
| ) | ||
|
|
||
| messages: list = [] | ||
|
|
||
| async def _collect() -> None: | ||
| async for message in query(prompt=prompt, options=options): | ||
| messages.append(message) | ||
|
|
||
| # Generous: the SDK drives the real CLI, which makes several large-context | ||
| # calls, and a small model on a CPU-only CI runner generates slowly (tens of | ||
| # seconds each). Bound it so a hung session fails loudly instead of riding | ||
| # the job timeout. | ||
| asyncio.run(asyncio.wait_for(_collect(), timeout=600)) | ||
| return messages | ||
|
|
||
|
|
||
| def test_claude_agent_sdk_smoke(messages_base_url, text_model_id, tmp_path): | ||
| """Claude Agent SDK completes a session against /v1/messages without error. | ||
|
|
||
| The smoke signal is integration health, not answer quality: the SDK drives a | ||
| full agentic session (system prompt, tools, inline system message) against | ||
| OGX, OGX routes it to the backing model, and the session terminates with a | ||
| successful ResultMessage. We deliberately do not assert on the model's text | ||
| output -- a small local model driving the Claude Code harness cannot be | ||
| relied on to produce a specific answer, but a regression like a rejected | ||
| system-role message (which would surface as a session error) is caught here. | ||
| """ | ||
| from claude_agent_sdk import ResultMessage | ||
|
|
||
| prompt = "What is the capital of France? Reply with only the city name and nothing else." | ||
| base_url = str(messages_base_url).rstrip("/") | ||
|
|
||
| messages = _run_query(prompt, base_url, text_model_id, cwd=str(tmp_path)) | ||
|
|
||
| results = [m for m in messages if isinstance(m, ResultMessage)] | ||
| assert results, f"Agent SDK session produced no ResultMessage; got: {[type(m).__name__ for m in messages]}" | ||
|
|
||
| result = results[-1] | ||
| assert result.subtype == "success" and not result.is_error, ( | ||
| f"Agent SDK session reported an error talking to /v1/messages: " | ||
| f"subtype={result.subtype} is_error={result.is_error} errors={result.errors}" | ||
| ) | ||
| # Confirm the request actually reached the backing model through /v1/messages. | ||
| model_usage = result.model_usage or {} | ||
| assert text_model_id in model_usage, ( | ||
| f"Expected model {text_model_id} in ResultMessage.model_usage; got: {list(model_usage)}" | ||
| ) | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I may be missing a reason this differs from the CLI smoke, but should the SDK path also run from
tmp_path?ClaudeAgentOptionshas acwdfield, and using it would avoid the spawned CLI picking up repo-local context whilebypassPermissionsis enabled.