test(messages): add live Claude Agent SDK smoke test against /v1/messages - #6010
Conversation
…ages Drive the upstream claude-agent-sdk Python package against a local OGX server so the SDK's session machinery exercises the Anthropic Messages path end to end. The SDK does not speak HTTP itself; it spawns the Claude Code CLI as a subprocess, so the test asserts on the parsed ResultMessage (successful subtype, no error, and the backing model present in model_usage) rather than on model text, which a small local model driving the Claude Code harness cannot be relied on to produce. The SDK drives the real CLI, which bakes the working directory, date, and platform into every request body, so request-body hashes are not reproducible and record/replay is not viable. The test therefore runs live against Ollama and self-skips unless both the claude-agent-sdk package and the claude binary are available. Add claude-agent-sdk to the test dependency group and a sibling CI workflow that provisions Ollama, installs the CLI, and runs the smoke test live. Signed-off-by: Charlie Doern <cdoern@redhat.com>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
leseb
left a comment
leseb
left a comment
There was a problem hiding this comment.
if this "The SDK does not speak HTTP itself — it spawns the Claude Code CLI as a subprocess and parses its streamed session output. " is true, can't we have a single test that uses just the SDK so this will validate both CLI and SDK at the same time? like reverting #5986?
skamenan7
left a comment
skamenan7
left a comment
There was a problem hiding this comment.
LGTM apart from two comments.
| """Run a single Agent SDK query() to completion and return all messages.""" | ||
| from claude_agent_sdk import ClaudeAgentOptions, query | ||
|
|
||
| options = ClaudeAgentOptions( |
There was a problem hiding this comment.
I may be missing a reason this differs from the CLI smoke, but should the SDK path also run from tmp_path? ClaudeAgentOptions has a cwd field, and using it would avoid the spawned CLI picking up repo-local context while bypassPermissions is enabled.
| {include-group = "test-common"}, | ||
| "autoevals", | ||
| "chromadb>=1.0.15", | ||
| "claude-agent-sdk>=0.2.0", # Drives the real Claude Code CLI in the Messages API SDK smoke test |
There was a problem hiding this comment.
The dependency wiring works, but it may be broader than needed. Since only this new live SDK smoke uses claude-agent-sdk, could we install it in this workflow or a narrower group instead of the shared test group?
Address review feedback on the Claude Agent SDK smoke test: Rename the merged integration-tests-messages-cli.yml to integration-tests-messages-clients.yml and run both the CLI and Agent SDK smoke tests from it. Both clients ultimately drive the same Claude Code CLI, so a single job shares the Ollama provisioning and CLI install and then runs both tests, rather than paying that setup twice across two workflows. Install claude-agent-sdk at workflow time instead of adding it to the shared test dependency group. Since CI syncs all groups, a group dependency would be installed for every integration-test job even though only this one live smoke test imports it; a workflow-scoped install keeps it narrow, mirroring how the CLI binary itself is installed. The test already self-skips when the package is absent. Run the Agent SDK session from an isolated tmp directory via ClaudeAgentOptions.cwd so the spawned CLI does not pick up repo-local context while permissions are bypassed. Signed-off-by: Charlie Doern <cdoern@redhat.com>
… into messages-claude-agent-sdk-smoke
|
▎ can't we have a single test that uses just the SDK... like reverting #5986? Combined instead of collapsed: both tests now run from one workflow (renamed to integration-tests-messages-clients.yml) — shared Ollama + CLI setup, run via -k '(test_claude_code_cli_smoke or test_claude_agent_sdk_smoke)'. No more duplicate setup or two workflows. Kept both because they assert different contracts: the CLI test checks claude -p --output-format json (no Python dep), the SDK test checks streamed message parsing (ResultMessage). They fail independently — if the claude-agent-sdk ↔ CLI version pairing breaks, the CLI canary still tells us whether OGX itself regressed. Extra cost is ~30s now that setup is shared. |
run-and-record-tests passes --pattern unquoted, so the parenthesized '(... or ...)' expression was word-split and the script rejected the bare 'or' token. Use the single shell word 'test_claude', which selects both test_claude_code_cli_smoke and test_claude_agent_sdk_smoke and survives the unquoted expansion. Signed-off-by: Charlie Doern <cdoern@redhat.com>
4 participants
What
Adds a live smoke test that drives the upstream
claude-agent-sdkPython package against a local OGX server, exercising the Anthropic Messages (/v1/messages) path end to end through a real client. This is the SDK companion to the Claude Code CLI smoke test merged in #5986 (RHAIENG-5377).The SDK does not speak HTTP itself — it spawns the Claude Code CLI as a subprocess and parses its streamed session output. So this covers a distinct client surface (the SDK's session machinery: message streaming,
ResultMessageparsing) on top of the same path: full system prompt, tool definitions, and an inline system-role message that the server must accept and dispatch to the backing provider.Why live, not record/replay
The SDK drives the real CLI, which bakes the working directory, date, and platform into every request body. The recording system keys on SHA256 of the request body, so those hashes are not reproducible across runs or machines — record/replay is not viable. The test runs live against Ollama and self-skips unless both the
claude-agent-sdkpackage and theclaudebinary are available.What it asserts
Integration health, not answer quality — a small local model driving the Claude Code harness cannot be relied on to produce specific text. The test checks the parsed
ResultMessage:subtype == "success"andis_error is Falsemodel_usageA regression like a rejected inline system-role message would surface as a session error and be caught here.
Changes
tests/integration/messages/test_claude_agent_sdk.py.github/workflows/integration-tests-messages-sdk.yml(provisions Ollama, installs the pinned CLI, runs the test live)claude-agent-sdk>=0.2.0added to thetestdependency group (locked to 0.2.87)Test plan
Ran live against local Ollama using the same CI command:
Result:
🤖 Generated with Claude Code