test(messages): add live Claude Agent SDK smoke test against /v1/messages

Drive the upstream claude-agent-sdk Python package against a local OGX
server so the SDK's session machinery exercises the Anthropic Messages
path end to end. The SDK does not speak HTTP itself; it spawns the Claude
Code CLI as a subprocess, so the test asserts on the parsed ResultMessage
(successful subtype, no error, and the backing model present in
model_usage) rather than on model text, which a small local model driving
the Claude Code harness cannot be relied on to produce.

The SDK drives the real CLI, which bakes the working directory, date, and
platform into every request body, so request-body hashes are not
reproducible and record/replay is not viable. The test therefore runs
live against Ollama and self-skips unless both the claude-agent-sdk
package and the claude binary are available.

Add claude-agent-sdk to the test dependency group and a sibling CI
workflow that provisions Ollama, installs the CLI, and runs the smoke
test live.

Signed-off-by: Charlie Doern <cdoern@redhat.com>

Author: cdoern

.github/workflows/README.md

@@ -18,6 +18,7 @@ OGX uses GitHub Actions for Continuous Integration (CI). Below is a table detail
 | Integration Responses, Conversations & Prompts Auth Tests | [integration-responses-conversations-auth-tests.yml](integration-responses-conversations-auth-tests.yml) | Run responses, conversations, and prompts auth tests with Kubernetes authentication |
 | SqlStore Integration Tests | [integration-sql-store-tests.yml](integration-sql-store-tests.yml) | Run the integration test suite with SqlStore |
 | Messages API - Claude Code CLI Smoke Test | [integration-tests-messages-cli.yml](integration-tests-messages-cli.yml) | Drive the real Claude Code CLI against /v1/messages (live, Ollama) |
+| Messages API - Claude Agent SDK Smoke Test | [integration-tests-messages-sdk.yml](integration-tests-messages-sdk.yml) | Drive the Claude Agent SDK against /v1/messages (live, Ollama) |
 | Integration Tests (Replay) | [integration-tests.yml](integration-tests.yml) | Run the integration test suites from tests/integration in replay mode |
 | Vector IO Integration Tests | [integration-vector-io-tests.yml](integration-vector-io-tests.yml) | Run the integration test suite with various VectorIO providers |
 | OpenAPI Generator SDK Validation | [openapi-generator-validation.yml](openapi-generator-validation.yml) | Validate OpenAPI Generator SDK generation |

.github/workflows/integration-tests-messages-sdk.yml

@@ -0,0 +1,85 @@
+name: Messages API - Claude Agent SDK Smoke Test
+
+run-name: Drive the Claude Agent SDK against /v1/messages (live, Ollama)
+
+on:
+  push:
+    branches:
+      - main
+      - 'release-[0-9]+.[0-9]+.x'
+  pull_request:
+    branches:
+      - main
+      - 'release-[0-9]+.[0-9]+.x'
+    types: [opened, synchronize, reopened]
+    paths:
+      - 'src/ogx/providers/inline/messages/**'
+      - 'src/ogx_api/**'
+      - 'tests/integration/messages/**'
+      - 'uv.lock'
+      - 'pyproject.toml'
+      - '.github/workflows/integration-tests-messages-sdk.yml'
+      - '.github/actions/setup-test-environment/action.yml'
+      - '.github/actions/run-and-record-tests/action.yml'
+      - 'scripts/integration-tests.sh'
+  merge_group:
+    branches:
+      - main
+      - 'release-[0-9]+.[0-9]+.x'
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/main' && github.run_id || github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+env:
+  # Pinned for reproducibility. The SDK spawns this CLI, which bakes the cwd,
+  # date, and platform into every request body, so its traffic cannot be
+  # recorded/replayed; this test runs live against Ollama instead. The
+  # claude-agent-sdk Python package itself is pinned via uv.lock.
+  CLAUDE_CODE_CLI_VERSION: '2.1.159'
+
+jobs:
+  claude-agent-sdk-smoke:
+    name: Claude Agent SDK smoke (ollama, live)
+    runs-on: ubuntu-latest
+    # CPU-only runners generate slowly; the live SDK session can take several
+    # minutes against the Ollama model. Keep ample headroom over the test's own
+    # 600s session timeout plus environment setup.
+    timeout-minutes: 30
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+
+      # Live mode (not replay) so the Ollama backend is provisioned and the SDK
+      # session reaches a real model. setup-runner's `uv sync --all-groups`
+      # installs the claude-agent-sdk Python package from the test group.
+      - name: Setup test environment
+        uses: ./.github/actions/setup-test-environment
+        with:
+          python-version: '3.12'
+          client-version: 'latest'
+          setup: 'ollama'
+          suite: 'messages'
+          inference-mode: 'live'
+
+      # The SDK requires the Claude Code CLI at runtime: it spawns the binary as
+      # a subprocess and parses its streamed output.
+      - name: Install Claude Code CLI
+        run: |
+          curl -fsSL https://claude.ai/install.sh | bash -s -- "${CLAUDE_CODE_CLI_VERSION}"
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+          "$HOME/.local/bin/claude" --version
+
+      - name: Run Claude Agent SDK smoke test
+        uses: ./.github/actions/run-and-record-tests
+        with:
+          stack-config: 'server:ci-tests'
+          setup: 'ollama'
+          suite: 'messages'
+          inference-mode: 'live'
+          pattern: 'test_claude_agent_sdk_smoke'

pyproject.toml

@@ -216,6 +216,7 @@ test = [
     {include-group = "test-common"},
     "autoevals",
     "chromadb>=1.0.15",
+    "claude-agent-sdk>=0.2.0",  # Drives the real Claude Code CLI in the Messages API SDK smoke test
     "datasets>=4.0.0",
     "elasticsearch>=8.16.0, <9.0.0",
     "google-genai>=2.3.0",

tests/integration/messages/test_claude_agent_sdk.py

@@ -0,0 +1,105 @@
+# Copyright (c) The OGX Contributors.
+# All rights reserved.
+#
+# This source code is licensed under the terms described in the LICENSE file in
+# the root directory of this source tree.
+
+"""Smoke test: drive the Claude Agent SDK against the OGX Messages API.
+
+Points the SDK at a local OGX server and runs a single prompt through the
+upstream `claude-agent-sdk` Python package (github.com/anthropics/
+claude-agent-sdk-python). The SDK does not speak HTTP itself: it spawns the
+Claude Code CLI as a subprocess and parses its streamed session output. This
+exercises a different client surface than the CLI smoke test -- the SDK's
+session machinery (message streaming, ResultMessage parsing) on top of the same
+end-to-end path through /v1/messages: full system prompt, tool definitions, and
+an inline system-role message that the server must accept and dispatch to the
+backing provider.
+
+This runs LIVE against a real backend, not in replay mode. The SDK drives the
+real CLI, which bakes the working directory, date, and platform into every
+request body, so the request-body hashes the recording system keys on are not
+reproducible across runs or machines; recording/replay is therefore not viable.
+
+The test self-skips unless both the `claude-agent-sdk` package and the `claude`
+binary are available, since the SDK requires the CLI at runtime.
+"""
+
+import asyncio
+import importlib.util
+import shutil
+
+import pytest
+
+CLAUDE_CLI = shutil.which("claude")
+HAS_SDK = importlib.util.find_spec("claude_agent_sdk") is not None
+
+pytestmark = pytest.mark.skipif(
+    CLAUDE_CLI is None or not HAS_SDK,
+    reason="claude-agent-sdk and the claude CLI must both be installed to run",
+)
+
+
+def _run_query(prompt: str, base_url: str, model: str) -> list:
+    """Run a single Agent SDK query() to completion and return all messages."""
+    from claude_agent_sdk import ClaudeAgentOptions, query
+
+    options = ClaudeAgentOptions(
+        model=model,
+        # Passed to the spawned CLI subprocess so it reaches OGX instead of
+        # api.anthropic.com.
+        env={
+            "ANTHROPIC_BASE_URL": base_url,
+            "ANTHROPIC_API_KEY": "dummy",
+            "ANTHROPIC_MODEL": model,
+        },
+        # The prompt is pure Q&A and triggers no tools, but bypass permissions
+        # so the non-interactive session can never stall on a permission prompt.
+        permission_mode="bypassPermissions",
+    )
+
+    messages: list = []
+
+    async def _collect() -> None:
+        async for message in query(prompt=prompt, options=options):
+            messages.append(message)
+
+    # Generous: the SDK drives the real CLI, which makes several large-context
+    # calls, and a small model on a CPU-only CI runner generates slowly (tens of
+    # seconds each). Bound it so a hung session fails loudly instead of riding
+    # the job timeout.
+    asyncio.run(asyncio.wait_for(_collect(), timeout=600))
+    return messages
+
+
+def test_claude_agent_sdk_smoke(messages_base_url, text_model_id):
+    """Claude Agent SDK completes a session against /v1/messages without error.
+
+    The smoke signal is integration health, not answer quality: the SDK drives a
+    full agentic session (system prompt, tools, inline system message) against
+    OGX, OGX routes it to the backing model, and the session terminates with a
+    successful ResultMessage. We deliberately do not assert on the model's text
+    output -- a small local model driving the Claude Code harness cannot be
+    relied on to produce a specific answer, but a regression like a rejected
+    system-role message (which would surface as a session error) is caught here.
+    """
+    from claude_agent_sdk import ResultMessage
+
+    prompt = "What is the capital of France? Reply with only the city name and nothing else."
+    base_url = str(messages_base_url).rstrip("/")
+
+    messages = _run_query(prompt, base_url, text_model_id)
+
+    results = [m for m in messages if isinstance(m, ResultMessage)]
+    assert results, f"Agent SDK session produced no ResultMessage; got: {[type(m).__name__ for m in messages]}"
+
+    result = results[-1]
+    assert result.subtype == "success" and not result.is_error, (
+        f"Agent SDK session reported an error talking to /v1/messages: "
+        f"subtype={result.subtype} is_error={result.is_error} errors={result.errors}"
+    )
+    # Confirm the request actually reached the backing model through /v1/messages.
+    model_usage = result.model_usage or {}
+    assert text_model_id in model_usage, (
+        f"Expected model {text_model_id} in ResultMessage.model_usage; got: {list(model_usage)}"
+    )