test(messages): add live Claude Code CLI smoke test against /v1/messages (#5986)

## What

Adds a smoke test that drives the **real** `@anthropic-ai/claude-code`
CLI against a local OGX server, verifying the Messages API works
end-to-end with the actual client. The CLI emits its full system prompt,
tool definitions, and an inline `system`-role message — all of which OGX
must accept and dispatch to the backing provider.

This is the upstream automated coverage for the Claude Code ↔
`/v1/messages` integration (RHAIENG-5376).

## Why live, not replay

The recording/replay system keys recordings on a SHA256 of the request
body. The Claude Code CLI bakes the **working directory, the date, and
the platform** into every request (its system prompt includes all
three), so those hashes are **not reproducible** across runs or
machines:

- random `tmp_path` cwd → different hash every run
- date changes daily → recordings rot
- recorded on macOS, CI runs Linux → platform string differs

So hash-based replay is fundamentally unworkable for the real CLI. The
test runs **live against Ollama** (native passthrough) instead, in a
dedicated workflow that provisions Ollama. OpenAI/translation is
intentionally excluded to avoid live OpenAI calls on every PR.

## What it asserts

Integration health, not answer quality. A small local model driving the
Claude Code harness can't be relied on to produce a specific reply, but
the test confirms via `--output-format json` that:

- the CLI exits 0,
- the session completed with no API error (`is_error: false`, `subtype:
success`),
- the request actually reached the backing model (`modelUsage` contains
the model id).

A regression such as a rejected `system`-role message (cf. #5982)
surfaces as a CLI API error and is caught here. The test self-skips
unless the `claude` binary is on PATH.

> Depends on #5982's fix (merged), which taught `/v1/messages` to accept
inline `system`-role messages.

## Test plan

- Ran live locally against Ollama (`ollama/llama3.2:3b-instruct-fp16`,
native passthrough): **passed**.
  ```
  ./scripts/integration-tests.sh --stack-config server:ci-tests \
    --setup ollama --suite messages --inference-mode live \
    --pattern test_claude_code_cli_smoke
  ```
- `pre-commit run` — ruff, mypy, generate-ci-docs, SHA-pinned-actions
checks pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Charlie Doern <cdoern@redhat.com>

Author: cdoern

.github/workflows/README.md

@@ -17,6 +17,7 @@ OGX uses GitHub Actions for Continuous Integration (CI). Below is a table detail
 | Integration Auth Tests | [integration-auth-tests.yml](integration-auth-tests.yml) | Run the integration test suite with Kubernetes authentication |
 | Integration Responses, Conversations & Prompts Auth Tests | [integration-responses-conversations-auth-tests.yml](integration-responses-conversations-auth-tests.yml) | Run responses, conversations, and prompts auth tests with Kubernetes authentication |
 | SqlStore Integration Tests | [integration-sql-store-tests.yml](integration-sql-store-tests.yml) | Run the integration test suite with SqlStore |
+| Messages API - Claude Code CLI Smoke Test | [integration-tests-messages-cli.yml](integration-tests-messages-cli.yml) | Drive the real Claude Code CLI against /v1/messages (live, Ollama) |
 | Integration Tests (Replay) | [integration-tests.yml](integration-tests.yml) | Run the integration test suites from tests/integration in replay mode |
 | Vector IO Integration Tests | [integration-vector-io-tests.yml](integration-vector-io-tests.yml) | Run the integration test suite with various VectorIO providers |
 | OpenAPI Generator SDK Validation | [openapi-generator-validation.yml](openapi-generator-validation.yml) | Validate OpenAPI Generator SDK generation |

.github/workflows/integration-tests-messages-cli.yml

@@ -0,0 +1,82 @@
+name: Messages API - Claude Code CLI Smoke Test
+
+run-name: Drive the real Claude Code CLI against /v1/messages (live, Ollama)
+
+on:
+  push:
+    branches:
+      - main
+      - 'release-[0-9]+.[0-9]+.x'
+  pull_request:
+    branches:
+      - main
+      - 'release-[0-9]+.[0-9]+.x'
+    types: [opened, synchronize, reopened]
+    paths:
+      - 'src/ogx/providers/inline/messages/**'
+      - 'src/ogx_api/**'
+      - 'tests/integration/messages/**'
+      - 'uv.lock'
+      - 'pyproject.toml'
+      - '.github/workflows/integration-tests-messages-cli.yml'
+      - '.github/actions/setup-test-environment/action.yml'
+      - '.github/actions/run-and-record-tests/action.yml'
+      - 'scripts/integration-tests.sh'
+  merge_group:
+    branches:
+      - main
+      - 'release-[0-9]+.[0-9]+.x'
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/main' && github.run_id || github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+env:
+  # Pinned for reproducibility. The CLI bakes the cwd, date, and platform into
+  # every request body, so its traffic cannot be recorded/replayed; this test
+  # runs live against Ollama instead. Bumping only changes the client behavior
+  # under test, not any committed recordings.
+  CLAUDE_CODE_CLI_VERSION: '2.1.159'
+
+jobs:
+  claude-code-cli-smoke:
+    name: Claude Code CLI smoke (ollama, live)
+    runs-on: ubuntu-latest
+    # CPU-only runners generate slowly; the live CLI session can take several
+    # minutes against the Ollama model. Keep ample headroom over the test's own
+    # 600s subprocess timeout plus environment setup.
+    timeout-minutes: 30
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+
+      # Live mode (not replay) so the Ollama backend is provisioned and the
+      # real CLI traffic reaches a real model.
+      - name: Setup test environment
+        uses: ./.github/actions/setup-test-environment
+        with:
+          python-version: '3.12'
+          client-version: 'latest'
+          setup: 'ollama'
+          suite: 'messages'
+          inference-mode: 'live'
+
+      - name: Install Claude Code CLI
+        run: |
+          curl -fsSL https://claude.ai/install.sh | bash -s -- "${CLAUDE_CODE_CLI_VERSION}"
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+          "$HOME/.local/bin/claude" --version
+
+      - name: Run Claude Code CLI smoke test
+        uses: ./.github/actions/run-and-record-tests
+        with:
+          stack-config: 'server:ci-tests'
+          setup: 'ollama'
+          suite: 'messages'
+          inference-mode: 'live'
+          pattern: 'test_claude_code_cli_smoke'

tests/integration/messages/test_claude_code_cli.py

@@ -0,0 +1,88 @@
+# Copyright (c) The OGX Contributors.
+# All rights reserved.
+#
+# This source code is licensed under the terms described in the LICENSE file in
+# the root directory of this source tree.
+
+"""Smoke test: drive the Claude Code CLI against the OGX Messages API.
+
+Points ANTHROPIC_BASE_URL at a local OGX server and runs a single prompt
+through the upstream @anthropic-ai/claude-code CLI. This exercises the real
+client end-to-end against /v1/messages: the CLI emits its full system prompt,
+tool definitions, and an inline system-role message, all of which the server
+must accept and dispatch to the backing provider before the model's reply
+comes back.
+
+This runs LIVE against a real backend, not in replay mode. The CLI bakes the
+working directory, date, and platform into every request body, so the
+request-body hashes the recording system keys on are not reproducible across
+runs or machines; recording/replay is therefore not viable for the real CLI.
+
+The test self-skips unless the `claude` binary is on PATH.
+"""
+
+import json
+import os
+import shutil
+import subprocess
+
+import pytest
+
+CLAUDE_CLI = shutil.which("claude")
+
+pytestmark = pytest.mark.skipif(
+    CLAUDE_CLI is None,
+    reason="claude-code CLI not installed; install @anthropic-ai/claude-code to run",
+)
+
+
+def test_claude_code_cli_smoke(messages_base_url, text_model_id, tmp_path):
+    """Claude Code CLI completes a request against /v1/messages without error.
+
+    The smoke signal is integration health, not answer quality: the CLI sends
+    its full agentic payload (system prompt, tools, inline system message) to
+    OGX, OGX routes it to the backing model, and the session completes with no
+    API error. We deliberately do not assert on the model's text output -- a
+    small local model driving the Claude Code harness cannot be relied on to
+    produce a specific answer, but a regression like a rejected system-role
+    message (which would surface as an API error) is caught here.
+    """
+    prompt = "What is the capital of France? Reply with only the city name and nothing else."
+
+    env = {
+        **os.environ,
+        "ANTHROPIC_BASE_URL": str(messages_base_url).rstrip("/"),
+        "ANTHROPIC_API_KEY": "dummy",
+        "ANTHROPIC_MODEL": text_model_id,
+    }
+
+    result = subprocess.run(
+        [
+            CLAUDE_CLI,
+            "--print",
+            "--output-format",
+            "json",
+            "--dangerously-skip-permissions",
+            prompt,
+        ],
+        cwd=tmp_path,
+        env=env,
+        capture_output=True,
+        text=True,
+        # Generous: the real CLI makes several large-context calls and a small
+        # model on a CPU-only CI runner generates slowly (tens of seconds each).
+        timeout=600,
+    )
+
+    assert result.returncode == 0, (
+        f"Failed to run Claude Code CLI: exit {result.returncode}\nstdout:\n{result.stdout}\nstderr:\n{result.stderr}"
+    )
+
+    data = json.loads(result.stdout)
+    assert data.get("is_error") is False and data.get("subtype") == "success", (
+        f"Claude Code CLI reported an error talking to /v1/messages: {json.dumps(data, indent=2)}"
+    )
+    # Confirm the request actually reached the backing model through /v1/messages.
+    assert text_model_id in data.get("modelUsage", {}), (
+        f"Expected model {text_model_id} in CLI modelUsage; got: {json.dumps(data.get('modelUsage', {}), indent=2)}"
+    )