Skip to content

Commit 162f9b8

Browse files
mergify[bot]mergify[bot]
authored
Merge branch 'main' into dependabot/github_actions/github/codeql-action/upload-sarif-4.37.6
2 parents a2d30c0 + 2d3673c commit 162f9b8

3 files changed

Lines changed: 233 additions & 21 deletions

File tree

‎tests/README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,11 @@ Some upstream tests are currently skipped, grouped by reason:
8181
- `test_tool_with_complex_schema`
8282
- `test_tool_without_schema`
8383

84+
**Structured output and tool-calling tests timing out on CPU:**
85+
- `test_openai_chat_completion_structured_output`
86+
- `test_simple_tool_call`
87+
- `test_streaming_tool_calls`
88+
8489
**Requires vLLM >= v0.12.0** ([ogx/ogx#4984](https://github.com/ogx/ogx/issues/4984)):
8590
- `test_openai_completion_guided_choice`
8691

‎tests/functional/notebooks/test_async_client.ipynb‎

Lines changed: 224 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,224 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Scenario: Async OpenAI Client Coverage\n",
8+
"\n",
9+
"Validates that `AsyncOpenAI` from `openai` operates properly with async/await methods against the OGX server:\n",
10+
"- **Server Health:** `client.get(\"/health\")`\n",
11+
"- **Models:** `client.models.list()`\n",
12+
"- **Responses API:** `client.responses.create()`\n",
13+
"- **Chat Completions:** `client.chat.completions.create()`\n",
14+
"- **Embeddings:** `client.embeddings.create()`"
15+
]
16+
},
17+
{
18+
"cell_type": "markdown",
19+
"metadata": {},
20+
"source": [
21+
"## Setup & Initialization\n",
22+
"\n",
23+
"Load configuration from environment variables and initialize `AsyncOpenAI`."
24+
]
25+
},
26+
{
27+
"cell_type": "code",
28+
"execution_count": null,
29+
"metadata": {},
30+
"outputs": [],
31+
"source": [
32+
"import os\n",
33+
"from openai import AsyncOpenAI\n",
34+
"from scripts.helpers import response_text\n",
35+
"\n",
36+
"base_url = os.environ.get(\"BASE_URL\", \"http://localhost:8321\")\n",
37+
"model = os.environ.get(\"INFERENCE_MODEL\")\n",
38+
"embedding_model = os.environ.get(\"EMBEDDING_MODEL\")\n",
39+
"embedding_dimension = int(os.environ.get(\"EMBEDDING_DIMENSION\", \"768\"))\n",
40+
"\n",
41+
"assert base_url, \"BASE_URL must be set\"\n",
42+
"assert model, \"INFERENCE_MODEL must be set\"\n",
43+
"\n",
44+
"openai_base_url = base_url.rstrip(\"/\")\n",
45+
"openai_base_url = (\n",
46+
" openai_base_url if openai_base_url.endswith(\"/v1\") else openai_base_url + \"/v1\"\n",
47+
")\n",
48+
"\n",
49+
"client = AsyncOpenAI(api_key=\"no-key-needed\", base_url=openai_base_url)"
50+
]
51+
},
52+
{
53+
"cell_type": "markdown",
54+
"metadata": {},
55+
"source": [
56+
"## Server Health Check (`/v1/health`)\n",
57+
"\n",
58+
"Verify `await client.get(\"/health\")` completes and returns status OK."
59+
]
60+
},
61+
{
62+
"cell_type": "code",
63+
"execution_count": null,
64+
"metadata": {},
65+
"outputs": [],
66+
"source": [
67+
"health_resp = await client.get(\"/health\", cast_to=object)\n",
68+
"assert health_resp is not None, \"Expected health response\"\n",
69+
"health_status = (\n",
70+
" health_resp.get(\"status\")\n",
71+
" if isinstance(health_resp, dict)\n",
72+
" else getattr(health_resp, \"status\", str(health_resp))\n",
73+
")\n",
74+
"assert health_status == \"OK\" or health_resp == \"OK\", (\n",
75+
" f\"Health check failed or unexpected response: {health_resp!r}\"\n",
76+
")"
77+
]
78+
},
79+
{
80+
"cell_type": "markdown",
81+
"metadata": {},
82+
"source": [
83+
"## Models List (`models.list`)\n",
84+
"\n",
85+
"Verify `await client.models.list()` returns the list of registered models."
86+
]
87+
},
88+
{
89+
"cell_type": "code",
90+
"execution_count": null,
91+
"metadata": {},
92+
"outputs": [],
93+
"source": [
94+
"models_resp = await client.models.list()\n",
95+
"assert models_resp is not None, \"Expected models response\"\n",
96+
"model_ids = [m.id for m in models_resp.data]\n",
97+
"\n",
98+
"assert len(model_ids) > 0, \"Expected at least one model in list\"\n",
99+
"assert any(model in mid or mid in model for mid in model_ids), (\n",
100+
" f\"Configured model {model!r} not found in model IDs: {model_ids}\"\n",
101+
")"
102+
]
103+
},
104+
{
105+
"cell_type": "markdown",
106+
"metadata": {},
107+
"source": [
108+
"## Responses API (`responses.create`)\n",
109+
"\n",
110+
"Verify async responses creation with `await client.responses.create()`."
111+
]
112+
},
113+
{
114+
"cell_type": "code",
115+
"execution_count": null,
116+
"metadata": {},
117+
"outputs": [],
118+
"source": [
119+
"response = await client.responses.create(\n",
120+
" model=model,\n",
121+
" input=\"Explain disestablishmentarianism to a smart five year old.\",\n",
122+
")\n",
123+
"assert response is not None, \"Expected response object\"\n",
124+
"assert getattr(response, \"status\", \"completed\") == \"completed\"\n",
125+
"out_text = getattr(response, \"output_text\", None) or response_text(response)\n",
126+
"assert out_text and len(out_text.strip()) > 0, (\n",
127+
" \"Expected non-empty output text from responses.create\"\n",
128+
")"
129+
]
130+
},
131+
{
132+
"cell_type": "markdown",
133+
"metadata": {},
134+
"source": [
135+
"## Chat Completions (`chat.completions.create`)\n",
136+
"\n",
137+
"Verify async chat completion with `await client.chat.completions.create()`."
138+
]
139+
},
140+
{
141+
"cell_type": "code",
142+
"execution_count": null,
143+
"metadata": {},
144+
"outputs": [],
145+
"source": [
146+
"chat_resp = await client.chat.completions.create(\n",
147+
" model=model,\n",
148+
" messages=[{\"role\": \"user\", \"content\": \"Reply with exactly one word: Hello\"}],\n",
149+
" temperature=0.0,\n",
150+
")\n",
151+
"assert chat_resp is not None, \"Expected chat completion response\"\n",
152+
"assert hasattr(chat_resp, \"choices\") and len(chat_resp.choices) > 0, (\n",
153+
" \"Expected non-empty choices\"\n",
154+
")\n",
155+
"content = chat_resp.choices[0].message.content\n",
156+
"assert content and len(content.strip()) > 0, \"Expected non-empty message content\""
157+
]
158+
},
159+
{
160+
"cell_type": "markdown",
161+
"metadata": {},
162+
"source": [
163+
"## Embeddings (`embeddings.create`)\n",
164+
"\n",
165+
"Verify async embedding creation with `await client.embeddings.create()` if an embedding model is configured."
166+
]
167+
},
168+
{
169+
"cell_type": "code",
170+
"execution_count": null,
171+
"metadata": {},
172+
"outputs": [],
173+
"source": [
174+
"if embedding_model:\n",
175+
" emb_resp = await client.embeddings.create(\n",
176+
" model=embedding_model,\n",
177+
" input=\"Async client embedding test\",\n",
178+
" )\n",
179+
" assert emb_resp is not None, \"Expected embedding response\"\n",
180+
" assert hasattr(emb_resp, \"data\") and len(emb_resp.data) > 0, (\n",
181+
" \"Expected embedding data\"\n",
182+
" )\n",
183+
" vector = emb_resp.data[0].embedding\n",
184+
" assert len(vector) == embedding_dimension, (\n",
185+
" f\"Expected vector dimension {embedding_dimension}, got {len(vector)}\"\n",
186+
" )\n",
187+
"else:\n",
188+
" print(\"EMBEDDING_MODEL not set, skipping async embedding test\")\n",
189+
" assert True"
190+
]
191+
},
192+
{
193+
"cell_type": "markdown",
194+
"metadata": {},
195+
"source": [
196+
"## Context Manager & Cleanup\n",
197+
"\n",
198+
"Verify that `AsyncOpenAI` works as an async context manager and closes properly."
199+
]
200+
},
201+
{
202+
"cell_type": "code",
203+
"execution_count": null,
204+
"metadata": {},
205+
"outputs": [],
206+
"source": [
207+
"async with AsyncOpenAI(\n",
208+
" api_key=\"no-key-needed\", base_url=openai_base_url\n",
209+
") as async_client:\n",
210+
" models = await async_client.models.list()\n",
211+
" assert models is not None\n",
212+
"\n",
213+
"await client.close()"
214+
]
215+
}
216+
],
217+
"metadata": {
218+
"language_info": {
219+
"name": "python"
220+
}
221+
},
222+
"nbformat": 4,
223+
"nbformat_minor": 4
224+
}

‎tests/run_integration_tests.sh‎

Lines changed: 4 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -62,27 +62,10 @@ function run_integration_tests() {
6262
# truncated without validation, masking the issue.
6363
# test_openai_completion_logprobs{,_streaming}: upstream schema defines logprobs as bool, should be int https://github.com/llamastack/llama-stack/issues/5253
6464
# test_openai_chat_completion_structured_output, test_simple_tool_call, test_streaming_tool_calls:
65-
# These tests time out when running against Qwen3.5-0.8B on CPU. The upstream
66-
# test fixtures hardcode a 30s timeout on the OpenAI client and default to 30s
67-
# on the OGX client (via OGX_CLIENT_TIMEOUT). Structured output and tool calling
68-
# require constrained decoding which is significantly slower on CPU, causing
69-
# requests to exceed the 30s limit. The timeouts are set upstream in
70-
# tests/integration/fixtures/common.py and cannot be overridden from our side
71-
# for the OpenAI client path.
72-
# test_openai_chat_completion_streaming, test_openai_chat_completion_streaming_with_n:
73-
# The ogx_open_client SDK serializes timeout=120 into the JSON request body
74-
# (unlike the OpenAI SDK which treats it as an HTTP client timeout). The Vertex AI
75-
# provider passes model_extra directly to Google's GenerateContentConfig which has
76-
# extra="forbid", causing a 400 error. Only affects the client_with_models
77-
# parametrization; the openai_client variant still tests streaming successfully.
78-
# test_inference_store_tool_calls: the ogx_open_client SDK types
79-
# OpenAIChoiceDelta.tool_calls as List[ChatCompletionMessageToolCall] (non-streaming
80-
# model with required fields) instead of List[ChoiceDeltaToolCall] (streaming model
81-
# with optional fields). When Gemini streams tool calls across multiple chunks,
82-
# continuation chunks lack required fields, deserialization fails, and the SDK
83-
# silently returns a raw dict instead of a typed object, causing AttributeError
84-
# on chunk.id access. Only affects client_with_models; openai_client passes.
85-
SKIP_TESTS="test_text_chat_completion_tool_calling_tools_not_in_request or test_text_chat_completion_structured_output or test_text_chat_completion_non_streaming or test_openai_chat_completion_non_streaming or test_openai_chat_completion_with_tool_choice_none or test_openai_chat_completion_with_tools or test_openai_format_preserves_complex_schemas or test_multiple_tools_with_different_schemas or test_tool_with_complex_schema or test_tool_without_schema or test_openai_completion_guided_choice or test_openai_embeddings_with_dimensions or test_openai_embeddings_with_encoding_format_base64 or test_openai_completion_logprobs or test_openai_completion_logprobs_streaming or test_openai_chat_completion_structured_output or test_simple_tool_call or test_streaming_tool_calls or test_openai_chat_completion_streaming or test_openai_chat_completion_streaming_with_n or test_inference_store_tool_calls"
65+
# These tests time out when running against Qwen3.5-0.8B on CPU in CI. Structured output
66+
# and tool calling require constrained decoding which is significantly slower on CPU,
67+
# exceeding the 30s fixture timeout.
68+
SKIP_TESTS="test_text_chat_completion_tool_calling_tools_not_in_request or test_text_chat_completion_structured_output or test_text_chat_completion_non_streaming or test_openai_chat_completion_non_streaming or test_openai_chat_completion_with_tool_choice_none or test_openai_chat_completion_with_tools or test_openai_format_preserves_complex_schemas or test_multiple_tools_with_different_schemas or test_tool_with_complex_schema or test_tool_without_schema or test_openai_completion_guided_choice or test_openai_embeddings_with_dimensions or test_openai_embeddings_with_encoding_format_base64 or test_openai_completion_logprobs or test_openai_completion_logprobs_streaming or test_openai_chat_completion_structured_output or test_simple_tool_call or test_streaming_tool_calls"
8669

8770
# Dynamically determine the path to config.yaml from the original script directory
8871
STACK_CONFIG_PATH="$SCRIPT_DIR/../distribution/config.yaml"

0 commit comments

Comments
 (0)