chore(ci): change CI to use ogx_open_client by aegeiger · Pull Request #6016 · ogx-ai/ogx

aegeiger · 2026-06-04T07:33:16Z

What does this PR do?

Fixes multiple integration test failures when running with the OpenAPI-generated SDK (ogx_open_client) instead of the Stainless SDK (ogx_client). This PR switches the integration test suite to use the new SDK and resolves all compatibility issues discovered during the transition.

Changes by commit:

fix(ci): Fix the OpenAPI SDK publish workflow to use OIDC trusted publishing instead of unconfigured secret-based auth.
chore(client-sdks): Switch all integration tests and the library client from ogx_client to ogx_open_client. Add fallback import chains for backward compatibility. Update model class references in test_rerank.py and adapt test_openai_vector_store_file_contents_with_extra_query for the new SDK's direct query parameter convention (vs. the OpenAI SDK's extra_query).
fix(client-sdks): Fix dict serialization in multipart form requests. The generated SDK was sending dicts as JSON strings, but the OGX server expects bracket-notation (key[subkey]=value). Affects expires_after and similar nested fields on file upload.
fix(client-sdks): Add a ChoiceDeltaToolCall streaming delta variant in the OpenAPI spec. The spec previously used the same ChatCompletionMessageToolCall schema (all fields required, no index) for both complete responses and streaming deltas, but streaming deltas on the wire include index and omit id/type/function on continuation chunks. This formalizes the de-facto protocol without changing anything on the wire.
fix(client-sdks): Fix enum deserialization in anyOf validators. The generated actual_instance_must_validate_anyof used isinstance() to check variant types, which failed for str,Enum types (e.g., OpenAIResponseInputToolChoiceMode) because a bare string like "auto" is not an isinstance of the enum class. Add a fallback that tries constructing the type from str/int values.
fix(tests): Fix four bugs in vision inference integration tests: guard against None in streaming delta.content, fix multi_image_data fixture reference (was using the function instead of the resolved value), fix wrong streaming API call (chunk.event.delta.text → chunk.choices[0].delta.content), and update stale GitHub image URL.

Test Plan

Integration tests were run against a live OpenAI backend with the ci-tests distribution:

# Ensure the generated SDK is installed with all template fixes
cp -r client-sdks/openapi/sdks/python/ogx_open_client/* \
  .venv/lib64/python3.12/site-packages/ogx_open_client/

# Run the full base test suite
uv run --no-sync ./scripts/integration-tests.sh \
  --stack-config server:ci-tests --setup gpt --suite base --inference-mode live

Results: 22 failures (down from 29 baseline), 0 regressions.

All 22 remaining failures are pre-existing and unrelated:

16x ranker/hybrid search returning empty results (keyword indexing issue across backends)
6x legacy /v1/completions endpoint 404 (not served by the ci-tests distribution with gpt setup — not a real CI configuration)

Tests fixed by this PR (7 resolved):

Test	Root cause	Fix
`test_openai_vector_store_file_contents_with_extra_query[client_with_models-*]` (x6 backends)	Generated SDK rejects `extra_query` kwarg	`_get_file_content` helper adapts params per client type
`test_inference_store_tool_calls[client_with_models-True]`	Streaming tool call chunks deserialized as raw dicts	`ChoiceDeltaToolCall` schema with relaxed fields
`test_list_response_input_items[client_with_models]`	`tool_choice: "auto"` fails anyOf enum validation	Try enum construction for str/int in validator
`test_image_chat_completion_multiple_images[True]`	`None` delta.content on streaming chunks	`or ""` guard
`test_image_chat_completion_multiple_images[False]`	`multi_image_data` fixture function used as value	Changed to `multi_image_url`
`test_image_chat_completion_streaming`	`.lower()` called on `None` delta.content	`if delta_content:` guard
`test_openai_chat_completion_image_url`	Stale GitHub URL (`meta-llama/ogx`)	Updated to `ogx-ai/ogx`

Replace token-based authentication (PYPI_API_TOKEN, TEST_PYPI_API_TOKEN) with OIDC trusted publishing via pypa/gh-action-pypi-publish, matching the approach used by the main pypi.yml workflow. Split the single publish-sdk job into three jobs: - build-sdk: generates and builds the package - publish-testpypi: publishes to TestPyPI (environment: testpypi) - publish-pypi: publishes to production PyPI (no environment) The previous workflow referenced secrets that were never configured and used a pypi-production environment that did not exist in the repository. Signed-off-by: E Geiger <egeiger@redhat.com>

…i-generated SDK Replace ogx_client (stainless-generated) with ogx_open_client (openapi-generated) as the SDK used by integration tests and the library client. Add fallback import chains in library_client.py and api_recorder.py for backward compatibility with the stainless package. Update test_rerank.py for model class names in the new SDK. Adapt test_openai_vector_store_file_contents_with_extra_query to pass include_embeddings/include_metadata as direct parameters when using the generated SDK, which exposes them as first-class query parameters rather than requiring the OpenAI SDK extra_query convention. Signed-off-by: E Geiger <egeiger@redhat.com>

There are two valid ways to serialize a dict in multipart requests: as a JSON string, or as bracket-notation key-value pairs (e.g. expires_after[anchor]=created_at&expires_after[seconds]=3600). The OGX server only supports the bracket-notation form. Fix the generated SDK REST client to use bracket-notation flattening for dict values in multipart form data instead of json.dumps. Signed-off-by: E Geiger <egeiger@redhat.com>

…zation The OpenAPI spec used ChatCompletionMessageToolCall for both complete responses and streaming deltas. However, on the wire streaming deltas include an index field and omit id/type/function on continuation chunks. The spec did not reflect this de-facto protocol, causing the generated SDK to fail when deserializing streaming tool call chunks. Formalize the existing streaming wire format by creating a separate ChoiceDeltaToolCall schema: index is required for correlating chunks, while id, type, and function are optional. Point OpenAIChoiceDelta tool_calls to the new schema. This does not change anything on the wire, it aligns the spec with what was already being sent. Signed-off-by: E Geiger <egeiger@redhat.com>

…str/int values The anyOf validator in generated SDK models used isinstance() to check if a value matched a variant type. This failed for str,Enum types like OpenAIResponseInputToolChoiceMode because a bare string like "auto" is not an isinstance of the enum class, even though the enum can be constructed from it. Add a fallback in the non-primitive branch that attempts to construct the target type from str/int values before rejecting. This correctly handles enum construction (e.g. OpenAIResponseInputToolChoiceMode("auto")) while leaving all other type checks unchanged. Signed-off-by: E Geiger <egeiger@redhat.com>

Fix four issues in vision inference tests: - Guard against None delta.content in streaming chunks by using "or empty string" fallback, preventing TypeError on concatenation - Fix test_image_chat_completion_streaming to check delta.content before calling .lower(), preventing AttributeError on None - Fix multi_image_data[2] reference to multi_image_url[2] in test_image_chat_completion_multiple_images, which was referencing the fixture function instead of the resolved fixture value - Fix chunk.event.delta.text to chunk.choices[0].delta.content in the second streaming loop of test_image_chat_completion_multiple_images - Update stale GitHub URL from meta-llama/ogx to ogx-ai/ogx in test_openai_chat_completion_image_url Signed-off-by: E Geiger <egeiger@redhat.com>

github-actions · 2026-06-04T07:34:05Z

✱ Stainless preview builds

This PR will update the llama-stack-client SDKs with the following commit message.

chore(ci): change CI to use ogx_open_client

Edit this comment to update it. It will appear in the SDK's changelogs.

✅ llama-stack-client-openapi studio · code · diff

Your SDK build had at least one "warning" diagnostic, but this did not represent a regression.
generate ⚠️

✅ llama-stack-client-python studio · code · diff

Your SDK build had at least one "warning" diagnostic, but this did not represent a regression.
generate ⚠️ → build ✅ → lint ✅ → test ✅
pip install https://pkg.stainless.com/s/llama-stack-client-python/7aae36631319e0af83f73e9b0ecea0e62051bf71/ogx_client-1.1.0a3-py3-none-any.whl

✅ llama-stack-client-node studio · code · diff

Your SDK build had at least one "warning" diagnostic, but this did not represent a regression.
generate ⚠️ → build ✅ → lint ✅ → test ✅
npm install https://pkg.stainless.com/s/llama-stack-client-node/f5ae47568da8e9ee9ab796a8875710c8c2303798/dist.tar.gz

✅ llama-stack-client-go studio · conflict

Your SDK build had at least one new note diagnostic, which is a regression from the base state.

New diagnostics (1 note)

💡 Schema/EnumHasOneMember: Confirm intentional use of `enum` with single member.

This comment is auto-generated by GitHub Actions and is automatically kept up to date as you push.
If you push custom code to the preview branch, re-run this workflow to update the comment.
Last updated: 2026-06-17 15:20:54 UTC

The validation workflow was generating the SDK as ogx_client (OPEN=0) but the codebase now imports from ogx_open_client. Update to OPEN=1 and fix all references from ogx_client to ogx_open_client in the install and verification steps. Signed-off-by: E Geiger <egeiger@redhat.com>

…rary client OGXAsLibraryClient constructs APIResponse/AsyncAPIResponse with stainless-specific kwargs (raw, client, cast_to, options, stream, stream_cls) and calls .parse() to deserialize responses. The ogx_open_client ApiResponse had a different constructor and no parse method, causing mypy errors and runtime failures in library mode. Add stainless-compatible constructors and .parse() methods to the api_response and async_api_response templates. For non-streaming, parse deserializes JSON via from_dict/model_validate. For streaming, parse constructs the stream class with the raw response. Also make AsyncStream accept client=None for library client usage. Add type: ignore on the ogx_client fallback import in library_client.py since both packages may be installed with incompatible type signatures. Signed-off-by: E Geiger <egeiger@redhat.com>

…utation The OpenAI SDK treats extra_body as a dict whose keys are merged into the request body at the top level. The generated SDK was storing extra_body as a literal field name, so OGX-specific extensions like embedding_model and provider_id never reached the server. This caused vector store creation to miss embedding config, breaking keyword and hybrid search. Add extra_body merging in partial_api.mustache: pop extra_body from kwargs and merge its contents before constructing the request model. Also fix a bug in model_generic.mustache where __getattribute__ unconditionally copied dict values, preventing mutation of additional_properties. Only create a new dict when OneOf/AnyOf unwrapping is actually needed. Signed-off-by: E Geiger <egeiger@redhat.com>

The model template emits __iter__/__getitem__/__len__ methods only for the first array field in a model. Several list response schemas had non-array fields (like object) before the data field, causing the template to skip iteration methods. This made iterating a response yield (field_name, value) tuples instead of data items. Add reorder_data_field_first() in build_hierarchy.py to ensure the data property comes first in any schema with an array data field. Also update uv.lock to ogx-open-client 1.0.3.dev4. Signed-off-by: E Geiger <egeiger@redhat.com>

Co-Authored-By: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…fully The api_recorder monkey-patches OgxClient._prepare_request to inject test IDs into request headers. The patched function calls the stored original via _original_methods dict. When ogx_open_client and ogx_client are both installed, a class identity mismatch can cause the patch to be applied to one OgxClient class while the originals are stored from another, leaving _original_methods without the expected keys. Use dict.get() instead of direct key access so the patched function handles missing originals gracefully. The originals are no-op methods (return None), so skipping them is safe. Signed-off-by: E Geiger <egeiger@redhat.com>

…ll_api patch OGXAsLibraryClient runs the OGX server in-process without HTTP. The stainless SDK routed requests through a self.request() override, but ogx_open_client uses a different call chain: API methods call api_client.call_api() which delegates to rest_client.request() and httpx. The request() override was never invoked, causing all library mode API calls to fail with network errors. Patch api_client.call_api() on initialization to intercept requests before they reach httpx. The patched method: - Parses the URL to extract the path and query parameters - Matches the route to a FastAPI endpoint handler via find_matching_route - Converts the request body to proper function kwargs via _convert_body - Executes the handler directly (in-process, async via event loop) - Wraps the result in a RESTResponse for the SDK to deserialize Handles non-streaming JSON responses, streaming SSE responses, FastAPI Response objects, and multipart file uploads. Signed-off-by: E Geiger <egeiger@redhat.com>

The rerank tests asserted isinstance(response, list) but ogx_open_client returns a RerankResponse object (which is iterable and supports len/indexing) rather than a bare list. Replace isinstance checks with len() >= 0 assertions that verify the response is sequence-like without requiring a specific type. Signed-off-by: E Geiger <egeiger@redhat.com>

…K compatibility Langchain and langgraph integrations access client.api_key to configure themselves. The stainless SDK OgxClient had this as a class attribute. Add api_key as an optional constructor parameter to both OgxClient and AsyncOgxClient templates, stored as an instance attribute for compatibility. Signed-off-by: E Geiger <egeiger@redhat.com>

devin-ai-integration

Devin Review found 2 potential issues.

View 6 additional findings in Devin Review.

devin-ai-integration · 2026-06-04T14:59:05Z

+        from urllib.parse import urlparse
+
+        from fastapi.responses import StreamingResponse
+        from ogx_open_client.rest import RESTResponse


🔴 Hard-coded ogx_open_client import breaks fallback to ogx_client

The top-level imports at src/ogx/core/library_client.py:30-52 implement a fallback mechanism: try ogx_open_client first, then fall back to ogx_client. However, the inline import at line 415 (from ogx_open_client.rest import RESTResponse) is hard-coded to only use ogx_open_client. When a user has only ogx_client installed (the ogx[client] extra), the top-level import succeeds via the fallback, but the first call to _async_in_process_call will raise an ImportError because ogx_open_client doesn't exist. This breaks the OGXAsLibraryClient for anyone using the old ogx_client package.

Prompt for agents

In src/ogx/core/library_client.py, the inline import at line 415 hard-codes `from ogx_open_client.rest import RESTResponse`, but the top-level imports (lines 30-52) have a fallback mechanism that tries ogx_open_client first, then falls back to ogx_client. The inline import needs to follow the same pattern. One approach: at the top of the file (near lines 30-52), also import RESTResponse through the same try/except fallback and store it, or use a helper that resolves the correct module at import time. Alternatively, replicate the try/except pattern inline: try importing from ogx_open_client.rest, and if that fails, import from ogx_client.rest.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-06-04T14:59:07Z

+    needs: build-sdk
+    runs-on: ubuntu-latest
+    permissions:


🔴 Missing environment in publish-pypi job breaks OIDC trusted publishing

The publish-pypi job is missing the environment configuration required for OIDC trusted publishing. The old workflow had environment: name: ${{ ... 'pypi-production' ... }} which resolved to pypi-production for PyPI publishes. The refactored publish-testpypi job correctly sets environment: testpypi (line 113), but publish-pypi (line 143) has no environment at all. PyPI's trusted publisher configuration requires the GitHub environment name to match; without it, the pypa/gh-action-pypi-publish OIDC token exchange will fail and production PyPI publishing will be broken.

Suggested change

needs: build-sdk

runs-on: ubuntu-latest

permissions:

needs: build-sdk

runs-on: ubuntu-latest

environment: pypi-production

permissions:

Was this helpful? React with 👍 or 👎 to provide feedback.

actually the trusted environment in pypi is "" so it will accept anything. As far as I can tell the workflow that's pushing ogx package is an empty environment so I wanted to create the option to scope things better in the future.

…patibility The call_api patch in OGXAsLibraryClient.__init__ unconditionally accessed self.api_client.call_api, but the stainless SDK OgxClient does not have an api_client attribute. Guard with hasattr checks so the fallback to the stainless SDK (which uses the request() override instead) still works. Also add try/except fallback for the ogx_open_client.rest import in _async_in_process_call to avoid ModuleNotFoundError when only the stainless SDK is installed. Signed-off-by: E Geiger <egeiger@redhat.com>

aegeiger · 2026-06-08T05:57:52Z

I think that the failing vllm test is a fluke, but seems like I can't re-run, would appreciate a maintainer's help.
As kind-of evidence to support this claim, I had this workflow fail on my fork and a re-run had it passing https://github.com/aegeiger/ogx/actions/runs/27092454314/job/80030723690?pr=1
It also passed here for a previous commit and I don't think the last commit should have introduced this error.

mergify · 2026-06-09T13:03:18Z

This pull request has merge conflicts that must be resolved before it can be merged. @aegeiger please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Fix LibraryClientHttpxResponse call in _async_in_process_call to pass content via response_body_bytes(), matching the updated constructor signature from main. Switch test_files.py import from ogx_client to ogx_open_client, consistent with all other integration test files. Replace vacuous len(response) >= 0 assertion in test_rerank with a proper check that the response is list-like, since len() can never return a negative value. Signed-off-by: E Geiger <egeiger@redhat.com>

github-actions · 2026-06-18T10:30:29Z

Recording workflow completed

Providers: bedrock, vllm

Recordings have been generated and will be committed automatically by the companion workflow.

View workflow run

Fork PR: Recordings will be committed if you have "Allow edits from maintainers" enabled.

…EDROCK env var The bedrock field rename from api_key to aws_bedrock_bearer_token changed the env var name in CI config files but the CI workflows still set the old AWS_BEARER_TOKEN_BEDROCK variable, causing SigV4 credential resolution failures in library-mode bedrock integration tests. Revert the CI config files to use the old field and env var names until the rename PR lands on main with matching workflow updates. Signed-off-by: E Geiger <egeiger@redhat.com>

Stainless no longer supports this project — their platform returns 401 with 'This org has migrated to stlc and is now read-only'. Remove the stainless-builds.yml workflow and clean up all CI references to it (matrix config, comments, docs). The client-sdks/stainless/ directory and OpenAPI Generator scripts that read those files locally are left intact as they serve the OpenAPI Generator SDK pipeline and do not call the Stainless API. Signed-off-by: E Geiger <egeiger@redhat.com>

mergify · 2026-06-18T14:55:21Z

This pull request has merge conflicts that must be resolved before it can be merged. @aegeiger please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

…en_client

…_TOKEN The distro codegen produces configs referencing ${env.AWS_BEDROCK_BEARER_TOKEN} (the canonical name from sample_run_config()), but CI workflows and scripts were setting the old env var name AWS_BEARER_TOKEN_BEDROCK. This mismatch caused the distro-codegen pre-commit hook to fight with any manual override in the ci-tests YAML files. Fix by updating the CI env var names to match the canonical name. The GitHub secret remains named AWS_BEARER_TOKEN_BEDROCK — only the env var exposed to the process changes. This reverts commit 1861bfb which was a workaround for the same mismatch from the other direction. Signed-off-by: E Geiger <egeiger@redhat.com>

leseb · 2026-06-22T08:48:36Z

I'm going to merge manually with Stainless SDK Builds / preview (pull_request_target) being red, this is expected and frankly we don't care because it is going away with this PR.

## Summary - add replay recordings for the two Bedrock streaming chat-completion hashes introduced after the ogx_open_client switch - fixes the missing-recording failures from PR ogx-ai#6016 without changing runtime behavior ## Test Plan - env AWS_BEDROCK_BEARER_TOKEN=replay-mode-dummy-key AWS_DEFAULT_REGION=us-west-2 uv run --no-sync ./scripts/integration-tests.sh --stack-config ci-tests --setup bedrock --suite bedrock --inference-mode replay - Result: 6 passed, 6 skipped Signed-off-by: Sumanth Kamenani <skamenan@redhat.com>

aegeiger added 6 commits June 4, 2026 10:27

aegeiger and others added 10 commits June 4, 2026 14:15

Recordings update from CI

c56a1c6

Co-Authored-By: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

chore(deps): update ogx_open_client

4a20fca

aegeiger marked this pull request as ready for review June 4, 2026 14:56

aegeiger requested review from bbrowning, cdoern, franciscojavierarceo, leseb, mattf, raghotham and skamenan7 as code owners June 4, 2026 14:56

devin-ai-integration Bot reviewed Jun 4, 2026

View reviewed changes

aegeiger force-pushed the chore/ci_using_ogx_open_client branch from 57ebe0d to 64ccd26 Compare June 7, 2026 12:22

leseb enabled auto-merge June 17, 2026 09:38

Merge branch 'main' into chore/ci_using_ogx_open_client

4e55588

auto-merge was automatically disabled June 17, 2026 11:09
Head branch was pushed to by a user without write access

mergify Bot removed the needs-rebase label Jun 17, 2026

leseb enabled auto-merge June 17, 2026 12:52

auto-merge was automatically disabled June 17, 2026 14:09
Head branch was pushed to by a user without write access

Merge branch 'main' into chore/ci_using_ogx_open_client

7eb7657

leseb enabled auto-merge June 17, 2026 15:36

leseb added this pull request to the merge queue Jun 17, 2026

github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Jun 17, 2026

leseb added this pull request to the merge queue Jun 18, 2026

github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Jun 18, 2026

leseb added this pull request to the merge queue Jun 18, 2026

github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Jun 18, 2026

aegeiger added 3 commits June 18, 2026 15:24

Merge branch 'main' into chore/ci_using_ogx_open_client

9183803

mergify Bot added the needs-rebase label Jun 18, 2026

Merge remote-tracking branch 'origin/main' into chore/ci_using_ogx_op…

6aa17cb

…en_client

mergify Bot removed the needs-rebase label Jun 21, 2026

leseb merged commit c9f432a into ogx-ai:main Jun 22, 2026
81 of 83 checks passed

This was referenced Jun 23, 2026

fix(tests): add Bedrock replay recordings #6166

Merged

fix(bedrock): preserve bearer token env compatibility #6167

Merged

aegeiger mentioned this pull request Jun 30, 2026

fix(ci): include ogx-open-client in unified release pipeline #6205

Merged

Navigation Menu

Uh oh!

chore(ci): change CI to use ogx_open_client - #6016

chore(ci): change CI to use ogx_open_client#6016
leseb merged 32 commits into
ogx-ai:mainfrom
aegeiger:chore/ci_using_ogx_open_client

aegeiger commented Jun 4, 2026

Uh oh!

github-actions Bot commented Jun 4, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Jun 4, 2026

Uh oh!

devin-ai-integration Bot Jun 4, 2026

Uh oh!

aegeiger Jun 4, 2026

Uh oh!

aegeiger commented Jun 8, 2026

Uh oh!

mergify Bot commented Jun 9, 2026

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

Uh oh!

mergify Bot commented Jun 18, 2026

Uh oh!

leseb commented Jun 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

aegeiger commented Jun 4, 2026

What does this PR do?

Test Plan

Uh oh!

github-actions Bot commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✱ Stainless preview builds

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

aegeiger Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

aegeiger commented Jun 8, 2026

Uh oh!

mergify Bot commented Jun 9, 2026

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

Uh oh!

mergify Bot commented Jun 18, 2026

Uh oh!

leseb commented Jun 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented Jun 4, 2026 •

edited

Loading