Skip to content

Commit 87dc40b

Browse files
cdoerncdoern
authored
feat(inference): bidirectional reasoning token passthrough for chat completions (#5038)
# What does this PR do? Add extra="allow" to OpenAIAssistantMessageParam so that provider-specific fields (reasoning, reasoning_content, thinking) are preserved when clients echo assistant messages back in multi-turn conversations. Without this, reasoning tokens returned by models like gpt-oss on Ollama were silently dropped during request deserialization, causing measurable quality degradation on multi-turn benchmarks (e.g. BFCL). The response direction already worked: the OpenAI SDK uses extra="allow" on its response types, so provider reasoning fields survive the provider->client path and are included in model_dump_json() output. This change closes the client->provider gap. No OpenAPI schema fields are added for reasoning content -- the passthrough is transparent (additionalProperties: true), keeping provider-specific details out of the public API contract. Add integration test test_openai_chat_completion_reasoning_passthrough that records a two-turn conversation with a reasoning model and asserts that reasoning tokens survive the round-trip. The test is skipped automatically when the configured model does not emit reasoning tokens. ## Test Plan new integration test using gpt-oss:20b relates to #5007 Signed-off-by: Charlie Doern <cdoern@redhat.com>
1 parent bf94583 commit 87dc40b

12 files changed

Lines changed: 1335 additions & 4 deletions

‎client-sdks/stainless/openapi.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4600,6 +4600,7 @@ components:
46004600
title: ListOpenAIChatCompletionResponse
46014601
description: Response from listing OpenAI-compatible chat completions.
46024602
OpenAIAssistantMessageParam:
4603+
additionalProperties: true
46034604
description: A message containing the model's (assistant) response in an OpenAI-compatible chat completion request.
46044605
properties:
46054606
role:
@@ -13051,6 +13052,7 @@ components:
1305113052
type: array
1305213053
- type: 'null'
1305313054
description: List of tool calls. Each tool call is an OpenAIChatCompletionToolCall object.
13055+
additionalProperties: true
1305413056
type: object
1305513057
title: OpenAIAssistantMessageParam
1305613058
description: A message containing the model's (assistant) response in an OpenAI-compatible chat completion request.
@@ -13084,6 +13086,7 @@ components:
1308413086
type: array
1308513087
- type: 'null'
1308613088
description: List of tool calls. Each tool call is an OpenAIChatCompletionToolCall object.
13089+
additionalProperties: true
1308713090
type: object
1308813091
title: OpenAIAssistantMessageParam
1308913092
description: A message containing the model's (assistant) response in an OpenAI-compatible chat completion request.

‎docs/static/deprecated-llama-stack-spec.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1144,6 +1144,7 @@ components:
11441144
title: ListOpenAIChatCompletionResponse
11451145
description: Response from listing OpenAI-compatible chat completions.
11461146
OpenAIAssistantMessageParam:
1147+
additionalProperties: true
11471148
description: A message containing the model's (assistant) response in an OpenAI-compatible chat completion request.
11481149
properties:
11491150
role:
@@ -9597,6 +9598,7 @@ components:
95979598
type: array
95989599
- type: 'null'
95999600
description: List of tool calls. Each tool call is an OpenAIChatCompletionToolCall object.
9601+
additionalProperties: true
96009602
type: object
96019603
title: OpenAIAssistantMessageParam
96029604
description: A message containing the model's (assistant) response in an OpenAI-compatible chat completion request.
@@ -9630,6 +9632,7 @@ components:
96309632
type: array
96319633
- type: 'null'
96329634
description: List of tool calls. Each tool call is an OpenAIChatCompletionToolCall object.
9635+
additionalProperties: true
96339636
type: object
96349637
title: OpenAIAssistantMessageParam
96359638
description: A message containing the model's (assistant) response in an OpenAI-compatible chat completion request.

‎docs/static/experimental-llama-stack-spec.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1256,6 +1256,7 @@ components:
12561256
title: ListOpenAIChatCompletionResponse
12571257
description: Response from listing OpenAI-compatible chat completions.
12581258
OpenAIAssistantMessageParam:
1259+
additionalProperties: true
12591260
description: A message containing the model's (assistant) response in an OpenAI-compatible chat completion request.
12601261
properties:
12611262
role:
@@ -9354,6 +9355,7 @@ components:
93549355
type: array
93559356
- type: 'null'
93569357
description: List of tool calls. Each tool call is an OpenAIChatCompletionToolCall object.
9358+
additionalProperties: true
93579359
type: object
93589360
title: OpenAIAssistantMessageParam
93599361
description: A message containing the model's (assistant) response in an OpenAI-compatible chat completion request.
@@ -9387,6 +9389,7 @@ components:
93879389
type: array
93889390
- type: 'null'
93899391
description: List of tool calls. Each tool call is an OpenAIChatCompletionToolCall object.
9392+
additionalProperties: true
93909393
type: object
93919394
title: OpenAIAssistantMessageParam
93929395
description: A message containing the model's (assistant) response in an OpenAI-compatible chat completion request.

‎docs/static/llama-stack-spec.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3021,6 +3021,7 @@ components:
30213021
title: ListOpenAIChatCompletionResponse
30223022
description: Response from listing OpenAI-compatible chat completions.
30233023
OpenAIAssistantMessageParam:
3024+
additionalProperties: true
30243025
description: A message containing the model's (assistant) response in an OpenAI-compatible chat completion request.
30253026
properties:
30263027
role:
@@ -11406,6 +11407,7 @@ components:
1140611407
type: array
1140711408
- type: 'null'
1140811409
description: List of tool calls. Each tool call is an OpenAIChatCompletionToolCall object.
11410+
additionalProperties: true
1140911411
type: object
1141011412
title: OpenAIAssistantMessageParam
1141111413
description: A message containing the model's (assistant) response in an OpenAI-compatible chat completion request.
@@ -11439,6 +11441,7 @@ components:
1143911441
type: array
1144011442
- type: 'null'
1144111443
description: List of tool calls. Each tool call is an OpenAIChatCompletionToolCall object.
11444+
additionalProperties: true
1144211445
type: object
1144311446
title: OpenAIAssistantMessageParam
1144411447
description: A message containing the model's (assistant) response in an OpenAI-compatible chat completion request.

‎docs/static/stainless-llama-stack-spec.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4600,6 +4600,7 @@ components:
46004600
title: ListOpenAIChatCompletionResponse
46014601
description: Response from listing OpenAI-compatible chat completions.
46024602
OpenAIAssistantMessageParam:
4603+
additionalProperties: true
46034604
description: A message containing the model's (assistant) response in an OpenAI-compatible chat completion request.
46044605
properties:
46054606
role:
@@ -13051,6 +13052,7 @@ components:
1305113052
type: array
1305213053
- type: 'null'
1305313054
description: List of tool calls. Each tool call is an OpenAIChatCompletionToolCall object.
13055+
additionalProperties: true
1305413056
type: object
1305513057
title: OpenAIAssistantMessageParam
1305613058
description: A message containing the model's (assistant) response in an OpenAI-compatible chat completion request.
@@ -13084,6 +13086,7 @@ components:
1308413086
type: array
1308513087
- type: 'null'
1308613088
description: List of tool calls. Each tool call is an OpenAIChatCompletionToolCall object.
13089+
additionalProperties: true
1308713090
type: object
1308813091
title: OpenAIAssistantMessageParam
1308913092
description: A message containing the model's (assistant) response in an OpenAI-compatible chat completion request.

‎src/llama_stack_api/inference/models.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
from enum import Enum, StrEnum
1414
from typing import Annotated, Any, Literal, Self
1515

16-
from pydantic import BaseModel, Field, field_validator, model_validator
16+
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
1717
from typing_extensions import TypedDict
1818

1919
from llama_stack_api.common.content_types import InterleavedContent
@@ -391,6 +391,8 @@ class OpenAIChatCompletionToolCall(BaseModel):
391391
class OpenAIAssistantMessageParam(BaseModel):
392392
"""A message containing the model's (assistant) response in an OpenAI-compatible chat completion request."""
393393

394+
model_config = ConfigDict(extra="allow")
395+
394396
role: Literal["assistant"] = Field(
395397
default="assistant", description="Must be 'assistant' to identify this as the model's response."
396398
)

‎tests/integration/ci_matrix.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,8 @@
66
{"suite": "vision", "setup": "ollama-vision"},
77
{"suite": "responses", "setup": "gpt"},
88
{"suite": "base-vllm-subset", "setup": "vllm"},
9-
{"suite": "reasoning", "setup": "vllm"}
9+
{"suite": "vllm-reasoning", "setup": "vllm"},
10+
{"suite": "ollama-reasoning", "setup": "ollama-reasoning"}
1011
],
1112
"stainless": [
1213
{"suite": "base", "setup": "ollama", "inference_mode": "record-if-missing"}

‎tests/integration/inference/recordings/00bbacbcc8180b2151778cdac99b204e347526bc5509d4972f5fdb4d4bed2434.json‎

Lines changed: 71 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎tests/integration/inference/recordings/27626d79a55535527b7d7ef07f3da44a52de3d7d2e56297491d1a62ede6d48c5.json‎

Lines changed: 62 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)