Skip to content

Expose the model's reasoning from ChatCompletions API #625

Description

@arunavos

This is some low-hanging fruit that will improve the library's usability for interacting with vLLM models.

If we were to add a Reasoning field to the ChatCompletionChunkChoiceDelta struct definition, as shown below, then we would be able to access the returned reasoning, if any.

// A chat completion delta generated by streamed model responses.
type ChatCompletionChunkChoiceDelta struct {
	// The contents of the chunk message.
	Content   string `json:"content"       api:"nullable"`
	// The reasoning produced by the model.
	Reasoning string `json:"reasoning"     api:"nullable"`
	// Deprecated and replaced by `tool_calls`. The name and arguments of a function
	// that should be called, as generated by the model.
	//
	// Deprecated: deprecated
	FunctionCall ChatCompletionChunkChoiceDeltaFunctionCall `json:"function_call"`
	// The refusal message generated by the model.
	Refusal string `json:"refusal"       api:"nullable"`
	// The role of the author of this message.
	//
	// Any of "developer", "system", "user", "assistant", "tool".
	Role      string                                   `json:"role"`
	ToolCalls []ChatCompletionChunkChoiceDeltaToolCall `json:"tool_calls"`
	// JSON contains metadata for fields, check presence with [respjson.Field.Valid].
	JSON struct {
		Content      respjson.Field
		FunctionCall respjson.Field
		Refusal      respjson.Field
		Role         respjson.Field
		ToolCalls    respjson.Field
		ExtraFields  map[string]respjson.Field
		raw          string
	} `json:"-"`
}

I would find this useful since it would allow me to return thinking content to the caller so they can see the reasoning process and know that the model has not hung unexpectedly, and also monitor that it is answering the prompt as intended.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Fields

    No fields configured for Feature.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions