recording: ./scripts/integration-tests.sh --stack-config server:ci-tests --suite responses --inference-mode record-if-missing --setup ollama

Author: mattf

docs/docs/api-openai/provider_matrix.md

@@ -22,6 +22,7 @@ inference provider, based on integration test results.
 | Provider | Tested | Passing | Failing | Coverage |
 |----------|--------|---------|---------|----------|
 | azure | 89 | 89 | 0 | 86% |
+| ollama | 88 | 88 | 0 | 85% |
 | openai | 104 | 104 | 0 | 100% |
 | vllm | 1 | 1 | 0 | 1% |
 
@@ -32,167 +33,168 @@ Models, endpoints, and versions used during test recordings.
 | Provider | Model(s) | Endpoint | Version Info |
 |----------|----------|----------|--------------|
 | azure | gpt-4o | lls-test.openai.azure.com | — |
+| ollama | llama3.2:3b-instruct-fp16 | — | openai sdk: 2.5.0 |
 | openai | gpt-4o, text-embedding-3-small | api.openai.com | openai sdk: 2.5.0 |
 | vllm | Qwen/Qwen3-0.6B | — | — |
 
 ## Basic Responses
 
-| Feature | azure | openai | vllm |
-| --- | --- | --- | --- |
-| extra body guided choice | ✅ | ✅ | — |
-| include logprobs non streaming | ✅ | ✅ | — |
-| include logprobs streaming | ✅ | ✅ | — |
-| include logprobs with function tools | ✅ | ✅ | — |
-| include logprobs with web search | ✅ | ✅ | — |
-| non streaming basic | ✅ | ✅ | — |
-| non streaming image | ✅ | ✅ | — |
-| non streaming multi turn | ✅ | ✅ | — |
-| non streaming multi turn image | ✅ | ✅ | — |
-| streaming basic | ✅ | ✅ | — |
-| streaming incremental content | ✅ | ✅ | — |
+| Feature | azure | ollama | openai | vllm |
+| --- | --- | --- | --- | --- |
+| extra body guided choice | ✅ | ⏭️ | ✅ | — |
+| include logprobs non streaming | ✅ | ⏭️ | ✅ | — |
+| include logprobs streaming | ✅ | ⏭️ | ✅ | — |
+| include logprobs with function tools | ✅ | ⏭️ | ✅ | — |
+| include logprobs with web search | ✅ | ⏭️ | ✅ | — |
+| non streaming basic | ✅ | ✅ | ✅ | — |
+| non streaming image | ✅ | ✅ | ✅ | — |
+| non streaming multi turn | ✅ | ✅ | ✅ | — |
+| non streaming multi turn image | ✅ | ✅ | ✅ | — |
+| streaming basic | ✅ | ✅ | ✅ | — |
+| streaming incremental content | ✅ | ✅ | ✅ | — |
 
 ## Conversation Responses
 
-| Feature | azure | openai | vllm |
-| --- | --- | --- | --- |
-| conversation backward compatibility | ✅ | ✅ | — |
-| conversation basic workflow | ✅ | ✅ | — |
-| conversation context loading | ✅ | ✅ | — |
-| conversation error handling | ⏭️ | ✅ | — |
-| conversation multi turn and streaming | ✅ | ✅ | — |
+| Feature | azure | ollama | openai | vllm |
+| --- | --- | --- | --- | --- |
+| conversation backward compatibility | ✅ | ✅ | ✅ | — |
+| conversation basic workflow | ✅ | ✅ | ✅ | — |
+| conversation context loading | ✅ | ✅ | ✅ | — |
+| conversation error handling | ⏭️ | ⏭️ | ✅ | — |
+| conversation multi turn and streaming | ✅ | ✅ | ✅ | — |
 
 ## File Search
 
-| Feature | azure | openai | vllm |
-| --- | --- | --- | --- |
-| file search filter by category | ✅ | ✅ | — |
-| file search filter by date range | ✅ | ✅ | — |
-| file search filter by region | ✅ | ✅ | — |
-| file search filter compound and | ✅ | ✅ | — |
-| file search filter compound or | ✅ | ✅ | — |
-| file search streaming events | ✅ | ✅ | — |
-| text format | ✅ | ✅ | — |
+| Feature | azure | ollama | openai | vllm |
+| --- | --- | --- | --- | --- |
+| file search filter by category | ✅ | ✅ | ✅ | — |
+| file search filter by date range | ✅ | ✅ | ✅ | — |
+| file search filter by region | ✅ | ✅ | ✅ | — |
+| file search filter compound and | ✅ | ✅ | ✅ | — |
+| file search filter compound or | ✅ | ✅ | ✅ | — |
+| file search streaming events | ✅ | ✅ | ✅ | — |
+| text format | ✅ | ✅ | ✅ | — |
 
 ## Mcp Authentication
 
-| Feature | azure | openai | vllm |
-| --- | --- | --- | --- |
-| mcp authorization backward compatibility | ✅ | ✅ | — |
-| mcp authorization bearer | ✅ | ✅ | — |
+| Feature | azure | ollama | openai | vllm |
+| --- | --- | --- | --- | --- |
+| mcp authorization backward compatibility | ✅ | ✅ | ✅ | — |
+| mcp authorization bearer | ✅ | ✅ | ✅ | — |
 
 ## Openai Responses
 
-| Feature | azure | openai | vllm |
-| --- | --- | --- | --- |
-| background false is synchronous | ✅ | ✅ | — |
-| background returns queued | ✅ | ✅ | — |
-| streaming invalid base64 image failure code is spec compliant | ✅ | ✅ | — |
-| with max output tokens | ✅ | ✅ | — |
-| with parallel tool calls and previous response | ✅ | ✅ | — |
-| with parallel tool calls disabled | ✅ | ✅ | — |
-| with parallel tool calls disabled streaming | ✅ | ✅ | — |
-| with parallel tool calls enabled | ✅ | ✅ | — |
-| with prompt cache key | ✅ | ✅ | — |
-| with prompt cache key and previous response | ✅ | ✅ | — |
-| with prompt cache key streaming | ✅ | ✅ | — |
-| with safety identifier | ✅ | ✅ | — |
-| with safety identifier and previous response | ✅ | ✅ | — |
-| with safety identifier streaming | ✅ | ✅ | — |
-| with service tier | ⏭️ | ✅ | — |
-| with service tier and previous response | ⏭️ | ✅ | — |
-| with service tier auto | ⏭️ | ✅ | — |
-| with service tier auto and previous response | ⏭️ | ✅ | — |
-| with service tier auto streaming | ⏭️ | ✅ | — |
-| with service tier flex | ⏭️ | ✅ | — |
-| with service tier flex streaming | ⏭️ | ✅ | — |
-| with service tier streaming | ⏭️ | ✅ | — |
-| with small max output tokens | ✅ | ✅ | — |
-| with top p | ✅ | ✅ | — |
-| with top p and previous response | ✅ | ✅ | — |
-| with top p streaming | ✅ | ✅ | — |
-| with truncation and previous response | ✅ | ✅ | — |
-| with truncation disabled | ✅ | ✅ | — |
-| with truncation disabled streaming | ✅ | ✅ | — |
+| Feature | azure | ollama | openai | vllm |
+| --- | --- | --- | --- | --- |
+| background false is synchronous | ✅ | ✅ | ✅ | — |
+| background returns queued | ✅ | ✅ | ✅ | — |
+| streaming invalid base64 image failure code is spec compliant | ✅ | ✅ | ✅ | — |
+| with max output tokens | ✅ | ✅ | ✅ | — |
+| with parallel tool calls and previous response | ✅ | ✅ | ✅ | — |
+| with parallel tool calls disabled | ✅ | ✅ | ✅ | — |
+| with parallel tool calls disabled streaming | ✅ | ✅ | ✅ | — |
+| with parallel tool calls enabled | ✅ | ✅ | ✅ | — |
+| with prompt cache key | ✅ | ✅ | ✅ | — |
+| with prompt cache key and previous response | ✅ | ✅ | ✅ | — |
+| with prompt cache key streaming | ✅ | ✅ | ✅ | — |
+| with safety identifier | ✅ | ✅ | ✅ | — |
+| with safety identifier and previous response | ✅ | ✅ | ✅ | — |
+| with safety identifier streaming | ✅ | ✅ | ✅ | — |
+| with service tier | ⏭️ | ✅ | ✅ | — |
+| with service tier and previous response | ⏭️ | ✅ | ✅ | — |
+| with service tier auto | ⏭️ | ✅ | ✅ | — |
+| with service tier auto and previous response | ⏭️ | ✅ | ✅ | — |
+| with service tier auto streaming | ⏭️ | ✅ | ✅ | — |
+| with service tier flex | ⏭️ | ✅ | ✅ | — |
+| with service tier flex streaming | ⏭️ | ✅ | ✅ | — |
+| with service tier streaming | ⏭️ | ✅ | ✅ | — |
+| with small max output tokens | ✅ | ✅ | ✅ | — |
+| with top p | ✅ | ✅ | ✅ | — |
+| with top p and previous response | ✅ | ✅ | ✅ | — |
+| with top p streaming | ✅ | ✅ | ✅ | — |
+| with truncation and previous response | ✅ | ✅ | ✅ | — |
+| with truncation disabled | ✅ | ✅ | ✅ | — |
+| with truncation disabled streaming | ✅ | ✅ | ✅ | — |
 
 ## Prompt Templates
 
-| Feature | azure | openai | vllm |
-| --- | --- | --- | --- |
-| basic prompt template | ✅ | ✅ | — |
-| multi variable prompt template | ✅ | ✅ | — |
-| multi version prompt template | ✅ | ✅ | — |
-| prompt template no variables | ✅ | ✅ | — |
-| prompt template with multi turn | ✅ | ✅ | — |
-| prompt template with streaming | ✅ | ✅ | — |
+| Feature | azure | ollama | openai | vllm |
+| --- | --- | --- | --- | --- |
+| basic prompt template | ✅ | ✅ | ✅ | — |
+| multi variable prompt template | ✅ | ✅ | ✅ | — |
+| multi version prompt template | ✅ | ✅ | ✅ | — |
+| prompt template no variables | ✅ | ✅ | ✅ | — |
+| prompt template with multi turn | ✅ | ✅ | ✅ | — |
+| prompt template with streaming | ✅ | ✅ | ✅ | — |
 
 ## Reasoning
 
-| Feature | azure | openai | vllm |
-| --- | --- | --- | --- |
-| reasoning basic streaming | ✅ | ✅ | ✅ |
+| Feature | azure | ollama | openai | vllm |
+| --- | --- | --- | --- | --- |
+| reasoning basic streaming | ✅ | ✅ | ✅ | ✅ |
 
 ## Responses Access Control
 
-| Feature | azure | openai | vllm |
-| --- | --- | --- | --- |
-| previous response id access control | — | ✅ | — |
-| user can access own resources after denial | — | ✅ | — |
-| user cannot access other users response input items | — | ✅ | — |
-| user cannot delete other users response | — | ✅ | — |
-| user cannot retrieve other users response | — | ✅ | — |
-| users have isolated responses | — | ✅ | — |
+| Feature | azure | ollama | openai | vllm |
+| --- | --- | --- | --- | --- |
+| previous response id access control | — | — | ✅ | — |
+| user can access own resources after denial | — | — | ✅ | — |
+| user cannot access other users response input items | — | — | ✅ | — |
+| user cannot delete other users response | — | — | ✅ | — |
+| user cannot retrieve other users response | — | — | ✅ | — |
+| users have isolated responses | — | — | ✅ | — |
 
 ## Responses Errors
 
-| Feature | azure | openai | vllm |
-| --- | --- | --- | --- |
-| completed response has no error | ✅ | ✅ | — |
-| invalid base64 image returns image error | ✅ | ✅ | — |
-| invalid image url returns image error | ✅ | ✅ | — |
-| non vision model returns error for image input | ✅ | ✅ | — |
-| non vision model with base64 image returns server error | ✅ | ✅ | — |
+| Feature | azure | ollama | openai | vllm |
+| --- | --- | --- | --- | --- |
+| completed response has no error | ✅ | ✅ | ✅ | — |
+| invalid base64 image returns image error | ✅ | ⏭️ | ✅ | — |
+| invalid image url returns image error | ✅ | ⏭️ | ✅ | — |
+| non vision model returns error for image input | ✅ | ⏭️ | ✅ | — |
+| non vision model with base64 image returns server error | ✅ | ✅ | ✅ | — |
 
 ## Structured Output
 
-| Feature | azure | openai | vllm |
-| --- | --- | --- | --- |
-| json schema array of integers | ✅ | ✅ | — |
-| json schema array of objects | ✅ | ✅ | — |
-| json schema array of strings | ✅ | ✅ | — |
-| json schema boolean types | ✅ | ✅ | — |
-| json schema float types | ✅ | ✅ | — |
-| json schema integer types | ✅ | ✅ | — |
-| json schema mixed types structures | ✅ | ✅ | — |
-| json schema nested objects | ✅ | ✅ | — |
-| json schema string types | ✅ | ✅ | — |
+| Feature | azure | ollama | openai | vllm |
+| --- | --- | --- | --- | --- |
+| json schema array of integers | ✅ | ✅ | ✅ | — |
+| json schema array of objects | ✅ | ✅ | ✅ | — |
+| json schema array of strings | ✅ | ✅ | ✅ | — |
+| json schema boolean types | ✅ | ✅ | ✅ | — |
+| json schema float types | ✅ | ✅ | ✅ | — |
+| json schema integer types | ✅ | ✅ | ✅ | — |
+| json schema mixed types structures | ✅ | ✅ | ✅ | — |
+| json schema nested objects | ✅ | ✅ | ✅ | — |
+| json schema string types | ✅ | ✅ | ✅ | — |
 
 ## Tool Responses
 
-| Feature | azure | openai | vllm |
-| --- | --- | --- | --- |
-| connector resolution mcp tool | ✅ | ✅ | — |
-| function call ordering 1 | ✅ | ✅ | — |
-| function call ordering 2 | ✅ | ✅ | — |
-| function call output list file | ✅ | ✅ | — |
-| function call output list image | ✅ | ✅ | — |
-| function call output list text | ✅ | ✅ | — |
-| function call output list text multi block | ✅ | ✅ | — |
-| max tool calls with function tools | ✅ | ✅ | — |
-| max tool calls with mcp tools | ✅ | ✅ | — |
-| mcp tool approval | ✅ | ✅ | — |
-| multi turn streaming web search | ✅ | ✅ | — |
-| non streaming custom tool | ✅ | ✅ | — |
-| non streaming file search | ✅ | ✅ | — |
-| non streaming file search empty vector store | ✅ | ✅ | — |
-| non streaming mcp tool | ✅ | ✅ | — |
-| non streaming multi turn tool execution | ✅ | ✅ | — |
-| non streaming web search | ✅ | ✅ | — |
-| parallel tool calls with function tools | ✅ | ✅ | — |
-| parallel tool calls with mcp tools | ✅ | ✅ | — |
-| sequential file search | ✅ | ✅ | — |
-| sequential mcp tool | ✅ | ✅ | — |
-| streaming multi turn tool execution | ✅ | ✅ | — |
-| streaming web search | ✅ | ✅ | — |
+| Feature | azure | ollama | openai | vllm |
+| --- | --- | --- | --- | --- |
+| connector resolution mcp tool | ✅ | ✅ | ✅ | — |
+| function call ordering 1 | ✅ | ✅ | ✅ | — |
+| function call ordering 2 | ✅ | ✅ | ✅ | — |
+| function call output list file | ✅ | ✅ | ✅ | — |
+| function call output list image | ✅ | ⏭️ | ✅ | — |
+| function call output list text | ✅ | ✅ | ✅ | — |
+| function call output list text multi block | ✅ | ✅ | ✅ | — |
+| max tool calls with function tools | ✅ | ✅ | ✅ | — |
+| max tool calls with mcp tools | ✅ | ✅ | ✅ | — |
+| mcp tool approval | ✅ | ✅ | ✅ | — |
+| multi turn streaming web search | ✅ | ✅ | ✅ | — |
+| non streaming custom tool | ✅ | ✅ | ✅ | — |
+| non streaming file search | ✅ | ✅ | ✅ | — |
+| non streaming file search empty vector store | ✅ | ✅ | ✅ | — |
+| non streaming mcp tool | ✅ | ✅ | ✅ | — |
+| non streaming multi turn tool execution | ✅ | ✅ | ✅ | — |
+| non streaming web search | ✅ | ✅ | ✅ | — |
+| parallel tool calls with function tools | ✅ | ✅ | ✅ | — |
+| parallel tool calls with mcp tools | ✅ | ✅ | ✅ | — |
+| sequential file search | ✅ | ✅ | ✅ | — |
+| sequential mcp tool | ✅ | ✅ | ✅ | — |
+| streaming multi turn tool execution | ✅ | ✅ | ✅ | — |
+| streaming web search | ✅ | ✅ | ✅ | — |
 
 ---