|
1 | 1 | # @tanstack/ai-client |
2 | 2 |
|
| 3 | +## 0.15.0 |
| 4 | + |
| 5 | +### Minor Changes |
| 6 | + |
| 7 | +- [#666](https://github.com/TanStack/ai/pull/666) [`c1ae8b9`](https://github.com/TanStack/ai/commit/c1ae8b94c83d70508975568eb4fc9b45f1af540b) - feat: support multimodal (image) tool results |
| 8 | + |
| 9 | + Tools may now return an `Array<ContentPart>` (e.g. a text part plus an image part) and have it transmitted to the model as structured multimodal tool output instead of a `JSON.stringify`'d blob. This unblocks use cases like returning a screenshot from a tool so the model can see it (issue [#363](https://github.com/TanStack/ai/issues/363)). |
| 10 | + - Detection is structural and opt-in by shape: a tool that returns a non-empty array whose every element is a valid `ContentPart` is passed through unchanged; strings and all other return values are serialized exactly as before, so there are no breaking changes. |
| 11 | + - The OpenAI Responses, Anthropic, and Google Gemini adapters convert the content parts into their native multimodal tool-output formats (`function_call_output.output`, `tool_result` content blocks, and `functionResponse.parts` respectively). Providers on the Chat Completions path (Groq, Ollama, Grok, OpenRouter chat) fall back to stringifying, which their APIs require. |
| 12 | + - AG-UI stream events (`TOOL_CALL_RESULT.content`, `TOOL_CALL_END.result`) remain string-only per the spec; the multimodal array travels on the tool message itself. |
| 13 | + |
| 14 | +- [#628](https://github.com/TanStack/ai/pull/628) [`8036b50`](https://github.com/TanStack/ai/commit/8036b5054330a180023c6e3225b8d2735a43a919) - Add typed runtime context for tools and middleware. |
| 15 | + |
| 16 | + Tools and middleware can now declare the runtime context shape they require, and |
| 17 | + `chat()`, `ChatClient`, and the framework `useChat` / `createChat` hooks infer |
| 18 | + the merged requirement and type-check the `context` option you pass against it. |
| 19 | + |
| 20 | + ```typescript |
| 21 | + type AppContext = { userId: string; db: Db } |
| 22 | + |
| 23 | + const listNotes = toolDefinition({ |
| 24 | + name: 'list_notes' /* ... */, |
| 25 | + }).server<AppContext>((_input, ctx) => |
| 26 | + ctx.context.db.notes.findMany({ userId: ctx.context.userId }), |
| 27 | + ) |
| 28 | + |
| 29 | + chat({ |
| 30 | + adapter, |
| 31 | + messages, |
| 32 | + tools: [listNotes], |
| 33 | + context: { userId, db }, // required and type-checked because listNotes declares AppContext |
| 34 | + }) |
| 35 | + ``` |
| 36 | + |
| 37 | + Runtime context is request-local application state for tool and middleware |
| 38 | + implementations (authenticated users, database clients, tenancy, feature flags, |
| 39 | + loggers, browser services). It is never sent to the model and is distinct from |
| 40 | + the AG-UI `RunAgentInput.context` protocol field. |
| 41 | + |
| 42 | + Untyped tools and middleware continue to receive `unknown` context and do not |
| 43 | + force a `context` option. Client tools receive client-local context via |
| 44 | + `ChatClient` / `useChat`; use `forwardedProps` to hand serializable client data |
| 45 | + to the server and map it into server context explicitly. See the new Runtime |
| 46 | + Context guide for details. |
| 47 | + |
| 48 | + Behavior change: tool output validation now also runs when a tool returns |
| 49 | + `undefined` or `null`. Previously these values bypassed `outputSchema` |
| 50 | + validation entirely; now the schema decides whether they are valid, so a tool |
| 51 | + whose schema forbids `undefined`/`null` surfaces a validation error |
| 52 | + (`output-error`) instead of silently passing. Tools whose schema permits |
| 53 | + `null`/`undefined` (e.g. nullable or void outputs) are unaffected. |
| 54 | + |
| 55 | +### Patch Changes |
| 56 | + |
| 57 | +- Updated dependencies [[`c1ae8b9`](https://github.com/TanStack/ai/commit/c1ae8b94c83d70508975568eb4fc9b45f1af540b), [`a452ae8`](https://github.com/TanStack/ai/commit/a452ae8bcda8abfdc6309983976ed0fbf6df1915), [`8036b50`](https://github.com/TanStack/ai/commit/8036b5054330a180023c6e3225b8d2735a43a919)]: |
| 58 | + - @tanstack/ai@0.24.0 |
| 59 | + - @tanstack/ai-event-client@0.4.3 |
| 60 | + |
3 | 61 | ## 0.14.1 |
4 | 62 |
|
5 | 63 | ### Patch Changes |
|
0 commit comments