You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat(gemini3): add support for conversation memory for gemini 3 models
- Full support for storing and replaying conversation history for Gemini 3 models, including tool calls and tool results, in Redis-backed memory
- Correct handling and grouping of parallel function (tool) calls for Gemini 3 in chat history, ensuring all parallel tool calls/results are grouped per step for accurate multi-turn context
- Correctly persist and replay Gemini 3 thoughtSignature as a sibling property on functionCall/text parts (not a separate part)
- Add and propagate stepIndex metadata for all tool_call and tool_result messages to distinguish parallel vs sequential tool calls
- Update prependConversationHistory to flush tool_call/tool_result buffers on both role change and stepIndex change, ensuring correct grouping for parallel function calls
- Tag tool calls with stepThoughtSig and stepIndex in both generate and stream paths; store on Redis metadata
- Store thoughtSignature and stepIndex on tool_call and tool_result messages in Redis, and in ChatMessageMetadata
- Eliminate all as casts in extractThoughtSignature and use strict null checks for lint compliance
|**Memory**| v9.12.0 | Per-user condensed memory that persists across conversations. LLM-powered condensation with S3, Redis, or SQLite backends. |[Memory Guide](docs/features/memory.md)|
|**Tool Execution Control**| v9.3.0 |`prepareStep` and `toolChoice` support for per-step tool enforcement in multi-step agentic loops. API-level control over tool calls. |[API Reference](docs/api/type-aliases/GenerateOptions.md#preparestep)|
50
-
|**File Processor System**| v9.1.0 | 17+ file type processors with ProcessorRegistry, security sanitization, SVG text injection |[File Processors Guide](docs/features/file-processors.md)|
51
-
|**RAG with generate()/stream()**| v9.2.0 | Pass `rag: { files }` to generate/stream for automatic document chunking, embedding, and AI-powered search. 10 chunking strategies, hybrid search, reranking. |[RAG Guide](docs/features/rag.md)|
|**Server Adapters**| v8.43.0 | Multi-framework HTTP server with Hono, Express, Fastify, Koa support. Full CLI for server management with foreground/background modes. |[Server Adapters Guide](docs/guides/server-adapters/index.md)|
54
-
|**Title Generation Events**| v8.38.0 | Emit `conversation:titleGenerated` event when conversation title is generated. Supports custom title prompts via `NEUROLINK_TITLE_PROMPT`. |[Conversation Memory Guide](docs/conversation-memory.md)|
55
-
|**Video Generation with Veo**| v8.32.0 | Video generation using Veo 3.1 (`veo-3.1`). Realistic video generation with many parameter options |[Video Generation Guide](docs/features/video-generation.md)|
56
-
|**Image Generation with Gemini**| v8.31.0 | Native image generation using Gemini 2.0 Flash Experimental (`imagen-3.0-generate-002`). High-quality image synthesis directly from Google AI. |[Image Generation Guide](docs/image-generation-streaming.md)|
57
-
|**HTTP/Streamable HTTP Transport**| v8.29.0 | Connect to remote MCP servers via HTTP with authentication headers, automatic retry with exponential backoff, and configurable rate limiting. |[HTTP Transport Guide](docs/mcp-http-transport.md)|
|**Memory**| v9.12.0 | Per-user condensed memory that persists across conversations. LLM-powered condensation with S3, Redis, or SQLite backends. |[Memory Guide](docs/features/memory.md)|
|**Tool Execution Control**| v9.3.0 |`prepareStep` and `toolChoice` support for per-step tool enforcement in multi-step agentic loops. API-level control over tool calls. |[API Reference](docs/api/type-aliases/GenerateOptions.md#preparestep)|
51
+
|**File Processor System**| v9.1.0 | 17+ file type processors with ProcessorRegistry, security sanitization, SVG text injection |[File Processors Guide](docs/features/file-processors.md)|
52
+
|**RAG with generate()/stream()**| v9.2.0 | Pass `rag: { files }` to generate/stream for automatic document chunking, embedding, and AI-powered search. 10 chunking strategies, hybrid search, reranking. |[RAG Guide](docs/features/rag.md)|
|**Server Adapters**| v8.43.0 | Multi-framework HTTP server with Hono, Express, Fastify, Koa support. Full CLI for server management with foreground/background modes. |[Server Adapters Guide](docs/guides/server-adapters/index.md)|
55
+
|**Title Generation Events**| v8.38.0 | Emit `conversation:titleGenerated` event when conversation title is generated. Supports custom title prompts via `NEUROLINK_TITLE_PROMPT`. |[Conversation Memory Guide](docs/conversation-memory.md)|
56
+
|**Video Generation with Veo**| v8.32.0 | Video generation using Veo 3.1 (`veo-3.1`). Realistic video generation with many parameter options |[Video Generation Guide](docs/features/video-generation.md)|
57
+
|**Image Generation with Gemini**| v8.31.0 | Native image generation using Gemini 2.0 Flash Experimental (`imagen-3.0-generate-002`). High-quality image synthesis directly from Google AI. |[Image Generation Guide](docs/image-generation-streaming.md)|
58
+
|**HTTP/Streamable HTTP Transport**| v8.29.0 | Connect to remote MCP servers via HTTP with authentication headers, automatic retry with exponential backoff, and configurable rate limiting. |[HTTP Transport Guide](docs/mcp-http-transport.md)|
58
59
59
60
-**AutoResearch** – Autonomous AI experiment engine inspired by Karpathy's autoresearch. Phase-gated tool access, git-backed safety, deterministic metric evaluation, and TaskManager integration for continuous unattended research. 12 research tools, 10 typed events, 9 CLI subcommands. → [AutoResearch Guide](docs/features/autoresearch.md)
60
61
-**Memory** – Per-user condensed memory that persists across all conversations. Automatically retrieves and stores memory on each `generate()`/`stream()` call. Supports S3, Redis, and SQLite storage with LLM-powered condensation. → [Memory Guide](docs/features/memory.md)
@@ -68,6 +69,7 @@ Extracted from production systems at Juspay and battle-tested at enterprise scal
68
69
-**Image Generation** – Generate images from text prompts using Gemini models via Vertex AI or Google AI Studio. Supports streaming mode with automatic file saving. → [Image Generation Guide](docs/image-generation-streaming.md)
69
70
-**RAG with generate()/stream()** – Just pass `rag: { files: ["./docs/guide.md"] }` to `generate()` or `stream()`. NeuroLink auto-chunks, embeds, and creates a search tool the AI can invoke. 10 chunking strategies, hybrid search, 5 reranker types. → [RAG Guide](docs/features/rag.md)
70
71
-**HTTP/Streamable HTTP Transport for MCP** – Connect to remote MCP servers via HTTP with authentication headers, retry logic, and rate limiting. → [HTTP Transport Guide](docs/mcp-http-transport.md)
72
+
- 🧠 **Gemini 3 Native Multi-turn Tool Calling** — Fixed multi-step agentic tool calling for Gemini 3 models on Vertex AI. The native `@google/genai` path now correctly replays `thoughtSignature` as a sibling field on each `functionCall` part, groups parallel tool calls by `stepIndex`, enforces a 5-minute default timeout on the generate path, and surfaces silent timeouts as proper `TimeoutError` instead of empty responses. Multi-execution session overlap (where `continueOrchestratorWorkflow` restarts the loop on the same `sessionId`) is addressed by an `executionId` per invocation as a composite grouping key — this prevents tool calls from two different executions colliding into the same Gemini model turn and causing the model to return 0 function calls.
71
73
- 🧠 **Gemini 3 Preview Support** - Full support for gemini-3-flash-preview and gemini-3-pro-preview with extended thinking capabilities
72
74
- 🎯 **Tool Execution Control** – Use `prepareStep` to enforce specific tool calls, change the LLM models per step in multi-step agentic executions. Prevents LLMs from skipping required tools. Use `toolChoice` for static control, or `prepareStep` for dynamic per-step logic. → [GenerateOptions Reference](docs/api/type-aliases/GenerateOptions.md#preparestep)
73
75
-**Structured Output with Zod Schemas** – Type-safe JSON generation with automatic validation using `schema` + `output.format: "json"` in `generate()`. → [Structured Output Guide](docs/features/structured-output.md)
Gemini 3 models use a **native `@google/genai` SDK path** inside NeuroLink that bypasses the Vercel AI SDK. This is required because the Vercel layer strips the `thoughtSignature` token that Gemini 3 attaches to every tool-calling response — without it, conversation history replays break after the first agentic step.
350
+
351
+
### How It Works
352
+
353
+
When NeuroLink detects a Gemini 3 model + tools, it routes to the native path automatically. You use the same SDK API — nothing changes on your end:
354
+
355
+
```typescript
356
+
const result =awaitneurolink.generate({
357
+
input: { text: "Run my agentic workflow..." },
358
+
provider: "vertex",
359
+
model: "gemini-3-flash-preview",
360
+
tools: { myTool: { ... } },
361
+
sessionId: "session-123", // enables Redis conversation history
362
+
maxSteps: 20,
363
+
});
364
+
```
365
+
366
+
### `thoughtSignature` and History Replay
367
+
368
+
Gemini 3 returns a `thoughtSignature` token with every response that includes function calls. This token must be echoed back as a **sibling field** on each `functionCall` part in conversation history:
369
+
370
+
```typescript
371
+
// How NeuroLink stores it in Redis → how it replays it to Gemini
Without this, Gemini treats each step as a new conversation and stops calling tools after step 1.
379
+
380
+
### Parallel Tool Calls and `stepIndex`
381
+
382
+
Within one agentic step, Gemini can return multiple function calls simultaneously. NeuroLink tags every stored tool call/result with a `stepIndex` (integer, increments per step) so that `prependConversationHistory` can group them into the correct single model turn:
383
+
384
+
```
385
+
Step 1 → model turn: [functionCall(toolA), functionCall(toolB)] // stepIndex: 1
386
+
user turn: [functionResponse(toolA), functionResponse(toolB)]
387
+
Step 2 → model turn: [functionCall(toolC)] // stepIndex: 2
388
+
user turn: [functionResponse(toolC)]
389
+
```
390
+
391
+
Putting two steps into separate model turns would create consecutive model turns, which Gemini rejects with a validation error.
When the same `sessionId` is used by multiple agentic loop invocations (for example, an orchestrator spawning a child agent via `continueOrchestratorWorkflow`), each invocation restarts `stepIndex` at 1. Without isolation, step 1 from Execution A and step 1 from Execution B would be grouped into the same model turn, producing an invalid Gemini history.
396
+
397
+
NeuroLink assigns a UUID `executionId` to each invocation and uses it as a prefix in the grouping key:
398
+
399
+
```
400
+
Execution A, step 1 → key "exec:<uuid-A>:1"
401
+
Execution B, step 1 → key "exec:<uuid-B>:1" ← never collide
402
+
```
403
+
404
+
Old messages without an `executionId` fall back to the `turn:<counter>:<stepIndex>` key for backward compatibility.
405
+
406
+
### Timeout Defaults
407
+
408
+
The native generate path defaults to **5 minutes** (300 s) to accommodate long multi-step agentic loops. Override with `options.timeout`:
409
+
410
+
```typescript
411
+
const result =awaitneurolink.generate({
412
+
...
413
+
timeout: "10m", // "2m", "300s", or milliseconds
414
+
});
415
+
```
416
+
417
+
If the timeout fires mid-stream, NeuroLink surfaces a `TimeoutError` rather than returning empty content silently.
418
+
419
+
### Constraints
420
+
421
+
-**No tools + JSON schema simultaneously** — Gemini 3 cannot use function calling and `structuredOutput` with a JSON schema at the same time. NeuroLink automatically disables tools when a JSON schema output is requested and logs a warning.
422
+
-**Gemini 3 only** — This native path activates only for model names matching the Gemini 3 pattern. All other Vertex AI models continue to use the standard Vercel AI SDK path.
0 commit comments