Skip to content

fix(google): add Gemini 3.1 Flash Live compatibility - #1186

Closed
swayamg20 wants to merge 2 commits into
livekit:mainfrom
swayamg20:fix/gemini-31-compatibility
Closed

fix(google): add Gemini 3.1 Flash Live compatibility#1186
swayamg20 wants to merge 2 commits into
livekit:mainfrom
swayamg20:fix/gemini-31-compatibility

Conversation

@swayamg20

@swayamg20 swayamg20 commented Mar 31, 2026

Copy link
Copy Markdown

Problem

Two things were broken when using gemini-3.1-flash-live-preview with the Live API:

  1. pushAudio() was sending mediaChunks (deprecated) Gemini 3.1 requires the audio field directly.
  2. generateReply() was using sendClientContent to trigger generation, Gemini 3.1 rejects this with "Request contains an invalid argument."

Additionally, empty tools and sessionResumption fields were being sent in session setup even when unpopulated, causing rejections on stricter model versions.

Changes

  • Replace mediaChunks with audio field in pushAudio() and all related event handling
  • Replace sendClientContent with sendRealtimeInput({ text }) in generateReply() — works across all Live API models
  • Only include tools in session config when there are actual declarations
  • Only include sessionResumption when a handle exists
  • Add gemini-3.1-flash-live-preview to LiveAPIModels type

Testing

Tested against gemini-3.1-flash-live-preview audio streaming and reply generation both work correctly.

Closes #1179

@changeset-bot

changeset-bot Bot commented Mar 31, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 0c40602

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 22 packages
Name Type
@livekit/agents-plugin-google Patch
@livekit/agents Patch
@livekit/agents-plugin-anam Patch
@livekit/agents-plugin-baseten Patch
@livekit/agents-plugin-bey Patch
@livekit/agents-plugin-cartesia Patch
@livekit/agents-plugin-deepgram Patch
@livekit/agents-plugin-elevenlabs Patch
@livekit/agents-plugin-hedra Patch
@livekit/agents-plugin-inworld Patch
@livekit/agents-plugin-lemonslice Patch
@livekit/agents-plugin-livekit Patch
@livekit/agents-plugin-neuphonic Patch
@livekit/agents-plugin-openai Patch
@livekit/agents-plugin-phonic Patch
@livekit/agents-plugin-resemble Patch
@livekit/agents-plugin-rime Patch
@livekit/agents-plugin-sarvam Patch
@livekit/agents-plugin-silero Patch
@livekit/agents-plugin-trugen Patch
@livekit/agents-plugin-xai Patch
@livekit/agents-plugins-test Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@CLAassistant

CLAassistant commented Mar 31, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

devin-ai-integration[bot]

This comment was marked as resolved.

Always send sessionResumption: {} to opt in on first connect.
The previous fix omitted it entirely when no handle existed, creating
a deadlock where the server never enabled resumption tracking.
@swayamg20
swayamg20 force-pushed the fix/gemini-31-compatibility branch from 577e906 to 0c40602 Compare March 31, 2026 09:19
@toubatbrian

toubatbrian commented Mar 31, 2026

Copy link
Copy Markdown
Contributor

Thanks for the work here. I’m taking over this effort in a follow-up PR: #1189.

Given the discussion and outcome in Python, we should hold off on the generateReply workaround for Gemini 3.1 until we have a proper upstream direction from the Gemini team. Reference: livekit/agents#5251.

So the current plan is to defer full Gemini 3.1 generateReply behavior changes until there is an official fix/guidance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

@livekit/agents-plugin-google 1.2.1 fails with gemini-3.1-flash-live-preview

3 participants