feat(voice): auto-disable realtime server-side turn detection - #2158
feat(voice): auto-disable realtime server-side turn detection#2158rosetta-livekit-bot[bot] wants to merge 3 commits into
Conversation
- restore withAzure's destructured signature; the pre-existing code already reassigned its params, so only the default needed moving out - name the session options type instead of repeating the object literal - drop xAI's session() override, a no-op the canDisableTurnDetection gate already covers - use a template literal for the runtime-change warning
🦋 Changeset detectedLatest commit: dfe73af The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if (hasTurnDetection) { | ||
| recognitionOptions.turnDetection = turnDetection; | ||
| recognitionOptions.turnDetection = this._resolvedTurnDetection; | ||
| } |
There was a problem hiding this comment.
🔴 Turning off a running session's turn-detection setting no longer takes effect in the speech recognizer
When turn detection is cleared at runtime, the recognizer is handed an "unchanged" value instead of an explicit reset (recognitionOptions.turnDetection = this._resolvedTurnDetection at agents/src/voice/agent_activity.ts:1123), so the previously configured mode (e.g. manual) keeps driving speech recognition.
Impact: After switching turn detection back to automatic, the agent can keep behaving as if the old mode (such as manual push-to-talk) were still active and stop replying on its own.
Null vs undefined semantics in AudioRecognition.updateOptions
AudioRecognition.updateOptions treats undefined as "leave unchanged" and only resets when a value (including null) is supplied: if (options.turnDetection !== undefined) { this.turnDetectionMode = options.turnDetection ?? undefined; } (agents/src/voice/audio_recognition.ts:543-545).
Previously the raw parameter was forwarded, so AgentSession.updateOptions({ turnDetection: null }) propagated null and cleared the recognizer's mode. Now _resolvedTurnDetection is forwarded, and _resolveTurnDetection(null ?? undefined) returns undefined, so the reset is silently dropped while this.turnDetectionMode on the activity is cleared — the two diverge. turnDetectionMode === 'manual' gates auto-commit of user turns in the recognizer (agents/src/voice/audio_recognition.ts:1204, 1377, 1396).
| if (hasTurnDetection) { | |
| recognitionOptions.turnDetection = turnDetection; | |
| recognitionOptions.turnDetection = this._resolvedTurnDetection; | |
| } | |
| if (hasTurnDetection) { | |
| // pass null (not undefined) so AudioRecognition clears its mode instead of keeping it | |
| recognitionOptions.turnDetection = this._resolvedTurnDetection ?? null; | |
| } |
Was this helpful? React with 👍 or 👎 to provide feedback.
| if (hasTurnDetection) { | ||
| recognitionOptions.turnDetection = turnDetection; | ||
| recognitionOptions.turnDetection = this._resolvedTurnDetection; | ||
| } |
There was a problem hiding this comment.
🔴 Turning turn detection back to automatic at runtime no longer takes effect on the live audio pipeline
When turn detection is cleared at runtime, the new value handed to the running audio pipeline (recognitionOptions.turnDetection = this._resolvedTurnDetection at agents/src/voice/agent_activity.ts:1123) is undefined, which the pipeline treats as "leave unchanged", so the previous mode (e.g. manual) keeps running.
Impact: A session that switches away from manual/vad turn detection at runtime keeps behaving as before, so user turns can stop being detected and the agent stops replying.
Why undefined is swallowed by AudioRecognition.updateOptions
AudioRecognition.updateOptions only applies the value when it is not undefined (agents/src/voice/audio_recognition.ts:543-545: if (options.turnDetection !== undefined) { this.turnDetectionMode = options.turnDetection ?? undefined; }). Before this PR the activity forwarded the raw argument, so turnDetection: null reached the pipeline as null and correctly reset turnDetectionMode to undefined. Now the activity forwards this._resolvedTurnDetection, which is undefined for null (and also for a detector that fails the runtime preconditions in _resolveTurnDetection), so the pipeline silently keeps the stale mode while the activity's own turnDetectionMode is cleared — the two diverge. turnDetectionMode === 'manual' gates turn commit/endpointing in many places (agents/src/voice/audio_recognition.ts:1051, 1204, 1377, 1396).
| if (hasTurnDetection) { | |
| recognitionOptions.turnDetection = turnDetection; | |
| recognitionOptions.turnDetection = this._resolvedTurnDetection; | |
| } | |
| if (hasTurnDetection) { | |
| recognitionOptions.turnDetection = this._resolvedTurnDetection ?? null; | |
| } |
Was this helpful? React with 👍 or 👎 to provide feedback.
cf886cd to
dfe73af
Compare
1 participant
Ports livekit/agents PR #6495.
Summary
canDisableTurnDetectioncapability and per-sessionturnDetectionDisabledcreation option.Source diff coverage
livekit-agents/livekit/agents/llm/realtime.pyagents/src/llm/realtime.ts: addedcanDisableTurnDetectioncapability andsession({ turnDetectionDisabled })option.livekit-agents/livekit/agents/llm/realtime_fallback_adapter.pylivekit-agents/livekit/agents/voice/agent_activity.pyagents/src/voice/agent_activity.ts: ported session-scoped realtime turn-detection resolution, validation/warnings, session creation flag, reuse guard, VAD wiring, interruption checks, commit behavior, and gatekeeping checks using TS naming and existing target flow.livekit-agents/livekit/agents/voice/agent_session.pyagents/src/voice/agent_session.ts: track whetherturnDetectionwas explicitly supplied and update that state on runtime changes.livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/experimental/realtime/realtime_model.pylivekit-plugins/livekit-plugins-google/livekit/plugins/google/realtime/realtime_api.pyplugins/google/src/realtime/realtime_api.ts: accept the new session option while leaving disabling unsupported.livekit-plugins/livekit-plugins-nvidia/livekit/plugins/nvidia/experimental/realtime/realtime_model.pylivekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/realtime_model.pyplugins/openai/src/realtime/realtime_model.ts: mark default server VAD as disable-capable, preserve Azure intent, and apply per-session disabled turn detection. Python's model-levelupdate_optionspropagation test has no JS counterpart because this target model has no model-level realtimeupdateOptionsAPI.livekit-plugins/livekit-plugins-phonic/livekit/plugins/phonic/realtime/realtime_model.pyplugins/phonic/src/realtime/realtime_model.ts: accept the new session option while leaving disabling unsupported.livekit-plugins/livekit-plugins-ultravox/livekit/plugins/ultravox/realtime/realtime_model.pylivekit-plugins/livekit-plugins-xai/livekit/plugins/xai/realtime/realtime_model.pyplugins/xai/src/realtime/realtime_model.ts: keep OpenAI-derived disabling unsupported for xAI and ignore the new session option.tests/fake_realtime.pyagents/src/voice/realtime_auto_disable_turn_detection.test.ts: JS fake realtime model/session recordsturnDetectionDisabledfor ported tests.tests/test_realtime/test_openai_realtime_model.pyplugins/openai/src/realtime/realtime_model.test.ts: added OpenAI realtime default/disabled/Azure/session assertions.tests/test_realtime_auto_disable_turn_detection.pyagents/src/voice/realtime_auto_disable_turn_detection.test.ts: added equivalent JS coverage for resolution matrix, runtime warnings, default warning suppression, session creation flag, and handoff reuse behavior. Fallback adapter cases are not applicable because the target lacks that adapter.Verification
pnpm buildOPENAI_API_KEY= pnpm test agents plugins/openai plugins/google plugins/phonic plugins/xaipnpm lintcue-cliruntime validation with a temporary fake realtime JS agent: assertedconversation_item_added(.item.message.content[0].text =~ "rtTurnDetectionEnabled.*false.*turnDetectionDisabled.*true")Notes
OPENAI_API_KEYenabled optional OpenAI integration tests; those failed on local Silero/model setup unrelated to this port. The rerun above unsetsOPENAI_API_KEYso optional external tests are skipped.pnpm api:checkis blocked by an unrelated package (@livekit/agents-plugin-liveavatar) withoutapi-extractor.json; filtered API checks for touched packages are currently blocked in@livekit/agentsby API Extractor's existingexport * aslimitation againstagents/dist/index.d.ts.Ported from livekit/agents#6495
Original PR description
Auto-disables a realtime model's server-side turn detection for the session when the user configured client-side turn-taking (a TurnDetector,
vad,manual, or an explicit interruption mode) and the model supports it, so client-side EoT/barge-in works without also passingturn_detection=Noneto the RealtimeModel.Adds
RealtimeCapabilities.can_disable_turn_detection(OpenAI sets it from the ctorturn_detectiongiven-ness) and asession(turn_detection_disabled=...)seam applied on the initialsession.updatevia the per-session opts copy, so the model is never mutated and stays reusable.AgentActivityresolves the effective state once (_rt_turn_detection_enabled) and routes every server-side turn-detection check through it.Other realtime plugins accept-and-ignore the flag (
can_disable_turn_detectionstays False): xAI (client turn-taking not stable in testing) and Gemini/others (nocommit_audio/clear_audio; turns are activity-signal based).Stacked on
longc/realtime-barge-in; this PR targets that branch, not main.