fix(voice): make STT the user_state source when turn_detection is stt - #6641
Open
biztex wants to merge 3 commits into
Open
fix(voice): make STT the user_state source when turn_detection is stt#6641biztex wants to merge 3 commits into
biztex wants to merge 3 commits into
Conversation
With turn_detection=stt, both VAD and STT wrote user_state: in noisy environments VAD flips it to speaking on background noise the STT never transcribes, breaking everything keyed on user state (user_away_timeout, filler triggering). The only workaround, vad=None, also gives up VAD-based interruption sensing - the two concerns are independent. Gate only the user_state writes in the on_start/end_of_speech hooks: when the user has opted into STT-driven turn detection, VAD-sourced events (ev is not None) skip the state update while STT speech events (ev is None) drive it. Every other VAD role is untouched - interruption sensing, endpointing, overlap tracking, pause/false-interruption timers all still run on VAD events, which is exactly the separation the issue asks for. Default and vad modes are byte-identical. Fixes livekit#5580
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…ng state in stt mode, "speaking" can be entered by writers the STT will never clear: claim_user_turn re-derives it on release from the VAD-driven silence event (background noise during a text turn), and update_options can switch to stt mid-speech. Dropping every VAD end-of-speech left the state stuck at "speaking" with the away timer never re-armed. Track whether an STT-driven segment is open (paired ev=None hook calls) and let the VAD end clear the state only when it is not - an STT-authored "speaking" still ends on the STT end-of-speech alone.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…-utterance the stt-driven speaking segment was only closed by an STT end-of-speech, but two paths destroy the stream with that event still pending: the pump recreating the node after an APIError, and _clear_user_turn resetting the stt. The segment now closes on both (mirroring the vad-task teardown), so user_state cannot stay "speaking" with the away timer unarmed and the user_speaking span open until a later utterance is transcribed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5580. Related: #5510 (the noisy-telephony pain this gives an escape hatch for).
Problem
With
turn_detection="stt", both VAD and STT driveuser_state. In noisy environments Silero VAD flips it to"speaking"on background noise the STT never transcribes, breaking everything keyed on user state —user_away_timeout, filler triggering, presence logic. The only workaround isvad=None, which also gives up VAD-based interruption sensing; the two concerns are independent, as the issue lays out.Change
This is the issue's Option A intent, implemented one layer higher than proposed — and that placement is the important part. The activity's
on_start_of_speech/on_end_of_speechhooks receiveev: VADEvent | None, whereNonemeans STT-sourced. Only the_update_user_statewrites are gated: when the user opted into STT-driven turn detection, VAD-sourced events skip the state update and STT events drive it. Gating the whole VAD branch inaudio_recognition(as the earlier stale attempt at this did) would have silently disabled the adaptive-interruption overlap wiring, pause/false-interruption timers, and endpointing that ride on the same hooks — all of which still run on VAD events here, so VAD keeps its interruption-sensing role exactly as the issue requests. No new configuration;"vad"/default modes are byte-identical.Tests
New hermetic
tests/test_stt_user_state_source.pyon the fake-session harness: insttmode, VAD start/end events leaveuser_stateat"listening"(the noise scenario) while STT-sourced events flip it normally; in default mode VAD drives it exactly as before.