fix(amd): synthesize the speech start when the end arrives without one - #6639
fix(amd): synthesize the speech start when the end arrives without one#6639biztex wants to merge 3 commits into
Conversation
When AMD attaches after the callee has already started speaking (outbound calls answered mid-greeting), the start-of-speech event fires before start_listening() opens the gate and is dropped. The later end-of-speech event then found no start timestamp and was discarded with a warning - leaving no timers armed, no classification running, and the verdict stalled until the 20s detection_timeout. The reported symptom: 7-16s of held playout on human calls, AMDResult.delay negative, speech_duration=0.00. Synthesize the missing start from the moment listening began (the earliest speech we could have observed) instead of dropping the signal: the no-speech timer is cancelled since speech clearly happened, and the normal short-greeting/long-speech verdict paths run. Ordinary orderings are unchanged, and events before start_listening() stay gated as documented. Fixes livekit#5616
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
The VAD reports end-of-speech only after a minimum trailing silence, so the computed end points back in time; when the listening gate opened during that pause the synthesized start was later than the end and the reported speech_duration went negative in the emitted AMDPredictionEvent.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
the session VAD runs independently of the AMD gate, so ringback/early-media audio can produce an end event that lands just after start_listening() while the speech itself finished before the gate opened. Clamping that into a zero-length greeting cancelled the no-speech timer and committed an instant HUMAN/short_greeting verdict from zero observed speech. Such a segment is now dropped like its start event was: the start is only synthesized when the speech actually overlaps the listening window, which also keeps the synthesized duration strictly positive.
|
Note on the red It is a teardown error in Evidence it isn't this branch: the previous commit here (b343925) passed the same workflow, this PR only touches Happy to send a separate PR closing the session in that test (or giving it a fixture that does) so the genai clients stop being left to the GC, if that's wanted. |
1 participant
Fixes #5616
Problem
AMD.execute()held playout for 7-16s on outbound human calls. The reported trace shows the verdict landing at +19.4s — the 20sdetection_timeout— withspeech_duration=0.00and a negativeAMDResult.delay, plus the tell-taleWARNING: on_user_speech_ended called before on_user_speech_started.Root cause
When AMD attaches after the SIP participant connects (per
examples/telephony/amd.py, and as @chenghao-mou hypothesized on the issue), the callee may already be mid-greeting. The start-of-speech event fires beforestart_listening()opens the gate, so the_listening_guarddrops it. The later end-of-speech event then finds_speech_started_at is Noneand was discarded entirely — no silence timer armed, no classification started, nothing scheduled at all. The classifier sat idle untildetection_timeoutforced anuncertainverdict, and speech-playout authorization was held the whole time.Change
Synthesize the missing start instead of dropping the end signal:
_speech_started_atis set to the moment listening began — the earliest speech the classifier could have observed (a conservative lower bound on the real duration) — the no-speech timer is cancelled (speech clearly happened), and the normal short-greeting / long-speech verdict paths run. Ordinary event orderings are untouched, and events arriving beforestart_listening()remain gated as documented.Tests
Three new cases in
tests/test_amd_classifier.pyon the existing harness: end-without-start now arms the verdict path and emits a HUMAN/short_greeting verdict (previously nothing until timeout); events beforestart_listening()stay no-ops; and a real start timestamp is never overwritten by the synthesis.