feat(live): improve conversation turn-taking and response pacing#125
Open
rich7420 wants to merge 3 commits into
Open
feat(live): improve conversation turn-taking and response pacing#125rich7420 wants to merge 3 commits into
rich7420 wants to merge 3 commits into
Conversation
The low-risk subset of the turn-taking cadence work (research-located root causes; the device-A/B-gated barge-in rework ships separately): - WS delta-revive latch: a user-speech interruption stops playback but late in-flight output-audio deltas restarted it mid-sentence. The stop now latches the interrupted response; its deltas are dropped until the next response boundary. Pure decision helpers + LiveOutputAudioDropLatchTests. - semantic_vad is the default turn detection for FRESH installs (struct default). Legacy installs keep their stored mode: save always persisted turnDetectionMode, so a stored server_vad is indistinguishable from a deliberate choice — silently flipping would stomp it. Fail-closed, idempotent by construction (re-derived from install age, no version flag). - Numeric brevity personas (1-3 short sentences per turn; lists one item at a time) in BOTH the bundled copy and src/prompts/registry.ts (the runtime override — kept in sync), and a ~800-token voice default for maxResponseOutputTokens (config-overridable) now applied on the WebRTC path via the post-connect session.update (the client-secret mint schema rejects the field — documented at the call site). ## Validation - bunx tsc --noEmit clean; TS suites incl. voiceprint canary: 425 pass / 0 fail. iOS device build: BUILD SUCCEEDED; simulator suites incl. the new latch tests + settings-migration coverage: TEST SUCCEEDED. - 4 review rounds + a fresh-eyes convergence review returning zero must-change items. ## Limitations / Follow-ups 1. Existing installs must switch Turn detection to Semantic manually (Live settings) — deliberate, see above. 2. Device A/B pending for the WebRTC mic-mute barge-in rework, echo hardening, and truncation flows (tracked in docs/live-turn-taking.md). (cherry picked from commit 156277b529e6dbd30be646b650a883f86e51e1b9)
docs/live-turn-taking.md: the three user-facing problems with measured root causes, what this increment ships (latch, semantic_vad default semantics, numeric personas, token cap incl. the mint-schema finding), the pending device-A/B plan (mic-unmute, echo hardening, truncation — and the echo-to-false-interrupt risk), and how the three verbosity controls compose. Plus a status note in the ambient-agent plan. - Docs-only; claims cite file paths in the current tree. 1. None. (cherry picked from commit 783062c3ce6f7bdd66b02d2f819e94626ba25807)
User-observed: the camera takes noticeably long to come up on Live start. Investigation (timing-instrumented start path): the camera was brought up DEAD LAST — behind the awaited parallel-mic warm-up (guaranteed by the voiceprint liveUpload coercion, hundreds of ms of voice-processing engine spin-up), the serial boot-context RPC, and the WebRTC connect including an up-to-8s session-config ack wait — despite the AVCaptureSession being fully independent of the handshake (frames buffer until the data channel is ready). - The camera task now launches BEFORE the mic warm-up + connect and is JOINED at the original point, so failure semantics are unchanged (a failed connect still reaches the catch block's stopVisualStream()). - startVisualStreamIfNeeded takes the session's EFFECTIVE config explicitly: before connect completes activeConfig is unset, so reading liveConfig would hit the draft config and bypass per-session overrides (the enrollment session's forced camera-off would leak a camera on). - New "recording mic" timing mark isolates the mic warm-up in the Start timing diagnostics; the visual mark now reports join wait + concurrent total. - xcodebuild build (device): BUILD SUCCEEDED; simulator suites (enrollment, settings migration, drop latch, identity): TEST SUCCEEDED. - Real-device latency delta to be read from the Start timing diagnostics on the next install. 1. The boot-context RPC remains serial before connect (deliberate reachability probe); making it concurrent is a further win if startup is still slow. 2. The up-to-8s config-ack wait inside connect is untouched (structural). (cherry picked from commit 7352a24184a4acc534d48fecce3a6e65a2bef9c5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Live conversations had three rough edges: Hawky jumped in during short mid-sentence pauses, spoke replies that ran too long for voice, and — after being interrupted — sometimes resumed the very reply the user had just cut off. This PR fixes all three, plus a startup-latency issue found along the way. Independent of #123; whichever merges second takes a trivial context rebase in LiveSessionStore.
response.createdand on the interrupted reply's own terminal event; cleared on stop/reconnect — so it can never swallow the next reply's audio and can never get stuck). Decision logic is extracted into pure helpers with 13 unit tests.semantic_vad(eagernessauto), which decides end-of-turn from what is being said instead of a fixed 500ms silence window — the direct fix for replies barging into thinking pauses. Existing installs keep their stored mode: a persistedserver_vadcannot be distinguished from a deliberate choice, so it is never silently flipped (users can switch in Live settings).session.updateon both transports; the client-secret mint schema rejects that field, so a unit-tested invariant keeps it off both mint paths.recording micmark in the Start-timing diagnostics makes the remaining startup costs measurable.Design notes and the follow-up plan live in
docs/live-turn-taking.md.Validation
bunx tsc --noEmitclean;bun run test: 4750 pass / 5 skip (3 failures are pre-existing flaky watcher-timeout tests on the main tip: the suite passes 39/0 in isolation and a parallel full run exits 0).semantic_vadpayload emitted on both transports; token cap present onsession.updateand absent from both mint bodies; persona copies byte-identical.Limitations / Follow-ups
docs/live-turn-taking.md).