feat(live): natural turn-taking P0 set#124
Closed
rich7420 wants to merge 3 commits into
Closed
Conversation
The low-risk subset of the turn-taking cadence work (research-located root causes; the device-A/B-gated barge-in rework ships separately): - WS delta-revive latch: a user-speech interruption stops playback but late in-flight output-audio deltas restarted it mid-sentence. The stop now latches the interrupted response; its deltas are dropped until the next response boundary. Pure decision helpers + LiveOutputAudioDropLatchTests. - semantic_vad is the default turn detection for FRESH installs (struct default). Legacy installs keep their stored mode: save always persisted turnDetectionMode, so a stored server_vad is indistinguishable from a deliberate choice — silently flipping would stomp it. Fail-closed, idempotent by construction (re-derived from install age, no version flag). - Numeric brevity personas (1-3 short sentences per turn; lists one item at a time) in BOTH the bundled copy and src/prompts/registry.ts (the runtime override — kept in sync), and a ~800-token voice default for maxResponseOutputTokens (config-overridable) now applied on the WebRTC path via the post-connect session.update (the client-secret mint schema rejects the field — documented at the call site). ## Validation - bunx tsc --noEmit clean; TS suites incl. voiceprint canary: 425 pass / 0 fail. iOS device build: BUILD SUCCEEDED; simulator suites incl. the new latch tests + settings-migration coverage: TEST SUCCEEDED. - 4 review rounds + a fresh-eyes convergence review returning zero must-change items. ## Limitations / Follow-ups 1. Existing installs must switch Turn detection to Semantic manually (Live settings) — deliberate, see above. 2. Device A/B pending for the WebRTC mic-mute barge-in rework, echo hardening, and truncation flows (tracked in docs/live-turn-taking.md). (cherry picked from commit 156277b529e6dbd30be646b650a883f86e51e1b9)
docs/live-turn-taking.md: the three user-facing problems with measured root causes, what this increment ships (latch, semantic_vad default semantics, numeric personas, token cap incl. the mint-schema finding), the pending device-A/B plan (mic-unmute, echo hardening, truncation — and the echo-to-false-interrupt risk), and how the three verbosity controls compose. Plus a status note in the ambient-agent plan. - Docs-only; claims cite file paths in the current tree. 1. None. (cherry picked from commit 783062c3ce6f7bdd66b02d2f819e94626ba25807)
User-observed: the camera takes noticeably long to come up on Live start. Investigation (timing-instrumented start path): the camera was brought up DEAD LAST — behind the awaited parallel-mic warm-up (guaranteed by the voiceprint liveUpload coercion, hundreds of ms of voice-processing engine spin-up), the serial boot-context RPC, and the WebRTC connect including an up-to-8s session-config ack wait — despite the AVCaptureSession being fully independent of the handshake (frames buffer until the data channel is ready). - The camera task now launches BEFORE the mic warm-up + connect and is JOINED at the original point, so failure semantics are unchanged (a failed connect still reaches the catch block's stopVisualStream()). - startVisualStreamIfNeeded takes the session's EFFECTIVE config explicitly: before connect completes activeConfig is unset, so reading liveConfig would hit the draft config and bypass per-session overrides (the enrollment session's forced camera-off would leak a camera on). - New "recording mic" timing mark isolates the mic warm-up in the Start timing diagnostics; the visual mark now reports join wait + concurrent total. - xcodebuild build (device): BUILD SUCCEEDED; simulator suites (enrollment, settings migration, drop latch, identity): TEST SUCCEEDED. - Real-device latency delta to be read from the Start timing diagnostics on the next install. 1. The boot-context RPC remains serial before connect (deliberate reachability probe); making it concurrent is a further win if startup is still slow. 2. The up-to-8s config-ack wait inside connect is untouched (structural). (cherry picked from commit 7352a24184a4acc534d48fecce3a6e65a2bef9c5)
Collaborator
Author
|
Superseded by a standalone re-land off main — turn-taking is functionally independent of voiceprint, so it shouldn't be gated on #123's review. Replacement PR incoming from |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #123 (voiceprint v1). The low-risk subset of the conversation-cadence work — the three user-reported problems (premature replies on short pauses, overly long spoken answers, inconsistent barge-in) with research-located root causes:
response.createdAND the interrupted response's own terminal event, cleared on stop/reconnect — cannot eat the next response's audio, cannot strand).server_vadis indistinguishable from a deliberate user choice, so we refuse to silently flip it (fail-closed; idempotent by construction, no migration flag).session.update(the GA client-secret mint schema rejects the field — documented, with a unit-tested never-on-mint invariant).recording mictiming mark isolates the warm-up in Start-timing diagnostics.Design doc:
docs/live-turn-taking.md(root causes, what ships, the pending device-A/B plan for the WebRTC mic-mute barge-in rework + echo hardening, and how the three verbosity controls compose).Validation
bunx tsc --noEmitclean; TS suites incl. the voiceprint canary: 424 pass / 0 fail on this stack.Limitations / Follow-ups