Skip to content

feat(live): improve conversation turn-taking and response pacing#125

Open
rich7420 wants to merge 3 commits into
mainfrom
live-turn-taking
Open

feat(live): improve conversation turn-taking and response pacing#125
rich7420 wants to merge 3 commits into
mainfrom
live-turn-taking

Conversation

@rich7420

@rich7420 rich7420 commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Live conversations had three rough edges: Hawky jumped in during short mid-sentence pauses, spoke replies that ran too long for voice, and — after being interrupted — sometimes resumed the very reply the user had just cut off. This PR fixes all three, plus a startup-latency issue found along the way. Independent of #123; whichever merges second takes a trivial context rebase in LiveSessionStore.

  • Interrupted replies stay interrupted (WS transport). Stopping playback on user speech was not enough: audio deltas still in flight for the cancelled reply would restart playback mid-sentence. The interruption now latches that reply, and its remaining audio is dropped until the next reply begins (released on response.created and on the interrupted reply's own terminal event; cleared on stop/reconnect — so it can never swallow the next reply's audio and can never get stuck). Decision logic is extracted into pure helpers with 13 unit tests.
  • Turn detection understands sentence breaks. New sessions default to OpenAI's semantic_vad (eagerness auto), which decides end-of-turn from what is being said instead of a fixed 500ms silence window — the direct fix for replies barging into thinking pauses. Existing installs keep their stored mode: a persisted server_vad cannot be distinguished from a deliberate choice, so it is never silently flipped (users can switch in Live settings).
  • Spoken replies sized for voice. The concise persona now gives the model concrete guidance — "1–3 short sentences per turn; for lists or steps, give one item and offer to continue" — kept byte-identical between the bundled copy and the gateway prompt registry that overrides it at runtime. A ~800-token output cap (config-overridable) backs the prompt up, applied via the post-connect session.update on both transports; the client-secret mint schema rejects that field, so a unit-tested invariant keeps it off both mint paths.
  • Camera comes up with the connection, not after it. The camera pipeline used to start dead last — behind the mic warm-up and the WebRTC connect including its up-to-8s session-config ack — despite being fully independent (frames buffer until the data channel is ready). It now starts concurrently and joins at the original point, so failure handling is unchanged. The effective session config is passed explicitly so per-session overrides cannot be bypassed, and a new recording mic mark in the Start-timing diagnostics makes the remaining startup costs measurable.

Design notes and the follow-up plan live in docs/live-turn-taking.md.

Validation

  • bunx tsc --noEmit clean; bun run test: 4750 pass / 5 skip (3 failures are pre-existing flaky watcher-timeout tests on the main tip: the suite passes 39/0 in isolation and a parallel full run exits 0).
  • iOS device build: BUILD SUCCEEDED, zero Swift errors; simulator: drop-latch and settings-default suites TEST SUCCEEDED (17 XCTest + 64 Swift Testing, including the 13 new latch tests).
  • Invariants spot-checked on the final tree: latch engage/drop/release/clear sites; semantic_vad payload emitted on both transports; token cap present on session.update and absent from both mint bodies; persona copies byte-identical.

Limitations / Follow-ups

  1. Existing installs keep their stored turn-detection mode and must switch to Semantic manually in Live settings (deliberate — silently flipping a stored value could override a real user choice).
  2. The WebRTC-transport barge-in rework (the app currently mutes the mic while the assistant speaks), echo hardening, and interruption truncation need on-device A/B testing and ship separately (plan in docs/live-turn-taking.md).
  3. The boot-context RPC still runs serially before connect; it is the next startup-latency candidate if camera bring-up still feels slow.

rich7420 added 3 commits July 13, 2026 20:18
The low-risk subset of the turn-taking cadence work (research-located root
causes; the device-A/B-gated barge-in rework ships separately):

- WS delta-revive latch: a user-speech interruption stops playback but late
  in-flight output-audio deltas restarted it mid-sentence. The stop now
  latches the interrupted response; its deltas are dropped until the next
  response boundary. Pure decision helpers + LiveOutputAudioDropLatchTests.
- semantic_vad is the default turn detection for FRESH installs (struct
  default). Legacy installs keep their stored mode: save always persisted
  turnDetectionMode, so a stored server_vad is indistinguishable from a
  deliberate choice — silently flipping would stomp it. Fail-closed,
  idempotent by construction (re-derived from install age, no version flag).
- Numeric brevity personas (1-3 short sentences per turn; lists one item at a
  time) in BOTH the bundled copy and src/prompts/registry.ts (the runtime
  override — kept in sync), and a ~800-token voice default for
  maxResponseOutputTokens (config-overridable) now applied on the WebRTC path
  via the post-connect session.update (the client-secret mint schema rejects
  the field — documented at the call site).

## Validation

- bunx tsc --noEmit clean; TS suites incl. voiceprint canary: 425 pass / 0
  fail. iOS device build: BUILD SUCCEEDED; simulator suites incl. the new
  latch tests + settings-migration coverage: TEST SUCCEEDED.
- 4 review rounds + a fresh-eyes convergence review returning zero
  must-change items.

## Limitations / Follow-ups

1. Existing installs must switch Turn detection to Semantic manually (Live
   settings) — deliberate, see above.
2. Device A/B pending for the WebRTC mic-mute barge-in rework, echo
   hardening, and truncation flows (tracked in docs/live-turn-taking.md).

(cherry picked from commit 156277b529e6dbd30be646b650a883f86e51e1b9)
docs/live-turn-taking.md: the three user-facing problems with measured root
causes, what this increment ships (latch, semantic_vad default semantics,
numeric personas, token cap incl. the mint-schema finding), the pending
device-A/B plan (mic-unmute, echo hardening, truncation — and the
echo-to-false-interrupt risk), and how the three verbosity controls compose.
Plus a status note in the ambient-agent plan.

- Docs-only; claims cite file paths in the current tree.

1. None.

(cherry picked from commit 783062c3ce6f7bdd66b02d2f819e94626ba25807)
User-observed: the camera takes noticeably long to come up on Live start.
Investigation (timing-instrumented start path): the camera was brought up DEAD
LAST — behind the awaited parallel-mic warm-up (guaranteed by the voiceprint
liveUpload coercion, hundreds of ms of voice-processing engine spin-up), the
serial boot-context RPC, and the WebRTC connect including an up-to-8s
session-config ack wait — despite the AVCaptureSession being fully independent
of the handshake (frames buffer until the data channel is ready).

- The camera task now launches BEFORE the mic warm-up + connect and is JOINED
  at the original point, so failure semantics are unchanged (a failed connect
  still reaches the catch block's stopVisualStream()).
- startVisualStreamIfNeeded takes the session's EFFECTIVE config explicitly:
  before connect completes activeConfig is unset, so reading liveConfig would
  hit the draft config and bypass per-session overrides (the enrollment
  session's forced camera-off would leak a camera on).
- New "recording mic" timing mark isolates the mic warm-up in the Start
  timing diagnostics; the visual mark now reports join wait + concurrent total.

- xcodebuild build (device): BUILD SUCCEEDED; simulator suites (enrollment,
  settings migration, drop latch, identity): TEST SUCCEEDED.
- Real-device latency delta to be read from the Start timing diagnostics on
  the next install.

1. The boot-context RPC remains serial before connect (deliberate reachability
   probe); making it concurrent is a further win if startup is still slow.
2. The up-to-8s config-ack wait inside connect is untouched (structural).

(cherry picked from commit 7352a24184a4acc534d48fecce3a6e65a2bef9c5)
@rich7420 rich7420 changed the title feat(live): natural turn-taking P0 set feat(live): improve conversation turn-taking and response pacing Jul 14, 2026
@rich7420 rich7420 marked this pull request as ready for review July 14, 2026 01:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant