fix(preconnect): send audio buffer with resolved track id

[pblazej]

Fixes #1060.

Root cause

The pre-connect buffer send was triggered by the agent becoming .active, which can happen as early as the join response — before the microphone publish resolves the local track's sid (assigned only at the end of _publish, in Participant.add(publication:)). The buffer then went out with an empty trackId attribute, and agents drop such buffers (pre-connect audio received but no trackId), falling back to live mic after a 3s timeout and losing the prelude.

Fix

Invert the trigger, mirroring the JS SDK: the send now starts when the recorder's track is published (didPublishTrack, sid in hand) and waits for the first active agent (AsyncCompleter<Participant.Identity> resumed on agent activation, with a fast path via agentParticipants for agents already active). The race is structurally impossible rather than guarded.

Hardening that falls out of the new shape:

• Exactly-once sending via atomic stream-ownership take inside one state.mutate (replaces the racy check-then-act sent flag); the flush path takes ownership the same way, so the stream can never have two consumers.
• Republish after a full reconnect no longer re-fires the send (guard + delegate removal after completion).
• No-agent sessions tear down silently as before (cancelled/timed-out waits don't invoke onError).
• Recording timeout no longer truncates fractional seconds and clamps non-finite values instead of trapping.
• Avatar workers (lk.publish_on_behalf) are excluded from agent selection, consistently with Room.agentParticipants.
• LocalAudioTrackRecorder.render no longer calls assertionFailure on conversion failure (crashes consumer debug builds); logs once per recording instead.

Public API surface is unchanged. sendAudioData(to:agents:on:) keeps its silent-no-op behavior for duplicate/empty calls; calling it before the track is published now throws .invalidState instead of sending a buffer the agent can't bind.

Known limitation (pre-existing outcome): if the only agent activates and then leaves before the publish completes, the buffer is sent to the departed agent and lost — same result as the old send-at-activation behavior when an agent dies mid-flow.

Tests

Moved PreConnectAudioBufferTests from the device-only LiveKitAudioTests to LiveKitCoreTests — TestAudioTrack plus an injected LocalAudioTrackRecorder make the whole flow runnable headless (audio fed via render(pcmBuffer:)). Coverage: attributes/data/exactly-once (including duplicate didPublishTrack re-fire), timeout-flush error path, recorder guards, and a regression test for the #1060 ordering (agent active before publish) that fails on main with trackId == "".

🤖 Generated with Claude Code

[pblazej]

E2E validation against a local server + real livekit-agents handler

Validated the full production chain headless (no simulators): local livekit-server --dev, a macOS CLI harness using the starter-app path — Session(tokenSource: LiteralTokenSource(...)) → session.start() with default preConnectAudio: true and real mic capture — and a Python script driving the real livekit-agents PreConnectAudioHandler (register-before-connect, 3s timeout, as RoomIO does) connected with a kind=agent token.

The agent joined before the client (the ordering that produced the empty trackId on 2.15.1). Agent-side log:

[agent] subscribed track sid=TR_AMNmo9Lg6bYSk2 source=2 audio_features=[2, 3, 4, 6]   # 6 = TF_PRECONNECT_BUFFER
livekit.agents: pre-connect audio received                                            # trackId guard passed
[agent] BIND OK: 12 frames, 1.190s prelude bound to TR_AMNmo9Lg6bYSk2

No pre-connect audio received but no trackId warning, no timeout waiting for pre-connect audio buffer, no live-mic fallback — the buffered prelude was bound to the published mic track.

🤖 Generated with Claude Code

[edvintb]

Validated on iOS + Bluetooth HFP (the original repro path). Cherry-picked both commits onto 2.15.2 and ran a real connect. Fix works end-to-end.

Client (Swift): the send is now triggered by publish, with the sid resolved before it goes out —

PreConnectAudioBuffer …didPublishTrack: Microphone track published, sending audio to the first active agent
LocalParticipant._publish … success LocalTrackPublication(sid: TR_AMDwU9FVqNsXg3, …)
PreConnectAudioBuffer.sendAudioData(to:trackSid:agents:on:) Sent 0.88s = 41KB of audio data to 1 agent(s)

No empty-trackId send, no sending without trackId fallback.

Agent (Python, livekit-agents 1.5.17): the buffer now arrives with a track id and is bound instead of dropped —

pre-connect audio received   track_id='TR_AMDwU9FVqNsXg3'  duration=0.88
pre-connect audio buffer pushed  track_id='TR_AMDwU9FVqNsXg3'
start reading stream  source='SOURCE_MICROPHONE'
[stt] first audio frame reached STT (+2167ms)

The pre-connect audio received but no trackId warning and the timeout waiting for pre-connect audio buffer (~3s) are both gone. Time-to-first-STT dropped from ~5228ms → ~2167ms — the eliminated 3s timeout.

This is the correct fix for the empty-trackId race also tracked in #1063 — inverting the trigger (send on publish, sid in hand) supersedes the weaker poll-for-sid-then-fall-back-to-empty approach. 👍 from me to merge.

[devin-ai-integration]

Devin Review found 1 new potential issue.

View 2 additional findings in Devin Review.

[devin-ai-integration]

🔴 Pre-connect audio feature relies on a helper that does not exist in the codebase

The audio stream is handed over using a helper that was never added to the project (.take() at Sources/LiveKit/Core/PreConnectAudioBuffer.swift:122 and :164), so the pre-connect audio code cannot be built at all.
Impact: The SDK fails to build, so nothing that depends on it ships.

Missing Optional.take() helper used for stream ownership transfer

The final commit of this PR ("refactor(preconnect): use Optional.take() for stream ownership transfer") introduces $0.audioStream.take() in stopRecording(flush:) (Sources/LiveKit/Core/PreConnectAudioBuffer.swift:122) and state.audioStream.take() in sendAudioData(to:trackSid:agents:on:) (Sources/LiveKit/Core/PreConnectAudioBuffer.swift:164).

Swift's standard library has no Optional.take() (that is a Rust idiom), and a repo-wide search finds no extension Optional and no func take definition anywhere in Sources/, Tests/, or test support. The two call sites in this file are the only occurrences of .take() in the repository.

The intended semantics are "read the value and set the optional to nil atomically within the same state.mutate". Either add a small mutating func take() -> Wrapped? extension on Optional under Sources/LiveKit/Support/, or inline the read-then-nil inside the existing state.mutate blocks.

Prompt for agents

The pre-connect buffer now transfers ownership of the recorded audio stream with `audioStream.take()` in two places: `stopRecording(flush:)` (Sources/LiveKit/Core/PreConnectAudioBuffer.swift:122) and the internal `sendAudioData(to:trackSid:agents:on:)` (Sources/LiveKit/Core/PreConnectAudioBuffer.swift:164). No `take()` method exists on `Optional` in the Swift standard library, and a repo-wide search shows no `extension Optional` or `func take` defined anywhere in this project, so these call sites do not resolve. Fix by either adding a small, documented `mutating func take() -> Wrapped?` extension on `Optional` in Sources/LiveKit/Support/ (returning the current value and setting self to nil), or by inlining the equivalent read-then-nil logic inside the existing `state.mutate` blocks so the ownership transfer stays atomic under the StateSync lock.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[pblazej]

https://developer.apple.com/documentation/swift/optional/take()

[pblazej]

Unrelated test failures fixed elsewhere.