AudioStream: frames mislabeled with requested sample_rate when audio filter fails to initialize (fallback streams 48 kHz as 16 kHz)

[Sri-Krishna-V]

Summary

When an AudioStream is created with both a non-native sample_rate and a noise-cancellation audio filter, and the filter fails to initialize, the fallback path delivers frames at the codec-native rate (48 kHz) but stamps them with the requested sample rate. Downstream consumers receive 3×-slowed audio with clean-looking metadata — there is no way to detect the mislabel from the frame metadata on the client side.

Environment

• livekit (python rtc) 1.1.13, livekit-plugins-noise-cancellation 0.2.6
• Linux x86_64 (WSL2) — an environment where the Krisp/BVC filter fails new_session and logs failed to initialize the audio filter. it will not be enabled for this session.

Reproduction

from livekit import rtc
from livekit.plugins import noise_cancellation

stream = rtc.AudioStream(
    track,                      # remote mic track (Opus, 48 kHz)
    sample_rate=16000,
    num_channels=1,
    noise_cancellation=noise_cancellation.BVC(),
)
async for event in stream:
    f = event.frame
    # On a machine where the BVC filter fails to initialize:
    # f.sample_rate == 16000, but f.data contains 48 kHz samples.

Observed effect in our app: a speech-to-text service fed from this stream received 3×-slowed "rumble" audio. Feeding the same track without noise_cancellation, or with sample_rate=48000, produces correct audio.

Root cause

In livekit-ffi/src/server/audio_stream.rs (FfiAudioStream::from_track):

1. Because BVC v2 supports_separate_rates(), input_sample_rate is set to the codec clock rate (48000) and the NativeAudioStream is created at 48 kHz — the filter is expected to do the 48 k → 16 k conversion.
2. If audio_filter.new_session(input_sample_rate, output_sample_rate, ...) returns None, the code logs the error and falls back to AudioStreamKind::Native(native_stream) — the 48 kHz stream created in step 1.
3. native_audio_stream_task then re-chunks frames using sample_rate = output_sample_rate (16000) for the emitted AudioFrame metadata, while the samples are still 48 kHz.

The same pattern exists in participant_audio_stream_task (from_participant).

Expected behavior

On filter-initialization failure, the fallback should deliver audio that matches the requested sample_rate — either by recreating the NativeAudioStream at output_sample_rate, or by inserting a resampler in the fallback path. (Alternatively/additionally: surface the failure to the client as an error or event, so applications can react — currently it is only an ffi-side log line.)

Workaround

Request the stream at the codec-native rate so input == output (the mislabel then cannot happen regardless of filter state) and resample client-side:

stream = rtc.AudioStream(track, sample_rate=48000, num_channels=1,
                         noise_cancellation=noise_cancellation.BVC())
resampler = rtc.AudioResampler(48000, 16000, num_channels=1)

Found while end-to-end testing a real-time interpretation agent (Sarvam STT downstream); happy to provide more detail or test a fix.