Worker-side auditok VAD strips ~20% of speech when used behind transcription-service

[damienlaine]

TL;DR

When linto-stt-whisper workers are used as Celery task workers behind linto-stt-transcription-service (the standard deployment for batch transcription jobs), they apply a second pass of VAD on each chunk they receive. The transcription-service already produced those chunks by cutting at WebRTC-VAD silences, then dispatches them in parallel. The worker-side VAD (default VAD=auditok) reconstructs the chunk by concatenating only the segments it considers speech and passes that truncated audio to Whisper. Everything auditok mistakenly marks as non-speech is dropped before Whisper sees it — and auditok is energy-based, so it over-filters whenever a speaker is far from the mic, speaks softly, or has lower-energy phonemes.

On a 26-minute multi-speaker French meeting, this single behavior costs ~20 % of the speech: pyannote's diarizer claims the audio is 99.7 % speech, but the words returned by the Whisper worker only cover ~79 % of it. We confirmed by running the same chunk with VAD=false and comparing the output — entire 30 s stretches of obvious meeting conversation come back.

This is two issues in one: a fragile default and a redundant pipeline.

Where the second VAD runs

The relevant code is linto_stt/engines/whisper/stt/processing/vad.py (remove_non_speech) called from decoding.py:83-92:

if VAD:
    _, speech_segments, _ = remove_non_speech(audio, use_sample=True, method=VAD, ...)
segments, info = model.transcribe(
    audio,
    ...
    vad_filter=speech_segments if VAD else False,
    **kwargs,
)

Note: vad_filter=speech_segments is the list returned by auditok, not faster-whisper's own Silero options. So Whisper's input is restricted to whatever auditok decided was speech.

And inside remove_non_speech (vad.py:79):

audio_speech = np.concatenate([audio[..., s:e] for s, e in segments], axis=-1)

→ everything outside segments is dropped.

VAD default is auditok (linto_stt/engines/whisper/stt/__init__.py:15-21), and auditok is rule-based on RMS energy. It is the wrong filter for meeting audio.

Why this is an interaction bug, not just a tuning bug

When the worker is used standalone (HTTP service on a whole audio file), having a VAD pre-pass makes sense — Whisper alone would otherwise iterate slowly through silent regions.

When the worker is used behind transcription-service, the upstream splitFile (in transcription-service, transcription/utils/audio.py) already segments the audio at silences via WebRTC VAD (this part does not drop anything — chunks tile the audio:
for start, stop in zip([0] + cuts, cuts + [len(audio)])).
At that point each chunk is mostly speech. Applying a second, more aggressive VAD on each chunk is at best wasted work and at worst — exactly what we observed — speech loss with no way for downstream code to know what was dropped.

Empirical impact (same conv as #128)

Same 26 min multi-speaker French meeting, whisper-large-v3-turbo, compute_type=int8, deployed via linto-deploy charts:

	gaps >5 s total	speech coverage
default (VAD=auditok)	303 s (19.2 %)	~78.9 %
VAD=false	254 s (16.1 %)	~83.9 %
VAD=false + the empty-PROMPT fix (other issue)	192 s (12.1 %)	~87.8 %

Cross-checking against pyannote's diarization output for the same audio: ~333 seconds of audio where pyannote sees a speaker speaking, Whisper output had no word. With VAD=auditok, several 30 s gaps fall on top of multi-speaker turns (pyannote shows speakers actively alternating during those gaps — not a false positive, not silence).

Suggested fixes (any of these would help)

1. Default VAD=false when SERVICE_MODE=task. When the worker is acting as a Celery task in a chunking pipeline, it knows it's not the audio segmenter and should not be filtering speech. The cheapest change is to update linto_stt/engines/whisper/stt/__init__.py:15-21 so the default depends on SERVICE_MODE, or to override VAD=false in the .envdefault of any deployment template that uses SERVICE_MODE=task.

2. Replace auditok with Silero by default. The code already supports VAD=silero (via faster-whisper's VadOptions in remove_non_speech). Silero is ML-based and dramatically better than auditok on meeting audio. Changing linto_stt/engines/whisper/stt/__init__.py:15 to os.environ.get("VAD", "silero") would make the default usable on real meetings without needing per-deployment overrides.

3. Document the failure mode. At a minimum, the README/.envdefault should warn that VAD=auditok is energy-based and unsuitable for far-field or low-SNR speech, and should not be used when the worker sits behind transcription-service.

For now, downstream users have to set VAD=false (or silero) explicitly in their deployment — which is only obvious after spending significant time chasing the missing speech and ruling out the upstream VAD chunker, the chunk merging, faster-whisper temperatures, and compute_type quantization. Worth a more accurate default and/or a docs update.

[Jeronymous]

The pipeline is not really redundant because linto-stt-transcription-service does not remove any silent part, it cuts in the middle of silences to make chunks, but keep everything.
Removing non-speech segments before Whisper is done to avoid hallucinations. For instance silence at the end of a chunk can make Whisper hallucinate things like "Subtitles done by amara.org".

Concerning silero VS auditok, at the time when I tested things, silero was doing weird things, either hallucinating speech or skipping important zones of speech in some cases (see for instance snakers4/silero-vad#396 and linto-ai/whisper-timestamped#242). And auditok seemed to be more robust.
But that can be reconsidered, testing the new versions of silero.

So we have to think more about what is the good solution.
I personnally don't like the solution "Default VAD=false when SERVICE_MODE=task". It would be awkward to have a default value that depends on the serving mode...
We also have to think independently of what is done in linto-stt-transcription-service. And if linto-stt-transcription-service really needs no VAD, then it should be the one disabling VAD in the worker (VAD setting could be passed as a new argument of the API if needed, instead of being an environment variable of the worker)