Skip to content

feat(audio): add DPDFNetFilter for real-time speech enhancement - #5667

Open
Aakash326 wants to merge 2 commits into
pipecat-ai:mainfrom
Aakash326:feat/dpdfnet-filter
Open

feat(audio): add DPDFNetFilter for real-time speech enhancement#5667
Aakash326 wants to merge 2 commits into
pipecat-ai:mainfrom
Aakash326:feat/dpdfnet-filter

Conversation

@Aakash326

Copy link
Copy Markdown

Please describe the changes in your PR. If it is addressing an issue, please reference that as well.

Adds DPDFNetFilter, an input audio filter wrapping DPDFNet (Ceva, Apache-2.0) — causal dual-path deep filtering models that run on CPU via ONNX Runtime. Like RNNoise and Koala, no API key or vendor account. New dpdfnet extra; no existing files change except pyproject.toml and uv.lock.

Partly addresses #3266.

Why

RNNoise is currently the only free, self-hostable denoiser in Pipecat. We benchmarked it against four alternatives on 40 two-speaker Hindi telephony clips (65.4 min, degraded one axis at a time, all arms scored on identical files). It came last on every quality axis we measured.

31 degraded clips:

Metric DPDFNet baseline DPDFNet-2 RNNoise
ROC-AUC (VAD) 0.869 0.887 0.749
STOI Δ −0.052 −0.030 −0.257
Speech attenuation 3.8 dB 3.7 dB 8.7 dB
Net gain (nr − attenuation) 10.1 dB 9.2 dB 2.7 dB
Turn latency p50 285 ms 289 ms 461 ms
CPU p90 / 20 ms packet 5.65 ms 16.67 ms 8.62 ms
Algorithmic delay 40 ms 40 ms 20 ms

On the crosstalk subset (n=19) RNNoise collapses to AUC 0.652 / STOI Δ −0.344, against DPDFNet-2 at 0.857 / −0.043. On clean audio, DPDFNet attenuates speech 0.5 dB where RNNoise takes 3.1 dB.

Caveats: RNNoise ran at VHQ, not the QQ default, so this is the model and not a resampling artefact. DPDFNet's 40 ms algorithmic delay is double RNNoise's and sits upstream of VAD, so it adds to end-of-turn latency. dpdfnet2 costs ~2.6× RNNoise's CPU — hence model=, with baseline landing within 0.02 AUC at roughly a third the cost. The corpus is Hindi telephony and deliberately biased toward hard clips, so read it as a comparison between arms, not as absolute scores.

Design

  • Inference runs off the event loop on a dedicated ThreadPoolExecutor(max_workers=1) — the pattern SileroVADAnalyzer and BaseSmartTurn already use. One worker is a correctness requirement: two would interleave the model's recurrent state.
  • Resampling mirrors RNNoiseFilter — two SOXRStreamAudioResampler instances, QQ default, none created when the transport already runs at the model's native rate. dpdfnet can resample internally but that is librosa.resample with no state across calls, so per-20 ms-chunk it produces boundary artefacts; the filter always passes sample_rate=None and does the conversion itself. A test asserts this so it doesn't get "simplified" later.
  • Everything fails open. Missing install, failed download, bad path, or an exception in inference all log and pass audio through. start() never raises (base_input calls it without a try/except) and filter() never raises (an exception there kills the session's audio task). Empty returns while buffering follow Skip empty audio frames after filter buffering #3828.
  • No ONNX Runtime tuning needed — dpdfnet's create_cpu_session() already pins 1 intra/inter-op thread and CPU EP, the same discipline silero.py applies by hand.

8 kHz and 48 kHz native model variants avoid resampling on telephony and WebRTC transports.

Testing

16 unit tests that patch StreamEnhancer at the filter's import site, so they run whether or not the extra is installed and never touch the network — no skipUnless, no committed ONNX blob. They cover every pass-through path, empty-while-buffering, odd-length input carrying its trailing byte, idempotent stop(), the 8k↔16k round trip preserving a 440 Hz tone, and reset-on-re-enable. Plus an opt-in real-model test gated on an env var. 16 passed, 1 skipped; ruff and uv lock --check clean; the lockfile adds only dpdfnet with no existing pin moved.

One question

Happy to ship this as a standalone pipecat-dpdfnet community package instead — it imports only BaseAudioFilter, the resampler types and the frame classes, so it lifts out unchanged. What made the choice unclear is that COMMUNITY_INTEGRATIONS.md defines categories and naming for STT, LLM, TTS, image, vision and serializers, but not for audio filters. Guidance on where filters belong would be useful, and probably worth adding to that doc either way.

Sai Aakash and others added 2 commits September 8, 2026 14:33
Adds an input audio filter wrapping DPDFNet, a family of causal dual-path
deep filtering models that run on CPU via ONNX Runtime. DPDFNet is
Apache-2.0 and needs no API key or vendor account, so it sits alongside
RNNoise and Koala as a self-hostable option.

Inference runs on a dedicated single-worker ThreadPoolExecutor, following
the pattern already used by SileroVADAnalyzer and BaseSmartTurn, because
filter() is awaited directly on the transport's audio task once per 20 ms
frame. Audio is resampled to and from the model's native rate with two
SOXRStreamAudioResampler instances, mirroring RNNoiseFilter; when the
transport already runs at the model's native rate no resampler is created
at all. dpdfnet's own resampling is deliberately bypassed by passing
sample_rate=None, because it is per-chunk librosa with no history across
chunks.

Every failure path degrades to passing audio through unmodified: a missing
dpdfnet install, a failed model download, and an exception during
inference. start() and filter() never raise, since base_input calls
start() without a try/except and an exception from filter() would kill the
session's audio task.

Select a model with model= to trade quality for CPU. 8 kHz and 48 kHz
native variants avoid resampling on telephony and WebRTC transports.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant