feat(audio): add DPDFNetFilter for real-time speech enhancement - #5667
Open
Aakash326 wants to merge 2 commits into
Open
feat(audio): add DPDFNetFilter for real-time speech enhancement#5667Aakash326 wants to merge 2 commits into
Aakash326 wants to merge 2 commits into
Conversation
Adds an input audio filter wrapping DPDFNet, a family of causal dual-path deep filtering models that run on CPU via ONNX Runtime. DPDFNet is Apache-2.0 and needs no API key or vendor account, so it sits alongside RNNoise and Koala as a self-hostable option. Inference runs on a dedicated single-worker ThreadPoolExecutor, following the pattern already used by SileroVADAnalyzer and BaseSmartTurn, because filter() is awaited directly on the transport's audio task once per 20 ms frame. Audio is resampled to and from the model's native rate with two SOXRStreamAudioResampler instances, mirroring RNNoiseFilter; when the transport already runs at the model's native rate no resampler is created at all. dpdfnet's own resampling is deliberately bypassed by passing sample_rate=None, because it is per-chunk librosa with no history across chunks. Every failure path degrades to passing audio through unmodified: a missing dpdfnet install, a failed model download, and an exception during inference. start() and filter() never raise, since base_input calls start() without a try/except and an exception from filter() would kill the session's audio task. Select a model with model= to trade quality for CPU. 8 kHz and 48 kHz native variants avoid resampling on telephony and WebRTC transports. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Aakash326
force-pushed
the
feat/dpdfnet-filter
branch
from
September 8, 2026 09:07
4a53e85 to
65454c7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please describe the changes in your PR. If it is addressing an issue, please reference that as well.
Adds
DPDFNetFilter, an input audio filter wrapping DPDFNet (Ceva, Apache-2.0) — causal dual-path deep filtering models that run on CPU via ONNX Runtime. Like RNNoise and Koala, no API key or vendor account. Newdpdfnetextra; no existing files change exceptpyproject.tomlanduv.lock.Partly addresses #3266.
Why
RNNoise is currently the only free, self-hostable denoiser in Pipecat. We benchmarked it against four alternatives on 40 two-speaker Hindi telephony clips (65.4 min, degraded one axis at a time, all arms scored on identical files). It came last on every quality axis we measured.
31 degraded clips:
baselineOn the crosstalk subset (n=19) RNNoise collapses to AUC 0.652 / STOI Δ −0.344, against DPDFNet-2 at 0.857 / −0.043. On clean audio, DPDFNet attenuates speech 0.5 dB where RNNoise takes 3.1 dB.
Caveats: RNNoise ran at
VHQ, not theQQdefault, so this is the model and not a resampling artefact. DPDFNet's 40 ms algorithmic delay is double RNNoise's and sits upstream of VAD, so it adds to end-of-turn latency.dpdfnet2costs ~2.6× RNNoise's CPU — hencemodel=, withbaselinelanding within 0.02 AUC at roughly a third the cost. The corpus is Hindi telephony and deliberately biased toward hard clips, so read it as a comparison between arms, not as absolute scores.Design
ThreadPoolExecutor(max_workers=1)— the patternSileroVADAnalyzerandBaseSmartTurnalready use. One worker is a correctness requirement: two would interleave the model's recurrent state.RNNoiseFilter— twoSOXRStreamAudioResamplerinstances,QQdefault, none created when the transport already runs at the model's native rate. dpdfnet can resample internally but that islibrosa.resamplewith no state across calls, so per-20 ms-chunk it produces boundary artefacts; the filter always passessample_rate=Noneand does the conversion itself. A test asserts this so it doesn't get "simplified" later.start()never raises (base_inputcalls it without atry/except) andfilter()never raises (an exception there kills the session's audio task). Empty returns while buffering follow Skip empty audio frames after filter buffering #3828.create_cpu_session()already pins 1 intra/inter-op thread and CPU EP, the same disciplinesilero.pyapplies by hand.8 kHz and 48 kHz native model variants avoid resampling on telephony and WebRTC transports.
Testing
16 unit tests that patch
StreamEnhancerat the filter's import site, so they run whether or not the extra is installed and never touch the network — noskipUnless, no committed ONNX blob. They cover every pass-through path, empty-while-buffering, odd-length input carrying its trailing byte, idempotentstop(), the 8k↔16k round trip preserving a 440 Hz tone, and reset-on-re-enable. Plus an opt-in real-model test gated on an env var.16 passed, 1 skipped; ruff anduv lock --checkclean; the lockfile adds onlydpdfnetwith no existing pin moved.One question
Happy to ship this as a standalone
pipecat-dpdfnetcommunity package instead — it imports onlyBaseAudioFilter, the resampler types and the frame classes, so it lifts out unchanged. What made the choice unclear is thatCOMMUNITY_INTEGRATIONS.mddefines categories and naming for STT, LLM, TTS, image, vision and serializers, but not for audio filters. Guidance on where filters belong would be useful, and probably worth adding to that doc either way.