Add livekit-wakeword crate with ONNX-based wake word detection by pham-tuan-binh · Pull Request #926 · livekit/rust-sdks

pham-tuan-binh · 2026-03-03T15:15:02Z

Summary

New livekit-wakeword crate with a stateless wake word detection pipeline
Pipeline: raw PCM audio → mel spectrogram → speech embeddings → classifier scores
Mel spectrogram and embedding models are bundled at compile time via include_bytes!
Wake word classifier models (e.g. hey_livekit.onnx) are loaded dynamically from disk at runtime
Supports multiple classifiers simultaneously, each returning a confidence score (0-1)

Test plan

cargo test -p livekit-wakeword — single end-to-end test exercises full pipeline with hey_livekit.onnx
Validates score output is in [0.0, 1.0] range
Validates too-short audio returns zero scores

ladvoc · 2026-03-03T20:31:39Z

The ONNX files should be stored in Git LFS:

git lfs track "*.onnx"

ladvoc

I have left some initial comments. Overall, looking good so far.

1egoman

Generally looks good by eye to me!

1egoman · 2026-03-04T20:56:35Z

+    fn resample_to_16k(&mut self, samples: &[i16]) -> Result<Vec<f32>, WakeWordError> {
+        let rs = self.resampler.as_mut().unwrap();
+        let chunk_in = rs.fft.chunk_size_input();
+        let chunk_out = rs.fft.chunk_size_output();
+
+        // Expected output length based on ratio
+        let expected_len = (samples.len() as f64 * 16000.0 / rs.input_rate as f64).round() as usize;
+
+        let mut output = Vec::with_capacity(expected_len);
+
+        let mut pos = 0;
+        while pos < samples.len() {
+            let remaining = samples.len() - pos;
+            let n = remaining.min(chunk_in);
+
+            // Fill input buffer, zero-pad if last chunk is short
+            for (i, v) in rs.input_buf.iter_mut().enumerate() {
+                *v = if i < n { samples[pos + i] as f32 / 32768.0 } else { 0.0 };
+            }
+
+            rs.fft.resample(&rs.input_buf.clone(), &mut rs.output_buf)?;
+
+            let take = if remaining < chunk_in {
+                // Last (partial) chunk: scale output proportionally
+                (n as f64 * chunk_out as f64 / chunk_in as f64).round() as usize
+            } else {
+                chunk_out
+            };
+
+            output.extend_from_slice(&rs.output_buf[..take]);
+
+            pos += chunk_in;
+        }
+
+        output.truncate(expected_len);
+        Ok(output)
+    }


thought: Is it worth using any of the pre-existing webrtc resampling functions like this?

Mostly asking because I'm guessing this needs to be relatively performant if it is going to be running on edge devices for wake word detection and the pre-existing webrtc algorithms might be taking advantage of some platform specific optimizations which could result in a nice speedup.

we defin can

i used resampler from @ladvoc's suggestion #926 (comment)

i don't have an opinion on which one to use, can run a simple benchmark for perf

Ah, I didn't see @ladvoc's comment and it largely is the same thing, ok cool makes sense! I hadn't considered the portability argument / maybe not wanting to link to libwebrtc.

Something that came to mind (maybe a bad idea / overengineering though) - make a WakeWordResampler trait, make a default resampler implementation using your resampler code, and put a generic in the WakeWordModel type so it effectively becomes WakeWordModel<impl WakeWordResampler>. fn new() could then have a regular version with just a model param which returns WakeWordModel<NoopResampler> and also maybe like a fn new_with_sample_rate(model, sample_rate) version which returns WakeWordModel<DefaultResampler>. That leaves the door open for other resampling approaches with different properties (ie, NoopResampler, LibWebrtcResampler, etc) and the ability to use typestates to change WakeWordModel method signatures down the line which could be interesting.

Hmm, i think that might be a bit overkill for now, but it does make resampling much more extensible.

Tbh, I don't think we'll change the resampling strategy for wake word in the future. Most of the times the user will go from higher sample rate to 16khz so I think all resampling method will be quite the same in terms of quality. The model is very robust as well.

Actually, I think I will change the resampling from FFT based to FIR based so it's much faster.

ladvoc · 2026-03-05T16:20:22Z

@pham-tuan-binh, the build issue on Windows ARM64 is related to the tract-linalg crate, a dependency of ort-tract, not properly supporting SIMD for that platform. Will need to see if there's a workaround.

ladvoc

LGTM ✅

Introduces the livekit-wakeword crate with a MelspectrogramModel that extracts mel-scaled spectrogram features from raw i16 PCM audio using ONNX Runtime. Includes tests verifying output shape and time-frame scaling.

- Move ONNX models from models/ to onnx/ to avoid naming collision - Flatten src/models/ into top-level src/ modules - Colocate unit tests with their respective modules - Remove prelude.rs in favor of direct imports

- Add WakeWordModel: stateless detection pipeline (mel -> embeddings -> classifier) - Classifiers loaded dynamically from ONNX files on disk - Hoist constants (SAMPLE_RATE, MEL_BINS, EMBEDDING_WINDOW, etc.) to lib.rs - Extract shared session builder helpers to eliminate duplication - Consolidate tests into a single end-to-end predict test - Add hey_livekit.onnx classifier model

Replace the default ONNX Runtime C++ backend with the pure-Rust tract backend via ort-tract to avoid upstream test failures from native library linking issues.

Accept plain slices (&[i16], &[f32]) instead of ndarray types in the public detect/predict methods, keeping ndarray as an internal detail.

Add a typed WakeWordError enum (via thiserror) covering Ort, Shape, Io, and ModelNotFound cases, replacing dynamic error types across all public methods.

Accept any common sample rate (16k–384k Hz) via a new sample_rate parameter on WakeWordModel::new(), resampling to 16 kHz internally using the resampler crate. Make embedding and melspectrogram modules pub(crate) since they are implementation details. Change MelspectrogramModel::detect() to accept &[f32] directly, avoiding a redundant i16→f32→i16→f32 round-trip when resampling.

Switch from ResamplerFft to ResamplerFir which accepts arbitrary input buffer sizes, eliminating chunk management and zero-padding complexity. Uses 64-sample latency (~1.3ms at 48kHz) with 90dB stopband attenuation.

tract-linalg has ARM64 assembly (.S files) that MSVC cannot compile. Use a build.rs to auto-detect the target and conditionally enable ort-tract, falling back to native ONNX Runtime on aarch64-pc-windows-msvc.

The Rust mel spectrogram was missing the x/10 + 2 post-processing normalization from the openWakeWord pipeline, causing near-zero classifier scores. Added positive/negative WAV sample tests with a 0.5 threshold to catch regressions.

…word

pham-tuan-binh · 2026-03-10T13:17:29Z

added a new test which sanity check if hey livekit model can correctly detect a positive and negative wakeword sample

Accept main's versioned_files format and simplified livekit-ffi scopes, re-add livekit-wakeword package with the new format.

pham-tuan-binh requested a review from ladvoc March 3, 2026 15:23

lukasIO reviewed Mar 3, 2026

View reviewed changes

Comment thread .changeset/add_livekit_wakeword_crate_with_onnx_based_wake_word_detection.md Outdated

ladvoc reviewed Mar 3, 2026

View reviewed changes

ladvoc requested a review from 1egoman March 3, 2026 21:11

1egoman reviewed Mar 4, 2026

View reviewed changes

ladvoc approved these changes Mar 10, 2026

View reviewed changes

pham-tuan-binh and others added 18 commits March 10, 2026 08:28

Add melspectrogram ONNX model with feature extraction and tests

1518e8a

Introduces the livekit-wakeword crate with a MelspectrogramModel that extracts mel-scaled spectrogram features from raw i16 PCM audio using ONNX Runtime. Includes tests verifying output shape and time-frame scaling.

Add embedding ONNX model with inference and tests

dc73a76

Reorganize wakeword crate: flatten modules and separate ONNX files

c9f8128

- Move ONNX models from models/ to onnx/ to avoid naming collision - Flatten src/models/ into top-level src/ modules - Colocate unit tests with their respective modules - Remove prelude.rs in favor of direct imports

Apply cargo fmt formatting

66ab435

Auto generate changeset

65f0ed1

Add Apache 2.0 license headers to wakeword source files

fe604ad

Add README for livekit-wakeword crate

7bbf86e

Scope changeset to livekit-wakeword and add package to knope.toml

a8248d7

Switch wakeword ONNX backend from ort default to tract

b584110

Replace the default ONNX Runtime C++ backend with the pure-Rust tract backend via ort-tract to avoid upstream test failures from native library linking issues.

Fix minimum audio duration in wakeword README (~2s, not ~4s)

c8b1e8a

Remove ndarray from public API signatures in livekit-wakeword

eb537e9

Accept plain slices (&[i16], &[f32]) instead of ndarray types in the public detect/predict methods, keeping ndarray as an internal detail.

Replace Box<dyn Error> with WakeWordError enum in public API

902d8c4

Add a typed WakeWordError enum (via thiserror) covering Ort, Shape, Io, and ModelNotFound cases, replacing dynamic error types across all public methods.

Move predict test to integration tests and run cargo fmt

7259832

Rename integration test to integration.rs and add license header

04ca175

Exclude tract backend on aarch64-pc-windows-msvc to fix MSVC build

cead83c

tract-linalg has ARM64 assembly (.S files) that MSVC cannot compile. Use a build.rs to auto-detect the target and conditionally enable ort-tract, falling back to native ONNX Runtime on aarch64-pc-windows-msvc.

pham-tuan-binh force-pushed the binhpham/livekit-wakeword branch from bedca64 to cead83c Compare March 10, 2026 01:29

pham-tuan-binh added 3 commits March 10, 2026 20:13

Merge remote-tracking branch 'origin/main' into binhpham/livekit-wake…

6fcf442

…word

Update changeset to reflect full PR scope

5ffb63c

Run cargo fmt on wakeword tests

341ffce

Merge origin/main, resolve knope.toml conflict

a37eb1c

Accept main's versioned_files format and simplified livekit-ffi scopes, re-add livekit-wakeword package with the new format.

pham-tuan-binh merged commit 2a87895 into main Mar 11, 2026
22 checks passed

pham-tuan-binh deleted the binhpham/livekit-wakeword branch March 11, 2026 10:31

knope-bot Bot mentioned this pull request Mar 11, 2026

Release packages #932

Merged

6 tasks

Uh oh!

Conversation

pham-tuan-binh commented Mar 3, 2026

Summary

Test plan

Uh oh!

Uh oh!

ladvoc commented Mar 3, 2026

Uh oh!

ladvoc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

1egoman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

1egoman Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

pham-tuan-binh Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

1egoman Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pham-tuan-binh Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

ladvoc commented Mar 5, 2026

Uh oh!

ladvoc left a comment

Choose a reason for hiding this comment

Uh oh!

pham-tuan-binh commented Mar 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

1egoman Mar 5, 2026 •

edited

Loading