Add Moonshine streaming ASR model support#2302
Open
nenad1002 wants to merge 24 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds support for a new streaming encoder-decoder ASR model family (“Moonshine”) to ONNX Runtime GenAI, integrating it into the existing streaming ASR pipeline (StreamingProcessor + Generator/TransducerState) alongside the existing Nemotron streaming transducer path.
Changes:
- Introduces
MoonshineStreamingModel+MoonshineStreamingStateimplementing a 5-session streaming AED pipeline with incremental cross-KV caching and chunk-wise commit logic. - Extends
StreamingProcessorwith a stateless per-chunk VAD verdict (IsChunkSilent) and adds a factory dispatch to select Nemotron vs Moonshine streaming processors bymodel.type. - Updates C++ and Python tests/fixtures to exercise multiple streaming ASR model directories and add Moonshine-specific segmentation coverage.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| test/python/test_onnxruntime_genai_api.py | Generalizes streaming ASR Python tests to run across multiple streaming ASR models and sets per-model expected type / WER thresholds. |
| test/python/conftest.py | Replaces single-model Nemotron fixture with a parametrized streaming-ASR model-path fixture. |
| test/c_api_tests.cpp | Parameterizes existing streaming ASR C API tests across multiple models; adds Moonshine VAD/segmentation end-to-end test helpers. |
| src/models/streaming_processor.h | Adds IsChunkSilent API for per-chunk VAD verdict and declares the streaming-processor factory. |
| src/models/streaming_processor.cpp | Implements IsChunkSilent and moves/expands factory dispatch to select Nemotron vs Moonshine processors by model type. |
| src/models/nemotron_streaming_processor.cpp | Removes the old CreateStreamingProcessor definition (now centralized in streaming_processor.cpp). |
| src/models/moonshine_streaming.h | Adds Moonshine streaming model/state declarations and detailed pipeline/state documentation. |
| src/models/moonshine_streaming.cpp | Implements the Moonshine streaming pipeline, including accumulation, incremental cross-KV caching, and commit logic. |
| src/models/moonshine_streaming_processor.h | Adds Moonshine-specific StreamingProcessor implementation header. |
| src/models/moonshine_streaming_processor.cpp | Implements Moonshine “thin processor” that emits {audio_chunk,is_silent,is_final} per chunk. |
| src/models/model.cpp | Registers Moonshine streaming model creation when model.type is streaming_enc_dec_asr. |
| src/models/model_type.h | Adds IsStreamingEncDecASR() model-type predicate. |
| src/generators.cpp | Routes streaming enc-dec ASR models through the TransducerState path (skip search/logits validations). |
| src/config.h | Extends config schema with overlap_samples and a model.moonshine section. |
| src/config.cpp | Parses overlap_samples and the moonshine config section; exempts streaming enc-dec ASR from context_length requirement. |
| examples/python/streaming_asr.py | Updates example docstring/notes to cover both Nemotron and Moonshine streaming ASR models. |
Comment on lines
+1696
to
+1704
| // Value-parameterized fixture so each StreamingASR test runs once per model. | ||
| // The fixture is named CAPITests so the tests keep the original | ||
| // CAPITests.StreamingASR* naming, e.g. | ||
| // "StreamingASR/CAPITests.StreamingASRCreate/nemotron_speech_streaming". | ||
| class CAPITests : public ::testing::TestWithParam<StreamingASRModel> { | ||
| protected: | ||
| std::string ModelPath() const { return std::string(MODEL_PATH) + GetParam().subdir; } | ||
| size_t ChunkSamples() const { return GetParam().chunk_samples; } | ||
| }; |
Comment on lines
+1890
to
+1893
| // Checking whether or not the model has the components necessary to run VAD. If not, skip the test. | ||
| // VAD requires having a "vad" section in the genai_config.json and the silero_vad.onnx file in the model directory. | ||
| // Even if a model is VAD-capable, VAD is disabled by default in InitializeVadFromConfig. | ||
| const bool model_has_vad = ModelHasVad(model_path); |
Comment on lines
+8
to
+10
| // 3. emits one NamedTensors per chunk with: | ||
| // "audio_chunk" : float32 [1, num_samplyoes] raw audio, | ||
| // "is_silent" : int64 [1] (1 iff VAD flagged this chunk silent), |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new moonshine_streaming model type: a 5-session ONNX pipeline (frontend / encoder / adapter / cross-KV / decoder-KV) driving an AED decoder with bidirectional cross-attention over accumulating memory. The main advantage of this model is its speed and relatively low CPU utilization, making it suitable for devices where Nemotron would require too much compute or memory. Both HF small and tiny version will be exported and run with onnxruntime-genai.
Highlights:
New model MoonshineStreamingModel and state (src/models/moonshine_streaming.{h,cpp}) built on the shared TransducerState interface, so it plugs into the existing streaming pump alongside Nemotron/Parakeet.
Chunked streaming decode with lookahead hold-back, LCP-based commit across chunks, per-chunk token cap, and BOS re-decode. AR loop reuses a pre-allocated [1,1] token tensor, and the committed prefix is teacher-forced in a single parallel decoder call.
Incremental cross-KV cache - new memory frames are projected once and concatenated into the cached k_cross / v_cross instead of recomputing from scratch.
VAD-aware segmentation - hard segment cap, min-segment gating, and mid-segment silence handling.
Python example: renames nemotron_speech.py to streaming_asr.py so one entry point covers all streaming ASR models.
Tests: extends c_api_tests.cpp and updates the Python API tests / conftest for the new model type.