Skip to content

Add Moonshine streaming ASR model support#2302

Open
nenad1002 wants to merge 24 commits into
mainfrom
nebanfic/moonshine-pr
Open

Add Moonshine streaming ASR model support#2302
nenad1002 wants to merge 24 commits into
mainfrom
nebanfic/moonshine-pr

Conversation

@nenad1002

Copy link
Copy Markdown
Contributor

Adds a new moonshine_streaming model type: a 5-session ONNX pipeline (frontend / encoder / adapter / cross-KV / decoder-KV) driving an AED decoder with bidirectional cross-attention over accumulating memory. The main advantage of this model is its speed and relatively low CPU utilization, making it suitable for devices where Nemotron would require too much compute or memory. Both HF small and tiny version will be exported and run with onnxruntime-genai.

Highlights:

New model MoonshineStreamingModel and state (src/models/moonshine_streaming.{h,cpp}) built on the shared TransducerState interface, so it plugs into the existing streaming pump alongside Nemotron/Parakeet.

Chunked streaming decode with lookahead hold-back, LCP-based commit across chunks, per-chunk token cap, and BOS re-decode. AR loop reuses a pre-allocated [1,1] token tensor, and the committed prefix is teacher-forced in a single parallel decoder call.

Incremental cross-KV cache - new memory frames are projected once and concatenated into the cached k_cross / v_cross instead of recomputing from scratch.

VAD-aware segmentation - hard segment cap, min-segment gating, and mid-segment silence handling.

Python example: renames nemotron_speech.py to streaming_asr.py so one entry point covers all streaming ASR models.
Tests: extends c_api_tests.cpp and updates the Python API tests / conftest for the new model type.

Copilot AI review requested due to automatic review settings July 17, 2026 02:56
@nenad1002
nenad1002 requested a review from a team as a code owner July 17, 2026 02:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds support for a new streaming encoder-decoder ASR model family (“Moonshine”) to ONNX Runtime GenAI, integrating it into the existing streaming ASR pipeline (StreamingProcessor + Generator/TransducerState) alongside the existing Nemotron streaming transducer path.

Changes:

  • Introduces MoonshineStreamingModel + MoonshineStreamingState implementing a 5-session streaming AED pipeline with incremental cross-KV caching and chunk-wise commit logic.
  • Extends StreamingProcessor with a stateless per-chunk VAD verdict (IsChunkSilent) and adds a factory dispatch to select Nemotron vs Moonshine streaming processors by model.type.
  • Updates C++ and Python tests/fixtures to exercise multiple streaming ASR model directories and add Moonshine-specific segmentation coverage.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
test/python/test_onnxruntime_genai_api.py Generalizes streaming ASR Python tests to run across multiple streaming ASR models and sets per-model expected type / WER thresholds.
test/python/conftest.py Replaces single-model Nemotron fixture with a parametrized streaming-ASR model-path fixture.
test/c_api_tests.cpp Parameterizes existing streaming ASR C API tests across multiple models; adds Moonshine VAD/segmentation end-to-end test helpers.
src/models/streaming_processor.h Adds IsChunkSilent API for per-chunk VAD verdict and declares the streaming-processor factory.
src/models/streaming_processor.cpp Implements IsChunkSilent and moves/expands factory dispatch to select Nemotron vs Moonshine processors by model type.
src/models/nemotron_streaming_processor.cpp Removes the old CreateStreamingProcessor definition (now centralized in streaming_processor.cpp).
src/models/moonshine_streaming.h Adds Moonshine streaming model/state declarations and detailed pipeline/state documentation.
src/models/moonshine_streaming.cpp Implements the Moonshine streaming pipeline, including accumulation, incremental cross-KV caching, and commit logic.
src/models/moonshine_streaming_processor.h Adds Moonshine-specific StreamingProcessor implementation header.
src/models/moonshine_streaming_processor.cpp Implements Moonshine “thin processor” that emits {audio_chunk,is_silent,is_final} per chunk.
src/models/model.cpp Registers Moonshine streaming model creation when model.type is streaming_enc_dec_asr.
src/models/model_type.h Adds IsStreamingEncDecASR() model-type predicate.
src/generators.cpp Routes streaming enc-dec ASR models through the TransducerState path (skip search/logits validations).
src/config.h Extends config schema with overlap_samples and a model.moonshine section.
src/config.cpp Parses overlap_samples and the moonshine config section; exempts streaming enc-dec ASR from context_length requirement.
examples/python/streaming_asr.py Updates example docstring/notes to cover both Nemotron and Moonshine streaming ASR models.

Comment thread test/c_api_tests.cpp
Comment on lines +1696 to +1704
// Value-parameterized fixture so each StreamingASR test runs once per model.
// The fixture is named CAPITests so the tests keep the original
// CAPITests.StreamingASR* naming, e.g.
// "StreamingASR/CAPITests.StreamingASRCreate/nemotron_speech_streaming".
class CAPITests : public ::testing::TestWithParam<StreamingASRModel> {
protected:
std::string ModelPath() const { return std::string(MODEL_PATH) + GetParam().subdir; }
size_t ChunkSamples() const { return GetParam().chunk_samples; }
};
Comment thread test/c_api_tests.cpp
Comment on lines +1890 to +1893
// Checking whether or not the model has the components necessary to run VAD. If not, skip the test.
// VAD requires having a "vad" section in the genai_config.json and the silero_vad.onnx file in the model directory.
// Even if a model is VAD-capable, VAD is disabled by default in InitializeVadFromConfig.
const bool model_has_vad = ModelHasVad(model_path);
Comment on lines +8 to +10
// 3. emits one NamedTensors per chunk with:
// "audio_chunk" : float32 [1, num_samplyoes] raw audio,
// "is_silent" : int64 [1] (1 iff VAD flagged this chunk silent),
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants