Add EasyMagpie vLLM-Omni serving

[vklimkov-nvidia]

Summary

• Adds EasyMagpieTTS streaming inference on vLLM-Omni for NemotronTTS, using a Nemotron-H autoregressive talker with a per-codebook local transformer over a 25 fps spectral codec.
• Implements two registered deployment topologies:
  • a talker-only pipeline for acoustic-token generation
  • a two-stage pipeline that connects the talker to a native stateful codec and produces 22.05 kHz waveform chunks
• Supports offline synthesis, OpenAI-compatible whole-text HTTP serving through POST /v1/audio/speech, and incremental text/audio streaming through WS /v1/audio/speech/stream.
• Includes checkpoint conversion, vLLM plugin registration, deployment configs, benchmark tools, notebooks, and focused unit tests.

Why

EasyMagpie generates stacked acoustic codes autoregressively and then decodes them into audio. Running both stages in vLLM-Omni provides batched, high-throughput inference while retaining progressive audio output for streaming and voice-agent use cases.

The conversion tool turns the training-time EasyMagpie and causal codec .nemo checkpoints into a self-contained model directory. It converts the model weights, precomputes the text-embedding lookup, bundles the native codec, saves the tokenizer, and optionally stores speaker embeddings. After conversion, serving requires vLLM/vLLM-Omni and this plugin package, but does not require NeMo at runtime.

The native codec keeps its causal history in vLLM-managed state pages, allowing each invocation to process only newly generated acoustic frames and emit asynchronous PCM chunks without replaying the full context.

Validation

• Repository isort and Black checks pass.
• pytest -q examples/tts/easymagpie_vllm_omni/tests: 70 passed, 1 skipped.
• A real EasyMagpie talker and 25 fps codec conversion completed successfully on an RTX A6000.
• Generated artifacts were validated for a consistent safetensors index, tokenizer and architecture metadata, with 956 talker tensors and 76 codec tensors.
• Conversion imports for add_special_tokens, NemotronHConfig, ModelLoadConfig, and load_easy_magpie_model resolve from the Speech checkout.
• The 128-request RTX A6000 reference at concurrency 32 measured:
  • model whole-text input: 67.75x RTF
  • HTTP whole-text service: 48.43x RTF
  • WebSocket incremental service: 44.12x RTF
  • zero playback underruns in both service modes

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.