Add VuiTTSService: local streaming TTS with Vui Nano (CUDA / Apple Silicon) - #5709
Open
mogwai wants to merge 4 commits into
Open
Add VuiTTSService: local streaming TTS with Vui Nano (CUDA / Apple Silicon)#5709mogwai wants to merge 4 commits into
mogwai wants to merge 4 commits into
Conversation
…licon) Vui Nano is a small, context-aware text-to-speech model trained on real conversations (219M active / 305M total parameters, Apache 2.0). The service runs the model in-process through the `vui-tts` package — on an NVIDIA GPU, or on MLX on Apple Silicon — and streams 24 kHz audio into the pipeline as each frame is decoded. Weights and the shipped voice prompts download from Hugging Face on first use. - `pipecat-ai[vui]` extra (Python 3.12 — the only version vui-tts ships for). - Voice is a shipped name (maeve / abraham / rhian / harry), a prompt .safetensors baked with Vui's scripts/build_prompts.py, or a .wav to clone; changeable at runtime via TTSUpdateSettingsFrame. - All engine work (load, prefill, decode, rewind) runs on one worker thread; interruption sets a cancel event the decode loop checks at every frame. - Tests mock the engine; examples for voice and update-settings; docs mock list updated. Verified on an M4 (MLX): a TTSSpeakFrame through the service renders 6.6s of audio in 82 TTSAudioRawFrames, 0.0% WER by moonshine ASR.
….1.3 vui-tts 1.1.3 makes CUDA Row.rewind() re-seed the codec context from the voice prompt, so a turn no longer decodes against the previous turn's tail. A .wav voice without a transcript now raises a clear error (sibling .txt, a baked prompt .safetensors, or `vui-tts[server]` for transcription) instead of failing on the missing openai-whisper import. Lock regenerated.
Over repeated renders the engine default of 0.9 drops or swaps the odd word (5-rep check on MLX and CUDA); 0.7 — what the LiveKit plugin and the published evals use — is clean. gen_config= still overrides.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
VuiTTSService— local, streaming text-to-speech with Vui Nano, a small, context-aware model trained on real conversations (219M active / 305M total parameters, Apache 2.0). It runs in-process through thevui-ttspackage: on an NVIDIA GPU, or on MLX on Apple Silicon. No API key; weights and the shipped voice prompts download from Hugging Face on first use. Audio streams into the pipeline frame by frame as it is decoded.Modelled on
PocketTTSService(local model,_stream_audio_frames_from_iterator, settings delta for the voice). New extrapipecat-ai[vui]— markedpython_version == '3.12', the only versionvui-ttsships for, so other splits lock unchanged.Design
maeve/abraham/rhian/harry), a prompt.safetensorsbaked with Vui'sscripts/build_prompts.py, or a.wavto clone (sibling.txttranscript, else transcribed). Runtime change viaTTSUpdateSettingsFramere-prefills on the next utterance.Verified
tests/test_vui_tts.py(mocked engine: frame ordering, int16 conversion, one prefill per voice, rewind per turn, voice update) — 2 passed.TTSSpeakFramethrough the service → 6.6 s of audio in 82TTSAudioRawFrames, 0.0% WER by moonshine ASR.ruff check/ruff formatclean;uv lockregenerated (addsvui-tts1.1.2 for the 3.12 split only).Examples:
examples/voice/voice-vui.py,examples/update-settings/tts/tts-vui.py. I'm the author of the model (Fluxions AI) and will maintain the service.Update — CUDA verified too (RTX 5090, torch 2.11, plain pip install without flash-attn): the mocked tests pass and a
TTSSpeakFramethrough the service renders 5.9 s of audio in 74TTSAudioRawFrames at 0.0% WER by moonshine. Now requiresvui-tts>=1.1.3, which fixes a CUDA multi-turn codec-context issue found in review of the LiveKit sibling plugin.Scenario verification (exact PR head, MLX on M4 and CUDA on a 5090): four utterances on one row with a
TTSUpdateSettingsFramevoice switch mid-way (abraham → maeve), then a realInterruptionFramemid-utterance followed by a clean utterance — all frames delivered, noErrorFrames, every utterance 0.0% WER by moonshine on CUDA (MLX: single-word sampling slips only). A 5-rep interrupted-vs-clean comparison showed no interruption-specific degradation; it did show the engine default temperature 0.9 slips more than 0.7, so the service now defaults to 0.7 (matching the LiveKit plugin and the published evals).