feat(voice): infer expressive speech register - #2197
feat(voice): infer expressive speech register#2197rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 6e74107 The changes in this PR will be included in the next version bump. This PR includes changesets to release 36 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| const EXPR_PREAMBLE = `You control speech delivery with a single XML marker tag: <expr/>. Every marker has a type attribute. Use only the marker types listed below, and where a type lists a label vocabulary, only those labels. Use the markers often and diversify them so the voice never sounds flat while ensuring the markers are appropriate for the moment. Write the words themselves the way people talk: use contractions ("I'm", "you're", "don't") — spelled-out forms like "I am" or "do not" sound stiff when spoken. | ||
|
|
||
| You control speech delivery with a single XML marker tag: <expr/>. Every marker has a type attribute. The types below are the ONLY ones this voice supports, and where a type lists a label vocabulary, use only those labels. Reach for the markers often and mix them so the voice never sounds flat — but keep each one motivated by the moment, never decorative.`; | ||
| Just as important is knowing when NOT to reach for a marker. Reserve surprise openers like "oh" or "ah" for genuine surprise — an ordinary request isn't one. Don't stack markers on short replies or decorate every sentence. If a reaction wouldn't happen in a real conversation, skip it — there's always another genuine beat to lean into. | ||
|
|
||
| Match your delivery to the REGISTER of the moment, and reassess every turn. When the moment is professional, high-stakes, or emotionally heavy — bad news, an emergency, real distress — keep delivery composed and restrained. When the moment is casual, playful, or celebratory, let it loosen and brighten. A serious turn in an otherwise casual conversation still gets a composed reply.`; |
There was a problem hiding this comment.
🟡 Spoken agents no longer told to say numbers and abbreviations out loud
The instruction telling the voice to write numbers, symbols, and abbreviations the way they are spoken was dropped (EXPR_PREAMBLE at agents/src/tts/_provider_format.ts:182-186) while the rest of the delivery guidance was rewritten, so amounts like "$42.50" and "Dr." are now handed to the voice unexpanded.
Impact: Expressive replies can read prices, dates, and abbreviations incorrectly or literally for every affected voice provider.
How the preamble rewrite lost the expansion rule
Before this PR the shared preamble began with:
Expand all numbers, symbols, and abbreviations into spoken form (e.g. $42.50 to forty-two dollars and fifty cents, Dr. to Doctor).
The new preamble replaces that text with the marker/knowing-when-not-to/register paragraphs and never restates the expansion rule. EXPR_PREAMBLE is the single shared header used by every provider block (agents/src/tts/_provider_format.ts:188, :214, :235, and the Fish builder at agents/src/tts/_provider_format.ts:316), so all expressive providers lose it at once. A repo-wide search for "Expand all numbers" / "spoken form" finds no other place that supplies this guidance, and the text transforms (filter_markdown, filter_emoji) do not perform number expansion.
Prompt for agents
The rewritten EXPR_PREAMBLE in agents/src/tts/_provider_format.ts dropped the pre-existing first line "Expand all numbers, symbols, and abbreviations into spoken form (e.g. $42.50 to forty-two dollars and fifty cents, Dr. to Doctor)." This preamble is shared by the Cartesia, Inworld, xAI, and Fish Audio instruction blocks, so every expressive provider now loses the number/abbreviation expansion guidance and the LLM may emit raw "$42.50" / "Dr." to the TTS. If the removal was not intentional in the ported upstream change, restore that sentence (or an equivalent) to the preamble; if it was intentional, confirm the expansion guidance is provided elsewhere in the injected prompt.
Was this helpful? React with 👍 or 👎 to provide feedback.
Ports livekit/agents#6632 to replace expressive presets with provider-neutral register inference.
Summary
Source diff coverage
livekit-agents/livekit/agents/tts/_provider_format.pyagents/src/tts/_provider_format.ts: ported the shared register rule, conditional Fish guidance, boolean and sparse opt-out semantics, and the generic sound/prosody governance infrastructure required for Inworld and xAI parity. Removed the JS-only provider preset bodies.livekit-agents/livekit/agents/voice/__init__.pyagents/src/voice/index.ts: removed thepresetsnamespace export.livekit-agents/livekit/agents/voice/agent_session.pyagents/src/voice/agent_session.tsandagents/src/voice/agent_activity.ts: removed preset resolution, documentedboolean | NonverbalOptions, and resolve sparse steering directly over defaults.livekit-agents/livekit/agents/voice/presets.pyagents/src/voice/presets.ts; removed its JS-only provider bodies and migrated all target example consumers toexpressive: true.tests/test_expressive_toggle.pytests/test_tokenizer_xml_markup.pyagents/src/tts/expr_markup.test.ts: ported every changed and added steering/register assertion to Vitest.Validation
pnpm buildpnpm --filter @livekit/agents typecheckpnpm --filter @livekit/agents lintpnpm --filter livekit-agents-examples lintpnpm vitest run agents/src/tts/expr_markup.test.ts agents/src/tts/markup_utils.test.ts(69 passed)cue-clivoice-mode run againstexpressive-agent-js: serious-news turn resolved without framework errors and selected calm, gentle, slow Inworld deliveryFull-suite baseline issues encountered:
pnpm vitest run agents: 1,290 passed, 11 unrelated failures inagents/src/voice/amd.test.ts; AMD source/tests are unchanged from the base branch.pnpm --filter livekit-agents-examples test: 96 passed, 1 unrelated AgentTask assertion failure plus two existing unhandled FakeLLM errors; affected files are unchanged from the base branch.pnpm --filter @livekit/agents api:check: blocked by the branch's existing API Extractor limitation onexport * assyntax in generateddist/index.d.ts.Source: livekit/agents#6632
Ported from livekit/agents#6632
Original PR description
rebase of #6571 onto main (auto-closed when #6406 merged). stacked on #6631 — the first two commits are that PR; review the last three.
our evals (~2,000 generations, 5 personas × serious/neutral/casual moments, 4 llms) suggest the agent persona and conversational context choose the appropriate register on their own, so this replaces the formal/casual presets with:
nonverbal_soundsaccepts a bool, and dicts are sparse opt-outs:{"laughing": False}removes laughter and nothing elsemeasured at the default (everything on): 0.00 laughs/turn at serious moments across all four llms, 0.67–0.83 at casual ones, judge fit 4.59/5 overall.
{"nonverbal_sounds": False, "disfluencies": False}produced 0 sounds and 1 filler across 360 generations. follow-up testing with production-size prompts (the real 35K-char hotel receptionist prompt + its 14 tools) showed no degradation: tag compliance holds, zero markup in tool arguments over 192 generations.one semantic change to sign off on:
NonverbalOptions()(empty dict) now means "no opt-outs" — everything stays on;nonverbal_sounds=Falseis the all-off form.full findings + reproducible eval harness shared separately.