feat(tts): add Fish Audio expressive markup

[rosetta-livekit-bot]

Ports livekit/agents#6631 to add Fish Audio s2 models to expressive mode. Fish now receives native emotion, sound, break, and emphasis brackets, with steering-aware sound/example filtering and clean transcript handling.

Adds the minimal speech-steering infrastructure the JS target lacked, including public steering types and queryable markup capabilities.

Verification

• pnpm vitest run agents/src/tts/expr_markup.test.ts agents/src/tts/markup_utils.test.ts
• pnpm vitest run agents (1284 passed, 2 skipped; 11 pre-existing AMD failures remain, plus one EOT retry timing failure that passed in isolation)
• pnpm vitest run agents/src/inference/eot/transports.test.ts
• pnpm --filter @livekit/agents build
• pnpm build
• pnpm --filter @livekit/agents lint (passes with existing warnings)
• focused ESLint and Prettier checks for all changed files
• cue-cli voice run with fishaudio/s2.1-pro: emitted excited/laughing markers, synthesized Fish audio, and completed without framework errors
• pnpm --filter @livekit/agents api:check attempted; blocked by API Extractor not supporting the repository's existing export * as syntax

Source diff coverage

• Adapted: livekit-agents/livekit/agents/tts/_provider_format.py -> agents/src/tts/_provider_format.ts. Ported the Fish s2 emotion/sound vocabulary, instruction examples, English-only labels, steering filters, capability matrix, aliases, expression intensification, break-duration mapping, emphasis conversion, transcript registration, normalization, and intentional per-sentence chunking behavior. Added the target-missing steering propagation in agents/src/tts/tts.ts, agents/src/voice/agent_session.ts, agents/src/voice/agent_activity.ts, agents/src/voice/presets.ts, and public exports.
• Adapted: tests/test_tokenizer_xml_markup.py -> agents/src/tts/expr_markup.test.ts. Ported all 10 Fish Audio test cases to Vitest without adding source-absent behavioral tests.
• Not applicable: none. Every source file and behavior has a target counterpart and was ported.

Source: livekit/agents#6631

---

Ported from livekit/agents#6631

Original PR description

rebase of #6528 onto main (the original was auto-closed when tina/refactor-expressive-mode merged in #6406).

adds fish audio (s2 models) to expressive mode, same shape as inworld/xai/cartesia:

• emotion tags (discrete vocabulary: regretful, hopeful, happy, excited, curious, surprised, sad, empathetic, sarcastic) -> [very EMOTION]
• breaks -> [break] / [long-break] by duration
• emphasis, the only wrapping prosody -> prefix form: are you [emphasis] sure
• sounds: [laughing], [chuckling], [clear throat], governed by the laughing / reflex_sounds steering fields; disabled sounds are dropped from the advertised vocabulary and examples like the other providers
• disfluent few-shot examples render only while disfluencies is enabled, so examples never contradict the delivery guidelines

no _MAX_INPUT_LEN entry on purpose: fish's markers are sentence-scoped (every sentence carries its own emotion tag), so we stick with normal per-sentence synthesis instead of batching longer expressive chunks — no steering is lost and time-to-first-audio stays low.

adapted to the post-#6406 architecture during the rebase: fish registers in the new _PROVIDER_MARKUP (plain tag-name list), and transcript tests use split_all_markup — bracket cues are handled at the aligned-transcript source by drop_bracket_cues, matching the other providers.

verified live through the inference gateway (fishaudio/s2.1-pro + gemma-4-31b-it) and with offline sweeps across several llms checking tag placement, conversion, transcript stripping, and steering filters. 1624 unit tests + strict mypy green on the rebased stack.

[changeset-bot]

🦋 Changeset detected

Latest commit: de2ab8b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 36 packages

Name	Type
@livekit/agents	Major
@livekit/agents-plugin-anam	Major
@livekit/agents-plugin-anthropic	Major
@livekit/agents-plugin-assemblyai	Major
@livekit/agents-plugin-baseten	Major
@livekit/agents-plugin-bey	Major
@livekit/agents-plugin-cartesia	Major
@livekit/agents-plugin-cerebras	Major
@livekit/agents-plugin-deepgram	Major
@livekit/agents-plugin-did	Major
@livekit/agents-plugin-elevenlabs	Major
@livekit/agents-plugin-fishaudio	Major
@livekit/agents-plugin-google	Major
@livekit/agents-plugin-hedra	Major
@livekit/agents-plugin-hume	Major
@livekit/agents-plugin-inworld	Major
@livekit/agents-plugin-lemonslice	Major
@livekit/agents-plugin-liveavatar	Major
@livekit/agents-plugin-livekit	Major
@livekit/agents-plugin-minimax	Major
@livekit/agents-plugin-mistral	Major
@livekit/agents-plugin-mistralai	Major
@livekit/agents-plugin-neuphonic	Major
@livekit/agents-plugin-openai	Major
@livekit/agents-plugin-perplexity	Major
@livekit/agents-plugin-phonic	Major
@livekit/agents-plugin-resemble	Major
@livekit/agents-plugin-rime	Major
@livekit/agents-plugin-runway	Major
@livekit/agents-plugin-sarvam	Major
@livekit/agents-plugin-silero	Major
@livekit/agents-plugin-soniox	Major
@livekit/agents-plugin-tavus	Major
@livekit/agents-plugins-test	Major
@livekit/agents-plugin-trugen	Major
@livekit/agents-plugin-xai	Major

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[devin-ai-integration]

Devin Review found 1 potential issue.

View 2 additional findings in Devin Review.

[devin-ai-integration]

🟡 Expressive speech markers are turned on for every Fish Audio voice, not only the models that understand them

Expressive markup is enabled for any Fish Audio voice by keying only on the provider prefix (_markupProviderKey() at agents/src/inference/tts.ts:387-395), even though the new markers are only understood by the s2 family, so older Fish voices get emotion/sound/pause markers they cannot interpret.
Impact: A user running an older Fish Audio voice with expressive mode can hear bracketed marker text read aloud or get garbled delivery.

Provider-key derivation vs. the s2-only marker vocabulary

PROVIDER_MARKUP/llmInstructions gain a fishaudio entry (agents/src/tts/_provider_format.ts:983-1001), and the vocabulary is explicitly documented as the "Fish Audio (s2 family) speech markers" (agents/src/tts/_provider_format.ts:120). But InferenceTTS._markupProviderKey() returns the raw provider segment of the model string for anything other than Inworld, which already has exactly this kind of gate (return model.includes('tts-2') ? 'inworld' : ''). Consequently fishaudio/s1, fishaudio/s1-mini, etc. also resolve to 'fishaudio', so _resolveExpressiveOptions() (agents/src/voice/agent_activity.ts:1917) sees markup support, the LLM is instructed to emit <expr .../> markers, and convertMarkup lowers them to [very excited], [laughing], [break], [emphasis] before they are sent to a model that does not implement them.

A model gate mirroring the Inworld one (only s2 models return 'fishaudio') would restore the intended scope.

Prompt for agents

The Fish Audio markup vocabulary added in agents/src/tts/_provider_format.ts is documented as s2-family only (see the FISHAUDIO_* tables and the doc comment "Fish Audio (s2 family) speech markers"), and the PR intent is to add Fish Audio s2 models to expressive mode. However, the provider key used to look up these tables comes from InferenceTTS._markupProviderKey() in agents/src/inference/tts.ts, which simply returns the provider segment of the gateway model string for every provider except Inworld. Inworld already has a model-level gate there (only models containing 'tts-2' return 'inworld', older ones return '' so expressive stays off). No equivalent gate exists for Fish, so any fishaudio/* model (including non-s2 voices) is treated as markup-capable: _resolveExpressiveOptions() in agents/src/voice/agent_activity.ts enables expressive, the LLM is taught the expr dialect, and convertMarkup emits [very excited]/[laughing]/[break]/[emphasis] to a model that may not support them. Add a Fish model gate alongside the Inworld one so only the s2 family maps to the 'fishaudio' markup key, and consider adding the supported Fish model strings to the inference TTS model unions for discoverability.

---

Was this helpful? React with 👍 or 👎 to provide feedback.