Merge pull request #5171 from pipecat-ai/evals-speech-language

evals: support setting language on built-in speech/transcription services

Author: markbackman

.gitignore

@@ -71,4 +71,6 @@ htmlcov/
 whisker_setup.py
 
 # MoQ transport
-*.pem
+*.pem
+# Release-eval run artifacts (logs and conversation recordings)
+scripts/release-evals/test-runs/

changelog/5171.added.md

@@ -0,0 +1 @@
+- Added an optional `language` key to the eval harness's built-in `user.speech:` and `judge.transcription:` blocks. Each built-in speech service builder (`kokoro`, `cartesia`, `whisper`, `moonshine`) now forwards `language` (a code like `zh` or a `Language`) into the service settings, so non-English audio evals can synthesize user turns and transcribe bot audio in the right language without the `factory:` escape hatch. Omitting `language` is unchanged; the TTS audio cache key now includes the language so English and non-English renders of the same text don't collide.

changelog/5171.fixed.2.md

@@ -0,0 +1,3 @@
+- Fixed `WhisperSTTService` silently transcribing in English when its model can't handle the configured language. The English-only models — every `.en` one, including the default `distil-medium.en` — accept any language and transcribe as English regardless, so `Settings(language=Language.ES)` produced fluent-looking English rather than an error. Constructing such a pairing now raises a `ValueError` naming the model and its supported languages; a mid-call switch via `STTUpdateSettingsFrame` reports a non-fatal `ErrorFrame` instead, leaving the pipeline running.
+
+  ⚠️ Code that set a non-English `language` on an English-only model was getting English transcripts and now raises at construction. Use a multilingual model (e.g. `large-v3-turbo`) or drop the `language`.

changelog/5171.fixed.md

@@ -0,0 +1 @@
+- Fixed `KokoroTTSService` failing to synthesize French and Mandarin. kokoro-onnx phonemizes through espeak-ng, which has no `zh` and no bare `fr` voice, so both raised `language "..." is not supported by the espeak backend` at synthesis time. Mandarin (including the `zh-CN`/`zh-HK`/`zh-TW` variants) now maps to `cmn` and French to `fr-fr`, with `fr-be`, `fr-ch` and `pt-br` mapped to the regional espeak-ng voices they have.

scripts/release-evals/README.md

@@ -35,7 +35,9 @@ The harness runs the judge, the user's voice, and the bot-speech transcriber
   [Moonshine](https://github.com/moonshine-ai/moonshine) (Whisper is available
   as an alternative via the scenario's `transcription:` block). All run from
   local ONNX/model files that download once on first use (cached under
-  `~/.cache/pipecat/evals/tts`). No keys, no per-run cost.
+  `~/.cache/pipecat/evals/tts`). No keys, no per-run cost. Non-English
+  transcription needs a multilingual model, which the English-only defaults
+  aren't — `language_switch_audio` pulls Whisper's `tiny` (75MB).
 - **Each bot's own credentials.** A bot is a real example, so it needs the same
   service API keys it normally would, in your `.env` (e.g. `$OPENAI_API_KEY`,
   `$CARTESIA_API_KEY`, `$DEEPGRAM_API_KEY`, ...). A bot whose keys are missing

scripts/release-evals/manifest.yaml

@@ -43,6 +43,7 @@ suite:
       - interruption_audio
       - interruption_text
       - language_switch
+      - language_switch_audio
   - bot: voice/voice-cartesia-http.py
     scenarios: [capital_question]
   - bot: voice/voice-cartesia-turns.py

scripts/release-evals/scenarios/language_switch_audio.yaml

@@ -0,0 +1,39 @@
+name: language_switch_audio
+
+# Audio modality: the user asks (in English) for Spanish replies, and the judge
+# evaluates the transcription of the bot's actual Spanish speech — the end-to-end
+# check that a non-English answer survives the bot's TTS and the harness's STT.
+# The text-modality `language_switch` covers the same instruction, but judges the
+# LLM's text and so never exercises either audio path.
+#
+# One transcriber serves the whole run, and this one is Spanish. The bot's
+# English greeting is therefore asserted on `tts_response` (the text the TTS
+# reports speaking) rather than `response` (the transcription), which would be
+# English decoded as Spanish. Only the post-switch turn asserts on `response`.
+
+user: !include user_audio.yaml
+
+judge:
+  modality: audio
+  eval:
+    service: ollama
+    model: gemma2:9b
+  transcription:
+    # Whisper rather than the default Moonshine: Moonshine's Spanish model
+    # returns nothing on the bot's synthesized Spanish. `tiny` because Whisper's
+    # own default model is English-only, and it is the smallest multilingual one
+    # (75MB) that transcribes this reply cleanly.
+    service: whisper
+    model: tiny
+    language: es
+
+turns:
+  # Wait for the bot's on-connect greeting before speaking (avoids barging into it).
+  - expect:
+      - event: tts_response
+        eval: "the bot opens the conversation in some way (a greeting, an introduction, an offer to help, or a question to get the user started)"
+
+  - user: "From now on, reply only in Spanish. Where is La Sagrada Familia?"
+    expect:
+      - event: response
+        eval: "the response is written in Spanish and says that La Sagrada Familia is in Barcelona"

src/pipecat/evals/scenario.py

@@ -107,7 +107,8 @@
                   modality: audio          # audio | text (default text)
                   speech:                  # required when modality is audio
                     service: kokoro        # local TTS that synthesizes the user turns
-                    voice: af_heart
+                    voice: af_heart        # voices are language-specific
+                    language: en           # optional; must match the voice
                     sample_rate: 16000     # optional
 
             ``audio`` streams synthesized user audio to the bot (exercising its
@@ -122,6 +123,7 @@
                   transcription:           # required when modality is audio
                     service: moonshine     # STT for the bot's audio (or whisper)
                     model: small-streaming # optional
+                    language: en           # optional; the language the bot speaks
                     padding_secs: 0        # optional; silence padded around the
                                            # segment (default: 2)
 
@@ -351,14 +353,17 @@ class EvalScenario:
             greeting is silent. True (audio): the bot speaks, and the judge
             evaluates the transcription of its actual audio.
         transcriber: Parsed from the ``judge.transcription:`` block; the STT
-            config (``service`` defaults to ``moonshine``, plus ``model``) used to
-            transcribe the bot's audio for the ``response`` event (``None`` in
-            text modality).
+            config (``service`` defaults to ``moonshine``, plus ``model`` and an
+            optional ``language`` code) used to transcribe the bot's audio for the
+            ``response`` event (``None`` in text modality). Set ``language`` when
+            the bot speaks a non-English language so the STT doesn't default to
+            English.
         user_audio: TTS config the harness uses to generate user audio. When
             present, the harness streams RTVI ``raw-audio`` (not ``send-text``)
             to the bot, exercising its STT for real. Mapping with ``service``,
-            ``voice``, and optional ``model`` / ``sample_rate`` /
-            ``api_key``. Omit for text-only evals (default).
+            ``voice``, and optional ``model`` / ``language`` / ``sample_rate`` /
+            ``api_key``. Set ``language`` (a code like ``zh``) to synthesize
+            non-English user turns. Omit for text-only evals (default).
         trigger_disconnect: Whether the harness fires the bot's
             ``on_client_disconnected`` handler when this scenario's connection
             ends. Bots often cancel their pipeline there, so this is False by

src/pipecat/evals/services.py

@@ -18,9 +18,45 @@
 from typing import Any
 
 from pipecat.services.llm_service import LLMService
-from pipecat.services.settings import NOT_GIVEN
+from pipecat.services.settings import NOT_GIVEN, _NotGiven
 from pipecat.services.stt_service import STTService
 from pipecat.services.tts_service import TTSService
+from pipecat.transcriptions.language import Language
+
+
+def _cfg_language(cfg: dict) -> Language | _NotGiven:
+    """Coerce a config's optional ``language`` value to a :class:`Language`.
+
+    ``Language`` is a ``StrEnum``, so both a code string (e.g. ``"zh"``) and a
+    ``Language`` are accepted. Each concrete service maps the ``Language`` to its
+    own provider code internally (via ``resolve_language``), so the eval layer
+    only needs to hand off a ``Language``.
+
+    Args:
+        cfg: A ``user.speech`` or ``judge.transcription`` config mapping.
+
+    Returns:
+        The resolved ``Language``, or ``NOT_GIVEN`` when ``language`` is absent,
+        which leaves the service's own default in place.
+
+    Raises:
+        ValueError: If ``language`` is set to a value that is not a recognized
+            language code.
+    """
+    value = cfg.get("language")
+    if value is None:
+        return NOT_GIVEN
+    if isinstance(value, str):
+        value = value.strip()
+        if not value:
+            return NOT_GIVEN
+    try:
+        return Language(value)
+    except ValueError as e:
+        raise ValueError(
+            f"Unknown language {value!r} in eval speech/transcription config; "
+            "expected a language code like 'zh' or a Language value."
+        ) from e
 
 
 def kokoro_service(voice_cfg: dict, sample_rate: int) -> TTSService:
@@ -29,17 +65,46 @@ def kokoro_service(voice_cfg: dict, sample_rate: int) -> TTSService:
     Kokoro runs an ONNX model locally (no API key, no per-run cost), so the eval
     suite synthesizes user audio for free. The model files are downloaded once
     on first use and cached under ``~/.cache/kokoro-onnx``.
+
+    Args:
+        voice_cfg: The ``user.speech`` config mapping:
+
+            - ``voice``: Kokoro voice id (e.g. ``af_heart``).
+            - ``language``: Optional language code (e.g. ``zh``) or ``Language``.
+              When omitted, Kokoro keeps its own default (English). Voices are
+              language-specific, so a non-English language needs a matching voice
+              — ``af_heart`` speaks US English whatever the language is set to.
+
+        sample_rate: Sample rate for the synthesized audio.
     """
     from pipecat.services.kokoro.tts import KokoroTTSService
 
     return KokoroTTSService(
-        settings=KokoroTTSService.Settings(voice=str(voice_cfg.get("voice", ""))),
+        settings=KokoroTTSService.Settings(
+            voice=str(voice_cfg.get("voice", "")),
+            language=_cfg_language(voice_cfg),
+        ),
         sample_rate=sample_rate,
     )
 
 
 def cartesia_service(voice_cfg: dict, sample_rate: int) -> TTSService:
-    """Build a Cartesia TTS service from the ``user_audio`` config."""
+    """Build a Cartesia TTS service from the ``user_audio`` config.
+
+    Args:
+        voice_cfg: The ``user.speech`` config mapping:
+
+            - ``voice``: Cartesia voice id.
+            - ``model``: Optional model (defaults to ``sonic-2``).
+            - ``api_key``: Optional key (falls back to ``$CARTESIA_API_KEY``).
+            - ``language``: Optional language code (e.g. ``zh``) or ``Language``.
+              When omitted, Cartesia keeps its own default (English).
+
+        sample_rate: Sample rate for the synthesized audio.
+
+    Raises:
+        RuntimeError: If no API key is given in the config or the environment.
+    """
     from pipecat.services.cartesia.tts import CartesiaHttpTTSService
 
     # Prefer an explicit api_key in the config; fall back to the env var so
@@ -55,6 +120,7 @@ def cartesia_service(voice_cfg: dict, sample_rate: int) -> TTSService:
         settings=CartesiaHttpTTSService.Settings(
             voice=str(voice_cfg.get("voice", "")),
             model=voice_cfg.get("model") or "sonic-2",
+            language=_cfg_language(voice_cfg),
         ),
         sample_rate=sample_rate,
     )
@@ -77,6 +143,23 @@ def whisper_service(config: dict) -> STTService:
     whose ``no_speech_prob`` jitters across ~0.4-0.6 run to run (a dropped segment
     yields no ``TranscriptionFrame``, so the harness then waits out the whole
     transcription timeout). Disable the filter with a permissive threshold.
+
+    Args:
+        config: The ``judge.transcription`` config mapping:
+
+            - ``device``: ``cpu`` (default) or ``cuda``.
+            - ``compute_type``: Whisper compute type (``int8`` on CPU).
+            - ``model``: Optional Whisper model (left unset to use Whisper's own).
+              Whisper's default is English-only, as is every ``.en`` model, so a
+              non-English ``language`` needs a multilingual model here (e.g.
+              ``large-v3-turbo``).
+            - ``language``: Optional language code (e.g. ``es``) or ``Language``.
+              When omitted, Whisper keeps its own default (English) — it does not
+              auto-detect, so a non-English bot needs this set.
+
+    Raises:
+        ValueError: If ``language`` names a language the chosen model can't
+            transcribe (raised by :class:`~pipecat.services.whisper.stt.WhisperSTTService`).
     """
     from pipecat.services.whisper.stt import WhisperSTTService
 
@@ -91,6 +174,7 @@ def whisper_service(config: dict) -> STTService:
         settings=WhisperSTTService.Settings(
             no_speech_prob=1.0,
             model=config.get("model", NOT_GIVEN),
+            language=_cfg_language(config),
         ),
     )
 
@@ -100,14 +184,32 @@ def moonshine_service(config: dict) -> STTService:
 
     Moonshine runs on the CPU via ONNX Runtime (no GPU, no API key) and is small
     and fast. On the short, isolated bot-answer segments the harness transcribes,
-    it tends to keep the answer where Whisper sometimes drops it. ``model`` selects
-    the architecture (a :class:`~pipecat.services.moonshine.stt.Model` value or
-    string; default ``Model.SMALL_STREAMING``).
+    it tends to keep the answer where Whisper sometimes drops it.
+
+    Args:
+        config: The ``judge.transcription`` config mapping:
+
+            - ``model``: Optional architecture, as a
+              :class:`~pipecat.services.moonshine.stt.Model` or the equivalent
+              string (default ``Model.SMALL_STREAMING``). Only ``base`` has
+              non-English models, so a non-English ``language`` needs it, and
+              raises at construction naming the models it does have if the
+              pairing has none.
+            - ``language``: Optional language code (e.g. ``es``) or ``Language``.
+              When omitted, Moonshine keeps its own default (English).
+
+    Prefer :func:`whisper_service` for a non-English bot: Moonshine's non-English
+    models transcribe synthesized speech unreliably, returning an empty transcript
+    or dropping the tail of an utterance, and an empty transcript is
+    indistinguishable from a bot that said nothing.
     """
     from pipecat.services.moonshine.stt import Model, MoonshineSTTService
 
     return MoonshineSTTService(
-        settings=MoonshineSTTService.Settings(model=config.get("model") or Model.SMALL_STREAMING),
+        settings=MoonshineSTTService.Settings(
+            model=config.get("model") or Model.SMALL_STREAMING,
+            language=_cfg_language(config),
+        ),
     )
 
 

src/pipecat/evals/speech.py

@@ -66,14 +66,15 @@ def tts_sample_rate(voice_cfg: dict) -> int:
 def tts_cache_key(voice_cfg: dict) -> str:
     """A stable identity for a ``user_audio`` config, for caching synthesized audio.
 
-    Covers the audio's semantic identity (service, voice, model) but not the
-    sample rate, so different rates reuse the same slot (a mismatch just triggers
-    regeneration in :meth:`EvalSpeech.generate`).
+    Covers the audio's semantic identity (service, voice, model, language) but not
+    the sample rate, so different rates reuse the same slot (a mismatch just
+    triggers regeneration in :meth:`EvalSpeech.generate`).
     """
     service = str(voice_cfg.get("service", "")).lower()
     voice = str(voice_cfg.get("voice", ""))
     model = str(voice_cfg.get("model", ""))
-    return "\x00".join((service, voice, model))
+    language = str(voice_cfg.get("language") or "")
+    return "\x00".join((service, voice, model, language))
 
 
 class EvalSpeech: