[WIP]Add CS-FLEURS code-switched ASR benchmark with Mixed Error Rate scoring

docs/evaluation/speech-audio.md

@@ -789,3 +789,73 @@ Output:
     st/test.jsonl    # one record per (src→tgt, audio) for translation
     audio/<locale>/...wav
 ```
+
+## CS-FLEURS
+
+[CS-FLEURS](https://huggingface.co/datasets/byan/cs-fleurs) ([paper](https://arxiv.org/abs/2509.14161)) is a massively multilingual *code-switched* extension of FLEURS, covering 52 languages and 113 code-switched language pairs. Each utterance is built with a Matrix language (grammatical frame) and an Embedded language (inserted words), so the reference transcription mixes two languages — and often two scripts — in one sentence.
+
+**Subtasks (ASR only):** one per test set — `cs-fleurs.read` (14 X-English pairs, human-read & human-validated; the paper's intended benchmark), `cs-fleurs.mms` (45 X-English pairs, concatenative MMS-TTS), `cs-fleurs.xtts-test1` (16 X-English pairs, generative XTTS-v2), `cs-fleurs.xtts-test2` (60 language pairs, generative XTTS-v2).
+
+**Splits:** `test`
+
+**Scoring — Mixed Error Rate (MER).** Because each utterance mixes two scripts, a single per-utterance CER-or-WER choice mis-scores it (e.g. whole-utterance CER over a `cmn-eng` sentence scores the embedded English at the character level). Instead, records set `use_mer=true` and the audio evaluator computes a Mixed Error Rate: scriptio-continua scripts (Han / kana / Hangul / Thai / Lao / Myanmar / Khmer) are counted by character and space-delimited scripts (e.g. Latin) by word, within the same utterance, then scored with the standard word-level edit distance. This matches the standard MER used for Mandarin-English code-switching (e.g. SEAME). The mixed figure is reported in the headline `wer*` column. `subset_for_metrics` is the code-switched pair (e.g. `ara-eng`), giving a per-pair breakdown; the group score module adds a per-test-set and overall entry-weighted figure. MER only affects CS-FLEURS — it is gated behind `use_mer`, which monolingual benchmarks (including `fleurs`) never set, so their CER/WER behavior is unchanged.
+
+In addition, a parallel **CER** column (`cer*`) is emitted for direct comparison with the [CS-FLEURS paper](https://arxiv.org/abs/2509.14161), which reports *"case insensitive and unpunctuated character error rate"*. For CS-FLEURS this CER is normalized the paper's way — lowercased and de-punctuated but with **combining marks preserved** (so Thai/Myanmar vowel and tone marks count), via the `lower_nopunct` normalization mode. So each pair gets both a code-switching-aware MER (`wer*`) and a paper-comparable CER (`cer*`).
+
+!!! warning "Known limitation: Thai/Myanmar diacritic stripping (reviewers' input welcome)"
+    The shared `multilingual` normalization removes Unicode combining marks (category `Mn`, and turns `Mc` into spaces) as "diacritics". This is correct for Latin accents but **destroys meaning-bearing vowel and tone marks in Thai/Myanmar** (e.g. `ที่` → `ท`). The paper-comparable **CER column (`cer*`) avoids this** by using the mark-preserving `lower_nopunct` normalization. However, the **MER column (`wer*`) still inherits the shared `multilingual` normalization**, so MER for Thai/Myanmar is computed over consonant skeletons (and `mer_grapheme` grapheme-clustering is consequently a no-op there, since marks are stripped before segmentation). This mark-stripping is **pre-existing** — it affects the plain `use_cer` path and `fleurs` too. A full fix makes the shared multilingual normalization script-aware (preserve `Mn`/`Mc` for Thai/Lao/Myanmar/Khmer/Indic), which would change `fleurs` numbers — deferred pending maintainer guidance. Affects only ~695 of ~49k CS-FLEURS records (Thai + Myanmar); use the `cer*` column for those.
+
+### Dataset Location
+
+- Benchmark group is defined in [`nemo_skills/dataset/cs-fleurs/__init__.py`](https://github.com/NVIDIA-NeMo/Skills/blob/main/nemo_skills/dataset/cs-fleurs/__init__.py); per-subtask config lives in `cs-fleurs/<test-set>/__init__.py`.
+- Original dataset is hosted on [HuggingFace](https://huggingface.co/datasets/byan/cs-fleurs) (CC-BY-NC-4.0).
+
+### Preparing CS-FLEURS Data
+
+Audio is downloaded automatically from HuggingFace. A single `prepare_data cs-fleurs` run produces all four test sets at once. Pass the parent group name (`cs-fleurs`), not the dotted subtask names. Use `--subsets` to prepare only some test sets.
+
+```bash
+ns prepare_data cs-fleurs \
+    --data_dir /path/to/data \
+    --cluster <cluster_name> \
+    --subsets read mms xtts-test1 xtts-test2
+```
+
+Output:
+
+```text
+<data_dir>/cs-fleurs/
+    read/test.jsonl         # one ASR record per (pair, audio)
+    mms/test.jsonl
+    xtts-test1/test.jsonl
+    xtts-test2/test.jsonl
+    raw/<read|mms|xtts>/<split>/audio/<lang3>/...wav
+```
+
+### Baseline results
+
+Reference numbers from **Whisper `large-v3`** (an external multilingual ASR model, *not* a SpeechLM/SALM model), transcribing the full `cs-fleurs.read` subset (5,818 utterances, greedy decoding) and scored with this benchmark's MER path (multilingual normalization). These are a sanity-check baseline for the metric and data pipeline, not a tuned result. MER is reported in the `wer` column; `subset_for_metrics` gives the per-language-pair breakdown.
+
+Example output:
+
+```text
+--------------------------------- cs-fleurs.read ---------------------------------
+language_pair  | num_entries | mer
+ara-eng        | 989         | 29.03
+ces-eng        | 326         | 16.39
+cmn-eng        | 1321        | 17.14
+deu-eng        | 298         | 10.98
+fra-eng        | 307         | 19.39
+hin-eng        | 233         | 41.58
+ita-eng        | 176         | 12.21
+jpn-eng        | 196         | 42.75
+kor-eng        | 466         | 34.75
+por-eng        | 338         | 15.23
+rus-eng        | 337         | 22.05
+slk-eng        | 314         | 24.62
+spa-eng        | 320         | 9.60
+tel-eng        | 197         | 60.16
+overall        | 5818        | 25.75
+```
+
+Numbers track expected difficulty: low for high-resource European pairs (spa/deu/ita ~10–12%), higher for low-resource or distinct-script languages (tel/jpn/hin/kor). A SpeechLM model evaluated through the standard pipeline reports the same MER metric.

nemo_skills/dataset/cs-fleurs/__init__.py

@@ -0,0 +1,41 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""CS-FLEURS: massively multilingual code-switched ASR evaluation.
+
+Code-switched speech recognition across the four CS-FLEURS test sets, each a
+sub-benchmark scored with per-language CER/WER (CER for scriptio-continua matrix
+languages, WER otherwise):
+
+- ``cs-fleurs.read``        14 X-English pairs, human-read speech (the paper's
+                            intended, human-validated benchmarking set)
+- ``cs-fleurs.mms``         45 X-English pairs, concatenative MMS-TTS speech
+- ``cs-fleurs.xtts-test1``  16 X-English pairs, generative XTTS-v2 speech
+- ``cs-fleurs.xtts-test2``  60 language pairs, generative XTTS-v2 speech
+
+Dataset: https://huggingface.co/datasets/byan/cs-fleurs (CC-BY-NC-4.0)
+Paper: CS-FLEURS: A Massively Multilingual and Code-Switched Speech Dataset
+       (https://arxiv.org/abs/2509.14161)
+"""
+
+REQUIRES_DATA_DIR = True
+IS_BENCHMARK_GROUP = True
+SCORE_MODULE = "nemo_skills.dataset.cs-fleurs.audio_score"
+
+BENCHMARKS = {
+    "cs-fleurs.read": {},
+    "cs-fleurs.mms": {},
+    "cs-fleurs.xtts-test1": {},
+    "cs-fleurs.xtts-test2": {},
+}

nemo_skills/dataset/cs-fleurs/audio_score.py

@@ -0,0 +1,117 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Group-level aggregation for the CS-FLEURS code-switched ASR benchmark.
+
+Each sub-benchmark (read / mms / xtts-test1 / xtts-test2) already reports a
+per-language-pair WER/CER breakdown via ``subset_for_metrics``. This module adds
+a per-test-set headline number (entry-weighted WER/CER, where CER is stored
+under the ``wer`` key for consistency with corpus-level aggregation) plus an
+overall entry-weighted figure across the prepared test sets.
+
+Note: the test sets differ in nature — ``read`` is human-validated while the
+others are synthetic (XTTS / MMS) — so the per-test-set numbers are the
+meaningful comparison; the overall figure is a convenience aggregate only.
+"""
+
+SUBSET_NAMES = ["read", "mms", "xtts-test1", "xtts-test2"]
+
+
+def compute_score(combined_metrics: dict) -> dict:
+    """Aggregate CS-FLEURS sub-benchmark metrics into per-test-set + overall scores.
+
+    ``combined_metrics`` maps each sub-benchmark name (e.g. ``cs-fleurs.read``)
+    to a dict of eval-mode -> metrics dict (as emitted by the audio metrics
+    computation). Returns ``{eval_mode: {subset_name: {...}, "overall": {...}}}``.
+    """
+    benchmarks = {k: v for k, v in combined_metrics.items() if k.split(".")[-1] in SUBSET_NAMES}
+    if not benchmarks:
+        return {}
+
+    weighted_metrics = ["wer", "wer_macro"]
+    summed_metrics = ["substitutions", "insertions", "deletions", "ref_words"]
+    # Generation/timing metrics that are entry-weighted like WER but only emitted
+    # when present: pure-ASR runs omit success_rate / no_answer, so we must not
+    # fabricate them as 0.
+    optional_weighted = ["success_rate", "avg_tokens", "no_answer"]
+
+    first_benchmark = next(iter(benchmarks.values()))
+    eval_modes = list(first_benchmark.keys())
+
+    def _summarize(metrics_list: list[dict]) -> dict | None:
+        total_entries = 0
+        total_gen_seconds = 0
+        gen_seconds_seen = False
+        weighted_sums = {m: 0.0 for m in weighted_metrics + optional_weighted}
+        weighted_counts = {m: 0 for m in weighted_metrics + optional_weighted}
+        sums = {m: 0 for m in summed_metrics}
+        # Track presence separately so a legitimately-zero metric (e.g. 0
+        # substitutions on a perfect subset) is still emitted, rather than
+        # dropped by a truthiness check on the sum.
+        seen = {m: False for m in summed_metrics}
+
+        for metrics in metrics_list:
+            num_entries = metrics["num_entries"]  # required; fail loudly if absent
+            if num_entries == 0:
+                continue
+            total_entries += num_entries
+            if "gen_seconds" in metrics:
+                total_gen_seconds += metrics["gen_seconds"]
+                gen_seconds_seen = True
+            for m in weighted_metrics + optional_weighted:
+                if m in metrics:
+                    weighted_sums[m] += metrics[m] * num_entries
+                    weighted_counts[m] += num_entries
+            for m in summed_metrics:
+                if m in metrics:
+                    sums[m] += metrics[m]
+                    seen[m] = True
+
+        if total_entries == 0:
+            return None
+
+        agg = {"num_entries": total_entries}
+        if gen_seconds_seen:
+            agg["gen_seconds"] = total_gen_seconds
+        for m in weighted_metrics:
+            if weighted_counts[m] > 0:
+                agg[m] = round(weighted_sums[m] / weighted_counts[m], 2)
+        for m in optional_weighted:
+            if weighted_counts[m] > 0:
+                value = weighted_sums[m] / weighted_counts[m]
+                agg[m] = int(value) if m == "avg_tokens" else value
+        for m in summed_metrics:
+            if seen[m]:
+                agg[m] = sums[m]
+        return agg
+
+    aggregated: dict[str, dict] = {}
+    for eval_mode in eval_modes:
+        per_subset = {}
+        for name, benchmark_data in benchmarks.items():
+            if eval_mode not in benchmark_data:
+                continue
+            summary = _summarize([benchmark_data[eval_mode]])
+            if summary is not None:
+                per_subset[name.split(".")[-1]] = summary
+
+        if not per_subset:
+            continue
+
+        overall = _summarize(list(per_subset.values()))
+        if overall is not None:
+            per_subset["overall"] = overall
+        aggregated[eval_mode] = per_subset
+
+    return aggregated

nemo_skills/dataset/cs-fleurs/languages.py

@@ -0,0 +1,163 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Static CS-FLEURS language metadata.
+
+CS-FLEURS metadata identifies languages by ISO 639-3 codes (e.g. ``ara``,
+``cmn``, ``deu``) inside a ``"<matrix>-<embedded>"`` ``language`` field, which
+differs from the BCP-47 locales used by the plain ``fleurs`` benchmark. This
+table maps each code to a display name and an ISO 639-1 code; the latter is
+passed to the audio evaluator as ``src_lang`` so that its number-normalization
+(``num2words``) and multilingual normalizer behave correctly. Codes without a
+ISO 639-1 form map to ``None`` (number normalization is then skipped, which the
+evaluator handles gracefully).
+"""
+
+from __future__ import annotations
+
+# ISO 639-3 code -> (display name, ISO 639-1 code or None for num2words/normalization)
+CS_FLEURS_LANGUAGES: dict[str, tuple[str, str | None]] = {
+    "ara": ("Arabic", "ar"),
+    "aze": ("Azerbaijani", "az"),
+    "ben": ("Bengali", "bn"),
+    "bul": ("Bulgarian", "bg"),
+    "cat": ("Catalan", "ca"),
+    "ceb": ("Cebuano", None),
+    "ces": ("Czech", "cs"),
+    "cmn": ("Mandarin Chinese", "zh"),
+    "cym": ("Welsh", "cy"),
+    "deu": ("German", "de"),
+    "ell": ("Greek", "el"),
+    "eng": ("English", "en"),
+    "fin": ("Finnish", "fi"),
+    "fra": ("French", "fr"),
+    "guj": ("Gujarati", "gu"),
+    "heb": ("Hebrew", "he"),
+    "hin": ("Hindi", "hi"),
+    "hun": ("Hungarian", "hu"),
+    "ind": ("Indonesian", "id"),
+    "isl": ("Icelandic", "is"),
+    "ita": ("Italian", "it"),
+    "jav": ("Javanese", "jv"),
+    "jpn": ("Japanese", "ja"),
+    "kan": ("Kannada", "kn"),
+    "kaz": ("Kazakh", "kk"),
+    "khm": ("Khmer", "km"),
+    "kir": ("Kyrgyz", "ky"),
+    "kor": ("Korean", "ko"),
+    "lao": ("Lao", "lo"),
+    "lav": ("Latvian", "lv"),
+    "lug": ("Ganda", "lg"),
+    "mal": ("Malayalam", "ml"),
+    "mar": ("Marathi", "mr"),
+    "mya": ("Burmese", "my"),
+    "nld": ("Dutch", "nl"),
+    "pan": ("Punjabi", "pa"),
+    "pol": ("Polish", "pl"),
+    "por": ("Portuguese", "pt"),
+    "ron": ("Romanian", "ro"),
+    "rus": ("Russian", "ru"),
+    "slk": ("Slovak", "sk"),
+    "spa": ("Spanish", "es"),
+    "swe": ("Swedish", "sv"),
+    "swh": ("Swahili", "sw"),
+    "tam": ("Tamil", "ta"),
+    "tel": ("Telugu", "te"),
+    "tgk": ("Tajik", "tg"),
+    "tgl": ("Tagalog", "tl"),
+    "tha": ("Thai", "th"),
+    "tur": ("Turkish", "tr"),
+    "ukr": ("Ukrainian", "uk"),
+    "urd": ("Urdu", "ur"),
+    "uzb": ("Uzbek", "uz"),
+    "vie": ("Vietnamese", "vi"),
+    "yor": ("Yoruba", "yo"),
+    "yue": ("Cantonese Chinese", None),
+    "zlm": ("Malay", "ms"),
+}
+
+# Matrix languages scored with Character Error Rate instead of Word Error Rate
+# (scriptio-continua: no explicit word boundaries). Mirrors the fleurs benchmark
+# CER_LOCALES, expressed in ISO 639-3.
+CER_LANGS: frozenset[str] = frozenset(
+    {
+        "cmn",  # Mandarin Chinese
+        "yue",  # Cantonese Chinese
+        "jpn",  # Japanese
+        "kor",  # Korean
+        "tha",  # Thai
+        "lao",  # Lao
+        "mya",  # Burmese
+        "khm",  # Khmer
+        "vie",  # Vietnamese
+    }
+)
+
+
+def split_pair(language: str) -> tuple[str, str]:
+    """Split a CS-FLEURS ``language`` field (``"<matrix>-<embedded>"``).
+
+    Returns ``(matrix_code, embedded_code)``. If no separator is present the
+    whole string is treated as the matrix language with an empty embedded code.
+    """
+    parts = language.replace("_", "-").split("-")
+    matrix = parts[0]
+    embedded = parts[1] if len(parts) > 1 else ""
+    return matrix, embedded
+
+
+def get_lang_name(code: str) -> str:
+    """Display name for an ISO 639-3 code.
+
+    Raises KeyError on an unknown code: every CS-FLEURS language is expected in
+    the table, so an unknown code signals a dataset/table mismatch we want to
+    fail on loudly rather than paper over.
+    """
+    return CS_FLEURS_LANGUAGES[code][0]
+
+
+def get_iso1(code: str) -> str | None:
+    """ISO 639-1 code for a (known) ISO 639-3 code.
+
+    Returns None for languages that have no ISO 639-1 form (e.g. ``ceb``,
+    ``yue``); number normalization is then skipped, which the evaluator handles.
+    Raises KeyError on an unknown code (see ``get_lang_name``).
+    """
+    return CS_FLEURS_LANGUAGES[code][1]
+
+
+def uses_cer(matrix_code: str) -> bool:
+    """Whether a code-switched pair with this matrix language is scored with CER."""
+    return matrix_code in CER_LANGS
+
+
+# Scripts whose characters carry spacing/non-spacing combining marks (vowel
+# signs, tone marks). For these, Mixed Error Rate must segment by grapheme
+# cluster, since per-codepoint splitting would separate a base consonant from
+# its marks. Han / kana / Hangul are precomposed (no combining marks), so
+# codepoint-level segmentation is already correct and they are intentionally
+# excluded here.
+GRAPHEME_CLUSTER_LANGS: frozenset[str] = frozenset(
+    {
+        "tha",  # Thai
+        "lao",  # Lao
+        "mya",  # Burmese
+        "khm",  # Khmer
+    }
+)
+
+
+def uses_grapheme(code: str) -> bool:
+    """Whether this language's script needs grapheme-cluster MER segmentation."""
+    return code in GRAPHEME_CLUSTER_LANGS