Merge remote-tracking branch 'origin/main' into speech/apptek-callcenter-dialogues

Author: pzelasko

docs/evaluation/speech-audio.md

@@ -663,15 +663,15 @@ Per-domain breakdowns are included automatically based on the `domain_label` fie
 
 CoVoST 2 is a large-scale multilingual corpus for speech recognition (ASR) and speech translation (AST), built on Common Voice audio with translation references from Facebook's [CoVoST v2](https://github.com/facebookresearch/covost) release.
 
-**Tasks:** ASR (monolingual transcription) and AST (X→en / en→X translation)
+**Subtasks:** `covost2.asr` (monolingual transcription) and `covost2.st` (X→en / en→X translation)
 
 **Splits:** `validation`, `test`
 
 For non-alphabetic scripts (`zh-CN`, `ja`), evaluation reports Character Error Rate (CER) instead of Word Error Rate (WER); the choice is made per-sample via the `use_cer` flag set during data preparation.
 
 ### Dataset Location
 
-- Benchmark is defined in [`nemo_skills/dataset/covost2/__init__.py`](https://github.com/NVIDIA-NeMo/Skills/blob/main/nemo_skills/dataset/covost2/__init__.py)
+- Benchmark group is defined in [`nemo_skills/dataset/covost2/__init__.py`](https://github.com/NVIDIA-NeMo/Skills/blob/main/nemo_skills/dataset/covost2/__init__.py); per-subtask config lives in `covost2/asr/__init__.py` and `covost2/st/__init__.py`.
 - Original benchmark source is hosted on [GitHub](https://github.com/facebookresearch/covost)
 
 ### Preparing CoVoST 2 Data
@@ -687,73 +687,63 @@ Unlike most other benchmarks on this page, **CoVoST 2 does not auto-download aud
 
 and the corresponding `validated.tsv` (columns: `path, split, lang, sentence`).
 
-The `--languages` flag selects which CoVoST 2 languages are prepared. For ASR it filters the source-language audio that is transcribed; for AST every valid X→en / en→X pair touching the listed languages is included. Omit it to prepare all 21 supported languages.
+A single `prepare_data covost2` run produces both subtasks at once — `covost2/asr/{split}.jsonl` and `covost2/st/{split}.jsonl`. There is no `--task` flag; pass the parent group name (`covost2`), not the dotted subtask names.
 
-=== "ASR"
+The `--languages` flag selects which CoVoST 2 languages are prepared. For ASR it filters the source-language audio that is transcribed; for ST every valid X→en / en→X pair touching the listed languages is included. Omit it to prepare all 21 supported languages.
 
-    ```bash
-    ns prepare_data covost2 \
-        --data_dir /path/to/data \
-        --cluster <cluster_name> \
-        --task ASR \
-        --languages de fr \
-        --split test \
-        --cv_data_dir /workspace/datasets/covost2 \
-        --validated_tsv /workspace/datasets/covost2/validated.tsv
-    ```
-
-=== "AST"
+```bash
+ns prepare_data covost2 \
+    --data_dir /path/to/data \
+    --cluster <cluster_name> \
+    --languages de fr es \
+    --split test \
+    --cv_data_dir /workspace/datasets/covost2 \
+    --validated_tsv /workspace/datasets/covost2/validated.tsv
+```
 
-    ```bash
-    ns prepare_data covost2 \
-        --data_dir /path/to/data \
-        --cluster <cluster_name> \
-        --task AST \
-        --languages de fr es \
-        --split test \
-        --cv_data_dir /workspace/datasets/covost2 \
-        --validated_tsv /workspace/datasets/covost2/validated.tsv
-    ```
+Output:
 
-Each `--task` produces a separate manifest: `{split}-asr.jsonl` or `{split}-ast.jsonl` (e.g. `test-asr.jsonl`).
+```
+<data_dir>/covost2/
+    asr/test.jsonl   # one record per (lang, audio) for transcription
+    st/test.jsonl    # one record per (src→tgt, audio) for translation
+    audio/<lang>/<split>/...wav
+```
 
 ## FLEURS
 
-[FLEURS](https://huggingface.co/datasets/google/fleurs) (Few-shot Learning Evaluation of Universal Representations of Speech) is Google's multilingual speech benchmark covering 102 locales. It supports both ASR and AST.
+[FLEURS](https://huggingface.co/datasets/google/fleurs) (Few-shot Learning Evaluation of Universal Representations of Speech) is Google's multilingual speech benchmark covering 102 locales.
+
+**Subtasks:** `fleurs.asr` (monolingual transcription) and `fleurs.st` (`en_us` → locale and locale → `en_us` translation)
 
 **Splits:** `train`, `dev`, `test`
 
 CER (rather than WER) is used for these locales: `cmn_hans_cn`, `yue_hant_hk`, `ja_jp`, `th_th`, `lo_la`, `my_mm`, `km_kh`, `ko_kr`, `vi_vn`.
 
 ### Dataset Location
 
-- Benchmark is defined in [`nemo_skills/dataset/fleurs/__init__.py`](https://github.com/NVIDIA-NeMo/Skills/blob/main/nemo_skills/dataset/fleurs/__init__.py)
+- Benchmark group is defined in [`nemo_skills/dataset/fleurs/__init__.py`](https://github.com/NVIDIA-NeMo/Skills/blob/main/nemo_skills/dataset/fleurs/__init__.py); per-subtask config lives in `fleurs/asr/__init__.py` and `fleurs/st/__init__.py`.
 - Original dataset is hosted on [HuggingFace](https://huggingface.co/datasets/google/fleurs)
 
 ### Preparing FLEURS Data
 
-Audio is downloaded automatically from HuggingFace. As with CoVoST 2, `--task` produces `{split}-asr.jsonl` or `{split}-ast.jsonl`.
+Audio is downloaded automatically from HuggingFace. A single `prepare_data fleurs` run produces both subtasks at once — `fleurs/asr/{split}.jsonl` and `fleurs/st/{split}.jsonl`. There is no `--task` flag; pass the parent group name (`fleurs`), not the dotted subtask names.
 
-The `--languages` flag selects which FLEURS locales are prepared. For ASR it filters the source-language audio that is transcribed; for AST every (`en_us` → locale) and (locale → `en_us`) pair across the listed locales is included. Omit it to prepare all 102 locales.
+The `--languages` flag selects which FLEURS locales are prepared. ASR records are emitted for the listed locales. ST records are emitted for every (`en_us` → locale) and (locale → `en_us`) pair touching the listed locales — even if `en_us` is not itself in `--languages`, since it is the pivot. Omit `--languages` to prepare all 102 locales.
 
-=== "ASR"
-
-    ```bash
-    ns prepare_data fleurs \
-        --data_dir /path/to/data \
-        --cluster <cluster_name> \
-        --task ASR \
-        --languages en_us de_de fr_fr \
-        --split test
-    ```
+```bash
+ns prepare_data fleurs \
+    --data_dir /path/to/data \
+    --cluster <cluster_name> \
+    --languages en_us de_de fr_fr es_419 it_it ja_jp \
+    --split test
+```
 
-=== "AST"
+Output:
 
-    ```bash
-    ns prepare_data fleurs \
-        --data_dir /path/to/data \
-        --cluster <cluster_name> \
-        --task AST \
-        --languages en_us de_de fr_fr es_419 it_it ja_jp \
-        --split test
-    ```
+```
+<data_dir>/fleurs/
+    asr/test.jsonl   # one record per (locale, audio) for transcription
+    st/test.jsonl    # one record per (src→tgt, audio) for translation
+    audio/<locale>/...wav
+```

nemo_skills/dataset/asr-leaderboard/prepare.py

@@ -16,121 +16,122 @@
 
 Downloads and formats datasets from the official HF Open ASR Leaderboard ESB
 test-only sorted dataset (hf-audio/esb-datasets-test-only-sorted). This is the
-same data source used by the official leaderboard and the offline NeMo eval
-pipeline, ensuring apples-to-apples WER comparison.
+same data source used by the official leaderboard, ensuring apples-to-apples
+WER comparison.
 
 Audio paths in JSONL: /dataset/asr-leaderboard/data/{dataset}/{sample_id}.flac
 
 Usage:
     ns prepare_data asr-leaderboard
     ns prepare_data asr-leaderboard --datasets librispeech_clean ami
-    ns prepare_data asr-leaderboard --datasets earnings22
-    ns prepare_data asr-leaderboard --no-audio  # skip saving audio files
+    ns prepare_data asr-leaderboard --no-audio
 """
 
 import argparse
 import json
 from pathlib import Path
 
+import numpy as np
 import soundfile as sf
-from datasets import load_dataset
+from datasets import Audio, load_dataset
 from tqdm import tqdm
 
+HF_REPO = "hf-audio/esb-datasets-test-only-sorted"
 SYSTEM_MESSAGE = "You are a helpful assistant. /no_think"
-MIN_AUDIO_DURATION = 0.1  # Skip audio shorter than this (causes mel spectrogram errors)
+AUDIO_SAMPLE_RATE = 16000
 
-# (hf_repo, config, split, text_field, id_field)
+# (config, split, text_field, id_field)
 DATASET_CONFIGS = {
-    "librispeech_clean": ("hf-audio/esb-datasets-test-only-sorted", "librispeech", "test.clean", "text", "id"),
-    "librispeech_other": ("hf-audio/esb-datasets-test-only-sorted", "librispeech", "test.other", "text", "id"),
-    "voxpopuli": ("hf-audio/esb-datasets-test-only-sorted", "voxpopuli", "test", "text", "id"),
-    "tedlium": ("hf-audio/esb-datasets-test-only-sorted", "tedlium", "test", "text", "id"),
-    "gigaspeech": ("hf-audio/esb-datasets-test-only-sorted", "gigaspeech", "test", "text", "id"),
-    "spgispeech": ("hf-audio/esb-datasets-test-only-sorted", "spgispeech", "test", "text", "id"),
-    "earnings22": ("hf-audio/esb-datasets-test-only-sorted", "earnings22", "test", "text", "id"),
-    "ami": ("hf-audio/esb-datasets-test-only-sorted", "ami", "test", "text", "id"),
+    "librispeech_clean": ("librispeech", "test.clean", "text", "id"),
+    "librispeech_other": ("librispeech", "test.other", "text", "id"),
+    "voxpopuli": ("voxpopuli", "test", "text", "id"),
+    "tedlium": ("tedlium", "test", "text", "id"),
+    "gigaspeech": ("gigaspeech", "test", "text", "id"),
+    "spgispeech": ("spgispeech", "test", "text", "id"),
+    "earnings22": ("earnings22", "test", "text", "id"),
+    "ami": ("ami", "test", "text", "id"),
 }
 
 
-def save_audio_and_format_entry(
-    entry, dataset_name, audio_dir, sample_idx, text_field="text", id_field="id", with_audio=True
-):
-    """Format a dataset entry and optionally save audio file."""
-    text = entry[text_field].strip()
+def extract_audio(audio_info):
+    """Extract audio array and sampling rate from a HF dataset audio entry.
 
-    system_message = {"role": "system", "content": SYSTEM_MESSAGE}
-    user_message = {"role": "user", "content": "Transcribe the following audio."}
+    Handles both the legacy dict format ({"array": ..., "sampling_rate": ...})
+    and the newer AudioDecoder object from torchcodec-based datasets library.
+    """
+    if audio_info is None:
+        return None, None
+    try:
+        audio_array = np.array(audio_info["array"])
+        sampling_rate = int(audio_info["sampling_rate"])
+        return audio_array, sampling_rate
+    except (KeyError, TypeError, IndexError):
+        return None, None
+
+
+def format_entry(entry, dataset_name, audio_dir, text_field, id_field, with_audio):
+    """Format a dataset entry into JSONL and optionally save the audio file."""
+    text = entry[text_field].strip()
+    if not text:
+        return None
 
     sample_id = str(entry[id_field]).replace("/", "_")
     audio_filename = f"{Path(sample_id).stem}.flac"
 
-    audio_info = entry.get("audio", {})
+    audio_array, sampling_rate = extract_audio(entry.get("audio"))
     duration = None
-    if isinstance(audio_info, dict) and "array" in audio_info and "sampling_rate" in audio_info:
-        audio_array = audio_info["array"]
-        sampling_rate = audio_info["sampling_rate"]
-        duration = len(audio_array) / sampling_rate
-
-        if duration < MIN_AUDIO_DURATION:
-            return None
 
+    if audio_array is not None and sampling_rate is not None:
+        duration = len(audio_array) / sampling_rate
         if with_audio:
             sf.write(str(audio_dir / audio_filename), audio_array, sampling_rate)
 
+    user_message = {"role": "user", "content": "Transcribe the following audio."}
     audio_meta = {"path": f"/dataset/asr-leaderboard/data/{dataset_name}/{audio_filename}"}
     if duration is not None:
         audio_meta["duration"] = float(duration)
     user_message["audio"] = audio_meta
 
-    formatted_entry = {
+    formatted = {
         "task_type": "ASR",
         "expected_answer": text,
-        "messages": [system_message, user_message],
+        "messages": [{"role": "system", "content": SYSTEM_MESSAGE}, user_message],
         "subset_for_metrics": dataset_name,
+        "id": entry[id_field],
     }
-
-    formatted_entry["id"] = entry[id_field]
     if "speaker_id" in entry:
-        formatted_entry["speaker_id"] = entry["speaker_id"]
+        formatted["speaker_id"] = entry["speaker_id"]
 
-    return formatted_entry
+    return formatted
 
 
 def prepare_dataset(dataset_name, output_dir, with_audio=True):
-    """Prepare a single ASR dataset."""
+    """Download, decode, and write a single ASR dataset to JSONL + audio files."""
     if dataset_name not in DATASET_CONFIGS:
         raise ValueError(f"Unknown dataset: {dataset_name}. Available: {list(DATASET_CONFIGS.keys())}")
 
-    hf_repo, hf_config, hf_split, text_field, id_field = DATASET_CONFIGS[dataset_name]
+    hf_config, hf_split, text_field, id_field = DATASET_CONFIGS[dataset_name]
 
-    print(f"Loading {dataset_name} from {hf_repo} (config={hf_config}, split={hf_split})...")
-    dataset = load_dataset(hf_repo, hf_config, split=hf_split, trust_remote_code=True)
+    print(f"Loading {dataset_name} from {HF_REPO} (config={hf_config}, split={hf_split})...")
+    dataset = load_dataset(HF_REPO, hf_config, split=hf_split)
+    if with_audio and "audio" in dataset.column_names:
+        dataset = dataset.cast_column("audio", Audio(sampling_rate=AUDIO_SAMPLE_RATE))
 
     output_file = output_dir / f"{dataset_name}.jsonl"
     audio_dir = output_dir / "data" / dataset_name
 
     if with_audio:
         audio_dir.mkdir(parents=True, exist_ok=True)
-        print(f"Saving audio files to {audio_dir}")
 
     print(f"Processing {len(dataset)} samples from {dataset_name}...")
-
     count = 0
-    skipped = 0
     with open(output_file, "w", encoding="utf-8") as fout:
-        for idx, entry in enumerate(tqdm(dataset, desc=dataset_name)):
-            formatted = save_audio_and_format_entry(
-                entry, dataset_name, audio_dir, idx, text_field=text_field, id_field=id_field, with_audio=with_audio
-            )
+        for entry in tqdm(dataset, desc=dataset_name):
+            formatted = format_entry(entry, dataset_name, audio_dir, text_field, id_field, with_audio)
             if formatted is None:
-                skipped += 1
                 continue
-            if formatted["expected_answer"]:
-                fout.write(json.dumps(formatted) + "\n")
-                count += 1
-
-    if skipped > 0:
-        print(f"Skipped {skipped} samples with audio < {MIN_AUDIO_DURATION}s")
+            fout.write(json.dumps(formatted) + "\n")
+            count += 1
 
     print(f"Saved {count} samples to {output_file}")
     return count
@@ -157,25 +158,19 @@ def main():
     output_dir.mkdir(parents=True, exist_ok=True)
 
     with_audio = not args.no_audio
-
-    if args.no_audio:
+    if not with_audio:
         print("Running without saving audio files.")
-    else:
-        print("Running with audio. Saving to data/{dataset}/")
 
     datasets_to_prepare = list(DATASET_CONFIGS.keys()) if "all" in args.datasets else args.datasets
 
     total_samples = 0
     for dataset_name in datasets_to_prepare:
         total_samples += prepare_dataset(dataset_name, output_dir, with_audio=with_audio)
 
-    # Combine all dataset JSONLs into test.jsonl
     combined_file = output_dir / "test.jsonl"
     print(f"\nCreating combined file: {combined_file}")
 
-    all_jsonl_files = sorted(output_dir.glob("*.jsonl"))
-    dataset_files = [f for f in all_jsonl_files if f.name != "test.jsonl"]
-
+    dataset_files = sorted(f for f in output_dir.glob("*.jsonl") if f.name != "test.jsonl")
     combined_count = 0
     with open(combined_file, "w", encoding="utf-8") as fout:
         for dataset_file in dataset_files:

nemo_skills/dataset/covost2/__init__.py

@@ -13,6 +13,10 @@
 # limitations under the License.
 
 REQUIRES_DATA_DIR = True
-METRICS_TYPE = "audio"
-EVAL_ARGS = "++eval_type=audio ++eval_config.normalization_mode=multilingual"
-GENERATION_ARGS = "++prompt_format=openai ++enable_audio=true"
+IS_BENCHMARK_GROUP = True
+SCORE_MODULE = "nemo_skills.dataset.fleurs.audio_score"
+
+BENCHMARKS = {
+    "covost2.asr": {},
+    "covost2.st": {},
+}

nemo_skills/dataset/covost2/asr/__init__.py

@@ -0,0 +1,17 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+METRICS_TYPE = "audio"
+EVAL_ARGS = "++eval_type=audio ++eval_config.normalization_mode=multilingual"
+GENERATION_ARGS = "++prompt_format=openai ++enable_audio=true"