Issue with transcribing audio as a list when using timestamps

[Filip858299]

I ran into an issue when transcribing audio provided as a list using a model with timestamps. I get the following error:
(model: nvidia/canary-1b-v2)

AttributeError: 'tuple' object has no attribute 'audio'
    audio=[audio.squeeze()[:audio_len] for audio, audio_len in zip(batch.audio, batch.audio_lens)]

I’m wondering how it’s possible to use the model with timestamps for this type of audio input - numpy/torch array.

Has anyone encountered this problem or has experience handling this scenario?

in EncDecMultiTaskModel._transcribe_output_processing(self, outputs, trcfg)
   1045 del enc_states, enc_mask, decoder_input_ids
   1047 if trcfg.timestamps and self.timestamps_asr_model is not None:
   1048     hypotheses = get_forced_aligned_timestamps_with_external_model(
-> 1049         audio=[audio.squeeze()[:audio_len] for audio, audio_len in zip(batch.audio, batch.audio_lens)],
   1050         batch_size=len(batch.audio),
   1051         external_ctc_model=self.timestamps_asr_model,
   1052         main_model_predictions=hypotheses,
   1053         timestamp_type='char' if merge_to_be_done else ['word', 'segment'],
   1054         viterbi_device=trcfg._internal.device,
   1055     )
   1056 elif trcfg.timestamps:
   1057     hypotheses = process_aed_timestamp_outputs(
   1058         hypotheses, self.encoder.subsampling_factor, self.cfg['preprocessor']['window_stride']
   1059     )

AttributeError: 'tuple' object has no attribute 'audio'

my code:
import io
import numpy as np
import soundfile as sf
import librosa  # potrebné na resampling

# ==============================
# LOAD MODEL
# ==============================
from nemo.collections.asr.models import EncDecMultiTaskModel
import torch
with open(mp3_path, "rb") as f:
    mp3_bytes = f.read()

data, sr_orig = sf.read(io.BytesIO(mp3_bytes))
target_sr = 16000 


ch1 = librosa.resample(data, orig_sr=sr_orig, target_sr=target_sr)

device = "cuda" if torch.cuda.is_available() else "cpu"
print("Using device:", device)

main_model = EncDecMultiTaskModel.restore_from(path_main_model, map_location=device)
main_model = main_model.to(device)
res = main_model.transcribe(ch2, timestamps=True, source_lang='sk', target_lang='sk')

[AaronZ345]

This was a bug in the tensor-input path, not a problem with your NumPy audio.

Older code assumed every timestamp batch was a PromptedAudioToTextMiniBatch and accessed batch.audio and batch.audio_lens. Audio arrays use the external dataloader path, where the batch is a tuple, which produced the exception you saw. It was fixed in #15291: the current code reads batch[0] and batch[1] for tuple batches, and there is now a regression test for a list of audio tensors with timestamps=True.

Upgrade to NeMo 2.7.0 or newer, then pass a list of one-dimensional float tensors:

audio = torch.from_numpy(ch1).float()
result = main_model.transcribe(
    [audio],
    batch_size=1,
    timestamps=True,
    source_lang="sk",
    target_lang="sk",
)

The input must already be mono at the model's expected sample rate, which is 16 kHz for Canary. If you cannot upgrade, passing a WAV filepath avoids this specific old tuple-batch bug.