Implement Multiturn support for EasyMagpie by Edresson · Pull Request #15764 · NVIDIA-NeMo/Speech

Edresson · 2026-06-08T12:03:42Z

What does this PR do ?

Implements multiturn support for EasyMagpieTTS, including multiturn Lhotse training/evaluation data handling, user-audio-conditioned inference, multiturn metrics/export support, and related tests.

Collection: TTS

Changelog

Added a new EasyMagpie multiturn Lhotse config: examples/tts/conf/magpietts/easy_magpietts_lhotse_multiturn.yaml.
Added support for restoring EasyMagpie models from a pretrained checkpoint before normal initialization.
Added multiturn Lhotse TTS dataset support for duplex / multi-turn conversational samples.
Added EasyMagpie multiturn user-audio inference mode via --easy_magpie_inference_mode multiturn_user_audio.
Added support for user-audio conditioning, user silence handling, user turn end-token handling, and max evaluation turns during multiturn inference.
Added distributed / multi-GPU inference support for multiturn EasyMagpie evaluation by sharding generation across ranks and merging rank-local outputs for evaluation.
Added grouped turn-level and sample-level metrics export for multiturn evaluation, including JSON/CSV outputs.
Added phoneme prediction export fields to generated .json / .csv evaluation outputs.
Added optional predicted-phoneme inference controls, including phoneme input type, phoneme sampling method, tokenizer override, and text-input dropout.
Added additional evaluation options for ASR / EOU batch sizes, emotion cosine similarity, emotion match rate, and text-annotation stripping for metrics.
Added Lhotse dataloader speaker filtering support.
Added / updated Nemotron-H decoder support needed for prefill in EasyMagpie multiturn inference.
Added unit tests for Lhotse TTS filters, MagpieTTS Lhotse dataset behavior, and EasyMagpieTTS model behavior.

Usage

Example EasyMagpie multiturn inference with a .nemo checkpoint:

# Run from the NeMo repository root.
# Replace paths with your local model, codec, evalset config, and output directory.

python examples/tts/magpietts_inference.py \
  --model_type easy_magpie \
  --easy_magpie_inference_mode multiturn_user_audio \
  --nemo_files /path/to/easy_magpie_model.nemo \
  --datasets_json_path /path/to/evalset_config.json \
  --out_dir /path/to/output \
  --codecmodel_path /path/to/codec_model.nemo \
  --batch_size 1 \
  --max_eval_turns 6 \
  --run_evaluation

copy-pr-bot · 2026-06-08T12:03:46Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Edresson · 2026-06-09T18:05:12Z

FYI, these changes was done to support prefill in one step on nemotron_h class.

Edresson · 2026-06-23T21:08:05Z

/ok to test da88283

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Signed-off-by: Shehzeen Hussain <shehzeenh@nvidia.com>

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Signed-off-by: Shehzeen Hussain <shehzeenh@nvidia.com>

Signed-off-by: Shehzeen Hussain <shehzeensh@gmail.com>

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Signed-off-by: Shehzeen Hussain <shehzeensh@gmail.com>

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

shehzeen · 2026-07-26T00:55:07Z

/ok to test 4156fc2

paarthneekhara · 2026-07-27T04:23:31Z

            if config.layers_block_type[i] == "mamba":
                self.conv_states.append(
-                    torch.zeros(batch_size, intermediate_size, conv_kernel_size, device=device, dtype=dtype)
+                    torch.zeros(batch_size, conv_dim, conv_kernel_size, device=device, dtype=dtype)


Is this the correct implementation? Seems different from what it used to be.

paarthneekhara · 2026-07-27T04:29:03Z

+                if seq_len <= 0:
+                    continue
+
+                boundary_trim = self.cfg.get("user_audio_boundary_trim", 0)


Can you add some comments on what this hyper-parameter does?

paarthneekhara · 2026-07-27T04:30:12Z

+                        trim_amount = max(1, base_trim + trim_delta)
+                        trim_amount = min(trim_amount, max(1, copy_len - 1))
+
+                        aug_choice = random.choices(


I feel like there is a lot of code for this trimming augmentation, Is this enabled during training? If not, we can drop some of this.

paarthneekhara · 2026-07-27T04:35:38Z

+                state.audio_prediction_end_idx.fill_(-1)
+                state.finished.zero_()


Why are such custom overrides needed for state? Please add some comments.

paarthneekhara · 2026-07-27T04:36:02Z

Left a few comments, but most changes made to easy_magpietts model make sense. User audio channel code could probably be simplified.

github-actions Bot added TTS common labels Jun 8, 2026

github-advanced-security AI found potential problems Jun 8, 2026

View reviewed changes

Comment thread examples/tts/easy_magpietts_inference_multiturn_runner.py Fixed

Comment thread examples/tts/easy_magpietts_inference_multiturn_runner.py Fixed

Comment thread examples/tts/easy_magpietts_inference_multiturn_runner.py Fixed

Edresson force-pushed the nemotron_tts_multiturn_user_audio_latest branch from 2600fb8 to 9cb8c01 Compare June 8, 2026 19:08

github-advanced-security AI found potential problems Jun 8, 2026

View reviewed changes

Comment thread examples/tts/magpietts_inference.py Fixed

Comment thread examples/tts/magpietts_inference.py Fixed

Comment thread examples/tts/magpietts_inference.py Fixed

Edresson force-pushed the nemotron_tts_multiturn_user_audio_latest branch from 9cb8c01 to a73132d Compare June 8, 2026 19:44

github-advanced-security AI found potential problems Jun 9, 2026

View reviewed changes

Comment thread nemo/collections/tts/data/text_to_speech_dataset_lhotse_multiturn.py Fixed

github-actions Bot removed the common label Jun 9, 2026

Edresson commented Jun 9, 2026

View reviewed changes

github-actions Bot added the common label Jun 18, 2026

github-advanced-security AI found potential problems Jun 20, 2026

View reviewed changes

Comment thread nemo/collections/tts/metrics/emotion_encoder.py Fixed

shehzeen reviewed Jun 22, 2026

View reviewed changes

Comment thread examples/tts/magpietts_inference.py Outdated

Edresson force-pushed the nemotron_tts_multiturn_user_audio_latest branch 2 times, most recently from c564930 to 1a43737 Compare June 23, 2026 15:36

github-advanced-security AI found potential problems Jun 23, 2026

View reviewed changes

Comment thread nemo/collections/tts/modules/magpietts_inference/inference.py Fixed

Comment thread nemo/collections/tts/modules/magpietts_inference/inference.py Fixed

Comment thread nemo/collections/tts/modules/magpietts_inference/inference.py Fixed

Edresson force-pushed the nemotron_tts_multiturn_user_audio_latest branch 3 times, most recently from ba6c288 to 5978897 Compare June 23, 2026 19:06

copy-pr-bot Bot temporarily deployed to public June 23, 2026 21:08 Inactive

copy-pr-bot Bot temporarily deployed to test June 23, 2026 21:10 Inactive

copy-pr-bot Bot temporarily deployed to public June 23, 2026 21:12 Inactive

copy-pr-bot Bot temporarily deployed to public June 23, 2026 21:13 Inactive

Edresson marked this pull request as ready for review June 23, 2026 21:14

Edresson requested a review from a team as a code owner June 23, 2026 21:14

Edresson requested review from blisc and paarthneekhara June 23, 2026 21:15

Edresson and others added 23 commits July 25, 2026 17:47

Apply Black

0cb8896

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Fix imports order

0b0e274

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Add unit tests

ce19f04

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Fix black check

25c5ed6

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Replace instantiate with safe_instantiate

5a88fb0

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Fix unit tests

d9952a3

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Add turn level volume normalization

57976f3

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

remove configurable cas encoder layers since it is not needed

0b3eea3

Signed-off-by: Shehzeen Hussain <shehzeenh@nvidia.com>

Left silence pad short user audios for safety

d32fbd3

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

cfg unconditional inference fix user channel

6c06d36

Signed-off-by: Shehzeen Hussain <shehzeenh@nvidia.com>

preserve language texts in manifests

29b1e31

Signed-off-by: Shehzeen Hussain <shehzeenh@nvidia.com>

docstring and setup tokenizer function reused

8a9e0f8

Signed-off-by: Shehzeen Hussain <shehzeensh@gmail.com>

Add missing docstring

7a709e1

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Fix streaming_prefill method name

7595664

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Add missing docstring

b339f25

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Fix merge issues

1ae3cdd

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Add missing docstring

73726b1

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Drop user audio conditioning on CFG

7f415fb

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

text context support

dce69f2

Signed-off-by: Shehzeen Hussain <shehzeensh@gmail.com>

Refactor text_to_speech_dataset_lhotse_multiturn.py

c375d26

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Move helpers out of magpietts_inference.py inference example

aa314d4

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Fix black formatting

52f3854

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

Fix Liint checks

4156fc2

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

shehzeen force-pushed the nemotron_tts_multiturn_user_audio_latest branch from 3276aa3 to 4156fc2 Compare July 26, 2026 00:54

paarthneekhara reviewed Jul 27, 2026

View reviewed changes

		state.audio_prediction_end_idx.fill_(-1)
		state.finished.zero_()

Uh oh!

Conversation

Edresson commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Changelog

Usage

Uh oh!

copy-pr-bot Bot commented Jun 8, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Edresson Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Edresson commented Jun 23, 2026

Uh oh!

shehzeen commented Jul 26, 2026

Uh oh!

paarthneekhara Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

paarthneekhara Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

paarthneekhara Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

paarthneekhara Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

paarthneekhara commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Edresson commented Jun 8, 2026 •

edited

Loading