Skip to content

Temporarily pin SALM Automodel dispatcher to torch#15963

Merged
DongjiGao merged 1 commit into
mainfrom
agent/pin-salm-torch-dispatcher
Jul 25, 2026
Merged

Temporarily pin SALM Automodel dispatcher to torch#15963
DongjiGao merged 1 commit into
mainfrom
agent/pin-salm-torch-dispatcher

Conversation

@DongjiGao

Copy link
Copy Markdown
Contributor

What changed

  • Pin examples/speechlm2/conf/salm_automodel.yaml to automodel_backend.dispatcher: torch.
  • Document DeepEP as an explicit opt-in for GPUs with NVLINK/NVSHMEM.
  • Add a focused config regression test.
  • Add TODO (Dongji) to remove the override once Automodel performs topology-aware dispatcher selection.

Why

This is a temporary fix to unblock the release.

Automodel's BackendConfig currently selects DeepEP when the package is installed and CUDA is available. It does not validate whether the GPU topology supports DeepEP. On PCIe-only systems, SALM MoE expert-parallel training can therefore enter DeepEP kernels and fail with an unexplained CUDA unspecified launch failure / EPException.

The long-term fix belongs in Automodel: select DeepEP only when the required NVLINK/NVSHMEM topology is available, otherwise fall back to Torch with a warning.

Impact

The example now uses the portable Torch dispatcher by default. Systems with compatible NVLINK/NVSHMEM topology can opt into DeepEP with:

model.automodel_backend.dispatcher=deepep

Validation

Using nvcr.io/nvidian/nemo-speech:26.06.rc4 on a two-GPU PCIe-only host:

  • Default BackendConfig resolved to dispatcher=deepep.
  • Automodel DeepEP dispatch reproduced intranode.cu:126 'unspecified launch failure' and deep_ep.cpp:147 'unspecified launch failure', followed by EPException / SIGABRT.
  • The portable Automodel Torch GroupedExperts forward/backward path passed on both ranks.
  • The patched YAML resolved to dispatcher=torch.
  • Focused config test passed: pytest -q --confcutdir=tests/collections/speechlm2 tests/collections/speechlm2/test_example_configs.py.
  • git diff --check, Black, isort, and Python compilation passed.

The original 8x H200 QA case should still be rerun in the release environment.

@copy-pr-bot

copy-pr-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@DongjiGao
DongjiGao requested review from KunalDhawan, karpnv, pzelasko and tango4j and removed request for karpnv July 24, 2026 20:23
@KunalDhawan

Copy link
Copy Markdown
Collaborator

/ok to test fe8310c

Signed-off-by: Dongji Gao <dongjig@nvidia.com>
@DongjiGao
DongjiGao force-pushed the agent/pin-salm-torch-dispatcher branch from fe8310c to 9ac7aa6 Compare July 24, 2026 21:47
@DongjiGao
DongjiGao marked this pull request as ready for review July 24, 2026 22:12
@DongjiGao

Copy link
Copy Markdown
Contributor Author

/ok to test 9ac7aa6

@DongjiGao
DongjiGao enabled auto-merge (squash) July 24, 2026 22:58
@KunalDhawan KunalDhawan added the r3.0.0 Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge. label Jul 24, 2026
@DongjiGao
DongjiGao disabled auto-merge July 24, 2026 23:01
@github-actions

Copy link
Copy Markdown
Contributor

[🤖]: Hi @DongjiGao 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully.

So it might be time to merge this PR or get some approvals.

@DongjiGao
DongjiGao merged commit 9b9733d into main Jul 25, 2026
234 of 238 checks passed
@DongjiGao
DongjiGao deleted the agent/pin-salm-torch-dispatcher branch July 25, 2026 02:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

r3.0.0 Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge. Run CICD

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants