[Feature][ROCm] Support WideTP and WideEP P/D topologies on AMD GPUs

[chaeminlim-mb]

Summary

AMD users need documented, supported P/D disaggregated-serving configurations for wider ROCm topologies:

• WideTP: tensor parallelism spanning multiple hosts, for example TP16-style prefill/decode instances.
• WideEP: DP/EP-style expert-parallel serving spanning multiple hosts, for example DP16/EP16 P/D instances.

This is a feature/support issue, not a pass announcement. The current evidence is mixed: smaller TP8/DP8EP controls are clean, WideTP has preliminary positive evidence that still needs final-stack validation, and WideEP high-concurrency MTP workloads still have blockers.

Your current environment

Output of python -m vllm.collect_env (one MI300X node; internal IPs/NICs redacted)

==============================
        System Info
==============================
OS                           : Ubuntu 24.04.4 LTS (x86_64)
GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version                : 22.0.0git (https://github.com/RadeonOpenCompute/llvm-project roc-7.2.2 26084 f58b06dce1f9c15707c5f808fd002e18c2accf7e)
CMake version                : version 3.31.10
Libc version                 : glibc-2.39

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0+git8514f05
Is debug build               : False
CUDA used to build PyTorch   : N/A
ROCM used to build PyTorch   : 7.2.53211
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0] (64-bit runtime)
Python platform              : Linux-6.8.0-110-generic-x86_64-with-glibc2.39

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : Could not collect
CUDA_MODULE_LOADING set to   :
GPU models and configuration : AMD Instinct MI300X (gfx942:sramecc+:xnack-)
Nvidia driver version        : Could not collect
cuDNN version                : Could not collect
HIP runtime version          : 7.2.53211
MIOpen runtime version       : 3.5.1
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           46 bits physical, 57 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  112
On-line CPU(s) list:                     0-111
Vendor ID:                               GenuineIntel
BIOS Vendor ID:                          Intel
Model name:                              INTEL(R) XEON(R) PLATINUM 8570
BIOS Model name:                         INTEL(R) XEON(R) PLATINUM 8570  CPU @ 2.1GHz
BIOS CPU family:                         179
CPU family:                              6
Model:                                   207
Thread(s) per core:                      1
Core(s) per socket:                      56
Socket(s):                               2
Stepping:                                2
CPU(s) scaling MHz:                      46%
CPU max MHz:                             4000.0000
CPU min MHz:                             800.0000
BogoMIPS:                                4200.00
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect user_shstk avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities ibpb_exit_to_user
L1d cache:                               5.3 MiB (112 instances)
L1i cache:                               3.5 MiB (112 instances)
L2 cache:                                224 MiB (112 instances)
L3 cache:                                600 MiB (2 instances)
NUMA node(s):                            2
NUMA node0 CPU(s):                       0-55
NUMA node1 CPU(s):                       56-111
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace

==============================
Versions of relevant libraries
==============================
[pip3] conch-triton-kernels==1.2.1
[pip3] numpy==2.1.3
[pip3] onnx==1.19.0
[pip3] onnx-ir==0.2.1
[pip3] onnxscript==0.7.0
[pip3] onnxslim==0.1.94
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0+git8514f05
[pip3] torchaudio==2.9.0+eaa9e4e
[pip3] torchvision==0.24.1+d801a34
[pip3] transformers==5.12.1
[pip3] triton==3.7.0
[pip3] triton_kernels==1.0.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : 7.2.53211-35e8c7bf89
vLLM Version                 : 0.1.dev0
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled
GPU Topology:
  ============================ ROCm System Management Interface ============================
================================ Weight between two GPUs =================================
       GPU0         GPU1         GPU2         GPU3         GPU4         GPU5         GPU6         GPU7
GPU0   0            15           15           15           15           15           15           15
GPU1   15           0            15           15           15           15           15           15
GPU2   15           15           0            15           15           15           15           15
GPU3   15           15           15           0            15           15           15           15
GPU4   15           15           15           15           0            15           15           15
GPU5   15           15           15           15           15           0            15           15
GPU6   15           15           15           15           15           15           0            15
GPU7   15           15           15           15           15           15           15           0

================================= Hops between two GPUs ==================================
       GPU0         GPU1         GPU2         GPU3         GPU4         GPU5         GPU6         GPU7
GPU0   0            1            1            1            1            1            1            1
GPU1   1            0            1            1            1            1            1            1
GPU2   1            1            0            1            1            1            1            1
GPU3   1            1            1            0            1            1            1            1
GPU4   1            1            1            1            0            1            1            1
GPU5   1            1            1            1            1            0            1            1
GPU6   1            1            1            1            1            1            0            1
GPU7   1            1            1            1            1            1            1            0

=============================== Link Type between two GPUs ===============================
       GPU0         GPU1         GPU2         GPU3         GPU4         GPU5         GPU6         GPU7
GPU0   0            XGMI         XGMI         XGMI         XGMI         XGMI         XGMI         XGMI
GPU1   XGMI         0            XGMI         XGMI         XGMI         XGMI         XGMI         XGMI
GPU2   XGMI         XGMI         0            XGMI         XGMI         XGMI         XGMI         XGMI
GPU3   XGMI         XGMI         XGMI         0            XGMI         XGMI         XGMI         XGMI
GPU4   XGMI         XGMI         XGMI         XGMI         0            XGMI         XGMI         XGMI
GPU5   XGMI         XGMI         XGMI         XGMI         XGMI         0            XGMI         XGMI
GPU6   XGMI         XGMI         XGMI         XGMI         XGMI         XGMI         0            XGMI
GPU7   XGMI         XGMI         XGMI         XGMI         XGMI         XGMI         XGMI         0

======================================= Numa Nodes =======================================
GPU[0]		: (Topology) Numa Node: 0
GPU[0]		: (Topology) Numa Affinity: 0
GPU[1]		: (Topology) Numa Node: 0
GPU[1]		: (Topology) Numa Affinity: 0
GPU[2]		: (Topology) Numa Node: 0
GPU[2]		: (Topology) Numa Affinity: 0
GPU[3]		: (Topology) Numa Node: 0
GPU[3]		: (Topology) Numa Affinity: 0
GPU[4]		: (Topology) Numa Node: 1
GPU[4]		: (Topology) Numa Affinity: 1
GPU[5]		: (Topology) Numa Node: 1
GPU[5]		: (Topology) Numa Affinity: 1
GPU[6]		: (Topology) Numa Node: 1
GPU[6]		: (Topology) Numa Affinity: 1
GPU[7]		: (Topology) Numa Node: 1
GPU[7]		: (Topology) Numa Affinity: 1
================================== End of ROCm SMI Log ===================================

==============================
     Environment Variables
==============================
VLLM_ROCM_USE_AITER_MLA=True
VLLM_MORIIO_CONNECTOR_READ_MODE=true
TORCHINDUCTOR_CACHE_DIR=/jit-cache/torchinductor
NCCL_IB_DISABLE=0
NCCL_IB_HCA=<redacted>
VLLM_NIXL_SIDE_CHANNEL_HOST=<redacted>
VLLM_ENABLE_V1_MULTIPROCESSING=1
VLLM_WORKER_MULTIPROC_METHOD=spawn
VLLM_ROCM_USE_AITER=1
VLLM_LOGGING_LEVEL=INFO
VLLM_CACHE_ROOT=/jit-cache/vllm
VLLM_MORIIO_NUM_WORKERS=1
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3600
VLLM_HOST_IP=<redacted>
VLLM_NIXL_SIDE_CHANNEL_PORT=5559
VLLM_ROCM_USE_AITER_MOE=True
NCCL_IB_GID_INDEX=3
VLLM_ROCM_USE_AITER_LINEAR=True
VLLM_MORIIO_QP_PER_TRANSFER=4
VLLM_MORIIO_POST_BATCH_SIZE=2
LD_LIBRARY_PATH=/app/deps/install/ucx/lib:/opt/rocm/lib:/usr/local/lib:
PYTORCH_ROCM_ARCH=gfx942
VLLM_TARGET_DEVICE=rocm
CMAKE_BUILD_TYPE=Release
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1

vLLM branch under test: chaemin/upstream-merge-20260611-synced. The MoRI/MoRIIO/AITER stack is built into the ROCm image from docker/Dockerfile.rocm.

The feature, motivation and pitch

vLLM already supports single-node-instance P/D on AMD GPUs (TP8, DP8EP). For larger DeepSeek-R1-class deployments, users need multi-node-instance support:

• TP16 instances for large-model prefill/decode throughput.
• DP16/EP16 instances for MoE-heavy workloads with expert parallelism across hosts.

This issue requests that these wider topologies become first-class supported configurations on AMD GPUs, with explicit user-facing config, documented support boundaries, and clear acceptance matrices.

Reproduction / validation matrix

A support claim should be based on explicit rungs rather than a single smoke result.

WideTP / TP16 target

• MTP1 GSM8K at the agreed concurrency.
• MTP1 1k/1k high-concurrency rungs.
• MTP1 8k/1k high-concurrency rungs including c77, c128, and c154.
• MTP3 8k/1k rungs including c77, c128, and c154.
• MTP3 GSM8K and 1k/1k coverage should be either completed or documented as an explicit limitation.

WideEP / DP16EP target

• MTP0 control: GSM8K and 1k/1k ladder.
• MTP1 1p1d: 8k/1k c128 and c154.
• MTP1 2p1d: replay past the c55 failure and cover c70/c77/c128.
• MTP1 1p2d: 8k/1k c128 with valid JSON and full completion.
• MTP3 1p1d: 8k/1k c128.

For the 8k/1k benchmark, use 8192 input tokens, 1024 output tokens, and num_prompts = 10 * concurrency.

vLLM launch flags used to exercise the matrix

Common base flags (all shapes):

--model /path/to/DeepSeek-R1-0528 \
--served-model-name DeepSeek-R1-0528 \
--dtype auto \
--trust-remote-code \
--distributed-executor-backend mp \
--kv-cache-dtype fp8_e4m3 \
--block-size 1 \
--no-enable-prefix-caching \
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}'

WideEP DP16/EP16 / 2 nodes per instance:

--tensor-parallel-size 1 \
--data-parallel-size 16 \
--data-parallel-size-local 8 \
--enable-expert-parallel \
--all2all-backend mori_high_throughput \
--moe-backend aiter \
--max-model-len 10240 \
--max-num-batched-tokens 4096 \
--gpu-memory-utilization 0.80

Multi-node DP requires --nnodes 2, --node-rank N, --master-addr <head>, --master-port <port>, and per-node --data-parallel-start-rank offsets.

WideTP TP16 / 2 nodes per instance:

--tensor-parallel-size 16 \
--max-model-len 16384 \
--max-num-batched-tokens 8192 \
--max-num-seqs 1024 \
--gpu-memory-utilization 0.72

TP16 also needs --nnodes 2 --node-rank N --master-addr <head> --master-port 29500 and matching GLOO_SOCKET_IFNAME / NCCL_SOCKET_IFNAME / TP_SOCKET_IFNAME values per host.

MTP speculative config:

--speculative-config '{"method":"deepseek_mtp","num_speculative_tokens":1}'   # MTP1
--speculative-config '{"method":"deepseek_mtp","num_speculative_tokens":3}'   # MTP3

MoRIIO READ-mode config shape:

{
  "kv_connector": "MoRIIOConnector",
  "kv_role": "kv_producer",
  "kv_connector_extra_config": {
    "read_mode": true,
    "qp_per_transfer": 4,
    "post_batch_size": 2,
    "num_workers": 1,
    "proxy_ip": "<proxy-host>",
    "proxy_port": "<zmq-port>",
    "proxy_ping_port": "<zmq-port>",
    "http_port": "<server-port>",
    "handshake_port": "<mori-handshake-port>",
    "notify_port": "<mori-notify-port>"
  }
}

For TP16 and DP16, set VLLM_MORIIO_NODE_HOSTS=<head>,<worker> per instance so each rank resolves the correct peer.

Benchmark command shape:

vllm bench serve --backend openai-chat \
  --base-url http://<proxy>:<port> \
  --endpoint /v1/chat/completions \
  --model DeepSeek-R1-0528 \
  --dataset-name random \
  --random-input-len 8192 \
  --random-output-len 1024 \
  --num-prompts $((10 * concurrency)) \
  --max-concurrency <concurrency> \
  --request-rate inf

Alternatives

• Continue using single-node TP8/DP8EP only. This avoids the multi-node complexity but caps throughput and scale for large MoE models.
• Run WideEP/WideTP via fork-only patches. The goal is to replace fork-only patches with upstream PRs.

Expected behavior

• WideTP and WideEP have clear user-facing configuration and support boundaries.
• Each supported topology completes the agreed matrix with completed == num_prompts.
• Failures produce logs and benchmark JSON suitable for triage.
• Unsupported or not-yet-validated rungs are documented explicitly instead of implied as supported.

Current behavior / evidence

• TP8 and DP8EP P/D controls are clean through the tested MTP1/MTP3 rungs.
• WideEP MTP0 control passes GSM8K and the 1k/1k ladder.
• WideTP/TP16 has preliminary high-concurrency passes, but it needs a rerun on the final upstream stack before being listed as supported without caveats.
• WideEP is not ready:
  • MTP1 1p1d 8k/1k c128 completed 1280/1280, but this is only a narrow pass.
  • MTP1 1p1d 8k/1k c154 is partial at 1491/1540.
  • MTP1 2p1d 8k/1k c55 is partial at 311/550, and higher rungs need replay.
  • MTP3 1p1d 8k/1k c128 is partial at 101/1280.
  • MTP1 1p2d 8k/1k c128 exited with rc=137 and no valid JSON.

Related PRs

• [Bugfix][ROCm][MoE] MoRI: pass num_qp_per_pe/quant_type explicitly; preserve router top-k for finalize #45225 provides MoRI dispatch configuration needed by the AMD P/D matrix.
• [Core][KV-transfer] MoRIIO: multi-node TP prefill→decode dispatch via published host list #45228 adds multi-node TP host-vector dispatch for WideTP.
• [Bugfix][KV-transfer] MoRIIO: READ-mode stability fixes (completion IDs, DP routing, drain, keepalive) #45230 adds MoRIIO READ-mode stability fixes needed before wide topologies can be reliable.
• [Bugfix][Core] DP: sync engines_running finish-state every step to prevent multi-node deadlock #45223 supports wider P/D runs by making DP finish behavior deterministic.
• [Bugfix][Core] shm_broadcast: bound idle reader waits and release read slots #45224 supports wider P/D runs by preventing indefinite shared-memory waits.
• [Bugfix] MoRIIO toy P/D proxy: add /health #45222 provides the proxy fixes used by the P/D validation harness.

AITER/MLA-specific fixes are covered separately by #45226 and #45227.

Stack order: #45230 depends on #45228; any review/rebase should wait until #45228 lands.

Caveats

• Do not use this issue to claim WideTP or WideEP are fully passing today.
• WideTP has preliminary positive evidence, but support wording should wait for a final-stack rerun that passes without caveats.
• WideEP remains blocked by the c154, 2p1d, MTP3 c128, and 1p2d failures above.
• If a subset is intended to be supported first, the issue should document that subset explicitly.

Acceptance criteria

• WideTP has documented AMD ROCm configuration and clears the agreed TP16 acceptance matrix on the final upstream PR stack.
• WideEP clears MTP0, MTP1, and MTP3 coverage at the required rungs, including the currently blocked c154, 2p1d, MTP3 c128, and 1p2d cases.
• The clean TP8, DP8EP, and WideEP MTP0 controls do not regress.
• High-concurrency failures return diagnosable logs and valid benchmark artifacts instead of silent wedges or rc=137/no-JSON outcomes.
• User-facing docs or release notes accurately state which AMD wide topologies are supported and which remain experimental or unsupported.

[github-actions]

CC @hongxiayang @tjtanaa @vllmellm for ROCm-related issue

[chaeminlim-mb]

Closing this tracking issue for now while we finish internal regression/write-up review. We can reopen or recreate it after the wording and scope are finalized.

[chaeminlim-mb]

Reopening with a rewritten, upstream-focused description. The previous body was a tracking-style placeholder with internal infra details; this version treats the issue as a feature request with a concrete support matrix, evidence, and acceptance criteria, while keeping the WideEP caveats visible.