Skip to content

Add MiniMax-M3 NVFP4 B300 single-node vLLM benchmark (EAGLE3 spec decode) #1929

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Ankur-singh wants to merge 2 commits into
base: main
Choose a base branch
from
+172 −0
Open

Add MiniMax-M3 NVFP4 B300 single-node vLLM benchmark (EAGLE3 spec decode) #1929

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e0d970e
Add MiniMax-M3 NVFP4 B300 single-node vLLM benchmark with EAGLE3 spec…
Ankur-singh Jun 25, 2026
ec78fc5
Update perf-changelog pr-link for #1929
Ankur-singh Jun 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .github/configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12295,6 +12295,43 @@ minimaxm3-fp8-b300-vllm:
- { tp: 4, ep: 4, dp-attn: true, conc-start: 64, conc-end: 128 }
- { tp: 8, ep: 8, dp-attn: true, conc-start: 128, conc-end: 512 }

# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of MiniMax-M3 NVFP4
# (nvidia/MiniMax-M3-NVFP4) B300 single-node vLLM, pairing the target with the
# Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens). The benchmark
# script overlays vllm-project/vllm PR #46380 (MiniMax-M3 modelopt NVFP4
# support, commit 6c08558) before serve and routes prompts through the chat
# template. Target weights are pre-staged read-only at
# /scratch/models/MiniMax-M3-NVFP4 (added to the STAGED_MODELS allow-list in
# launch_b300-nv.sh); the EAGLE3 draft is downloaded to the writable models dir.
minimaxm3-fp4-b300-vllm-mtp:
image: vllm/vllm-openai:vllm-minimax-m3-perf-x86_64-13.0.1-7a67223
model: nvidia/MiniMax-M3-NVFP4
model-prefix: minimaxm3
runner: b300
precision: fp4
framework: vllm
multinode: false
scenarios:
fixed-seq-len:
- isl: 1024
osl: 1024
search-space:
- { tp: 8, conc-start: 1, conc-end: 64, spec-decoding: mtp }
- { tp: 8, ep: 8, conc-start: 1, conc-end: 256, spec-decoding: mtp }
- { tp: 4, conc-start: 1, conc-end: 64, spec-decoding: mtp }
- { tp: 4, ep: 4, conc-start: 64, conc-end: 256, spec-decoding: mtp }
- { tp: 4, ep: 4, dp-attn: true, conc-start: 128, conc-end: 512, spec-decoding: mtp }
- { tp: 8, ep: 8, dp-attn: true, conc-start: 256, conc-end: 512, spec-decoding: mtp }
- isl: 8192
osl: 1024
search-space:
- { tp: 8, conc-start: 1, conc-end: 64, spec-decoding: mtp }
- { tp: 8, ep: 8, conc-start: 1, conc-end: 256, spec-decoding: mtp }
- { tp: 4, conc-start: 1, conc-end: 64, spec-decoding: mtp }
- { tp: 4, ep: 4, conc-start: 64, conc-end: 256, spec-decoding: mtp }
- { tp: 4, ep: 4, dp-attn: true, conc-start: 64, conc-end: 128, spec-decoding: mtp }
- { tp: 8, ep: 8, dp-attn: true, conc-start: 128, conc-end: 256, spec-decoding: mtp }

# MiniMax-M3 day-zero (https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3).
# 427B total / 26B active MoE with MSA sparse attention; MXFP8 checkpoint
# (MiniMaxAI/MiniMax-M3-MXFP8, ~444 GB) quantized by NVIDIA — native MX tensor
Expand Down
125 changes: 125 additions & 0 deletions benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_b300_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
#!/usr/bin/env bash

# MiniMax-M3 NVFP4 B300 single-node vLLM recipe with EAGLE3 speculative
# decoding — same shape as minimaxm3_fp8_b300_mtp.sh but uses the
# nvidia/MiniMax-M3-NVFP4 checkpoint. Applies vllm-project/vllm PR #46380
# (MiniMax-M3 modelopt NVFP4 support) from commit 6c08558 by overwriting the
# 3 changed files in the installed vLLM package before the server starts.

source "$(dirname "$0")/../../benchmark_lib.sh"

check_env_vars \
MODEL \
TP \
EP_SIZE \
DP_ATTENTION \
CONC \
ISL \
OSL \
MAX_MODEL_LEN \
RANDOM_RANGE_RATIO \
RESULT_FILENAME

# Apply vllm-project/vllm PR #46380 (Add MiniMax-M3 modelopt NVFP4 support, commit 6c08558).
# This patch is required for nvidia/MiniMax-M3-NVFP4: without it vLLM does not
# recognise the NVFP4 quant config and falls back to an unsupported path.
VLLM_DIR=$(python3 -c "import vllm, os; print(os.path.dirname(vllm.__file__))")
for f in \
model_executor/layers/fused_moe/experts/trtllm_nvfp4_moe.py \
model_executor/layers/quantization/modelopt.py \
model_executor/layers/quantization/utils/flashinfer_utils.py
do
curl -fsSL "https://raw.githubusercontent.com/vllm-project/vllm/6c08558/vllm/${f}" -o "${VLLM_DIR}/${f}"

Check warning on line 32 in benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_b300_mtp.sh

View check run for this annotation

Claude / Claude Code Review

vLLM patch curl loop silently swallows failures for 2 of 3 files 

The 3-file vLLM patch overlay loop (lines 25-32) runs `curl -fsSL ... -o` without `set -e` and without `|| exit 1`, so a transient 5xx / rate-limit / commit-reachability failure on file #2 (`modelopt.py`) or file #3 (`flashinfer_utils.py`) returns non-zero and the loop continues silently. The post-patch `python3 -c` only imports `TrtLlmNvFp4ExpertsModular` from file #1, so a partial patch is undetected and `vllm serve` boots on the comment's "unsupported path". Match the sister recipe `minimaxm3
Comment on lines +25 to +32

@claude claude Bot Jun 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The 3-file vLLM patch overlay loop (lines 25-32) runs curl -fsSL ... -o without set -e and without || exit 1, so a transient 5xx / rate-limit / commit-reachability failure on file #2 (modelopt.py) or file #3 (flashinfer_utils.py) returns non-zero and the loop continues silently. The post-patch python3 -c only imports TrtLlmNvFp4ExpertsModular from file #1, so a partial patch is undetected and vllm serve boots on the comment's "unsupported path". Match the sister recipe minimaxm3_fp8_b300_mtp.sh (which wraps its patch step with || { echo ...; exit 1; }) — either add || exit 1 to the curl, or assert all three modules import in the verification.

Extended reasoning...

The defect. Lines 25-32 fetch three vLLM source files from a pinned vllm-project/vllm commit and overwrite them in the installed package:

for f in \
  model_executor/layers/fused_moe/experts/trtllm_nvfp4_moe.py \
  model_executor/layers/quantization/modelopt.py \
  model_executor/layers/quantization/utils/flashinfer_utils.py
do
  curl -fsSL "https://raw.githubusercontent.com/vllm-project/vllm/6c08558/vllm/${f}" -o "${VLLM_DIR}/${f}"
done
python3 -c "from vllm.model_executor.layers.fused_moe.experts.trtllm_nvfp4_moe import TrtLlmNvFp4ExpertsModular; print('[nvfp4-patch] OK')"

The script has no set -e (set -x later is just command tracing) and no || exit 1 inside the loop. With curl -f, an HTTP 4xx/5xx exits curl with status 22 and leaves the target file untouched — the original installed copy stays in place.

Why the validation doesn't catch it. The post-loop python3 -c imports only TrtLlmNvFp4ExpertsModular from trtllm_nvfp4_moe.py (file #1). If file #2 (modelopt.py) or file #3 (flashinfer_utils.py) fails to download, that import still succeeds, the [nvfp4-patch] OK line prints, and the script proceeds to vllm serve with two unpatched modules. Per the script's own preamble: "without it vLLM does not recognise the NVFP4 quant config and falls back to an unsupported path" — exactly the failure modeled by files #2/#3 being unpatched.

Step-by-step proof.

  1. curl -fsSL for trtllm_nvfp4_moe.py succeeds → file [NVIDIA] Add TRT-LLM 70B FP8 via slurm #1 overwritten.
  2. curl -fsSL for modelopt.py hits a transient 503 from raw.githubusercontent.com → curl exits 22, no write to ${VLLM_DIR}/.../modelopt.py, original vLLM copy retained.
  3. Bash loop sees no set -e, no || exit 1 → continues to file [NVIDIA] update vllm b200 image. TODO: add logic for docker runner. #3.
  4. curl -fsSL for flashinfer_utils.py succeeds → file [NVIDIA] update vllm b200 image. TODO: add logic for docker runner. #3 overwritten.
  5. python3 -c imports from trtllm_nvfp4_moe (the file that WAS patched). Import succeeds, prints [nvfp4-patch] OK.
  6. vllm serve boots with the new trtllm_nvfp4_moe.py calling into an unpatched modelopt.py → NVFP4 quant config not recognized, fallback path triggers, benchmark fails opaquely at serve/inference time instead of at the patch step.

Why this is real. Verified that the script contains no set -[eE] or set -o errexit; the only set -e in benchmark_lib.sh is scoped to run_agentic_replay_and_write_outputs and doesn't propagate to callers. Verified curl 8.x behavior: -f -o file against a 404/5xx exits non-zero without writing the file. Pinned commit hashes on a stable CDN make this uncommon, but not zero — raw.githubusercontent.com does have transient 5xx windows.

Repo convention. The sister script benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_b300_mtp.sh:33 already gates its patch step with python3 - <<PYEOF || { echo ...; exit 1; } — explicit fail-fast. This recipe should match.

Fix. Minimal:

curl -fsSL "https://raw.githubusercontent.com/vllm-project/vllm/6c08558/vllm/${f}" -o "${VLLM_DIR}/${f}" || exit 1

Or assert the other two modules in the verification:

python3 -c "from vllm.model_executor.layers.fused_moe.experts.trtllm_nvfp4_moe import TrtLlmNvFp4ExpertsModular; from vllm.model_executor.layers.quantization import modelopt; from vllm.model_executor.layers.quantization.utils import flashinfer_utils; print('[nvfp4-patch] OK')"

(The second form only catches import-breaking download failures; the first is the strictly safer fix and matches the fp8 twin.)

done
python3 -c "from vllm.model_executor.layers.fused_moe.experts.trtllm_nvfp4_moe import TrtLlmNvFp4ExpertsModular; print('[nvfp4-patch] OK')"

DRAFT_MODEL="Inferact/MiniMax-M3-EAGLE3"

# The target weights are launched from MODEL_PATH (the b300 launcher points it
# at the pre-staged read-only /scratch/models/MiniMax-M3-NVFP4). The EAGLE3
# draft is not pre-staged and must be downloaded, so it cannot live next to the
# read-only target — fetch it into the writable models dir (/data/models)
# instead. When MODEL_PATH is unset (stand-alone runs) fall back to the HF cache.
if [[ -n "${MODEL_PATH:-}" ]]; then
if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then
hf download "$MODEL" --local-dir "$MODEL_PATH"
fi
DRAFT_MODEL_PATH="/data/models/${DRAFT_MODEL##*/}"
if [[ ! -d "$DRAFT_MODEL_PATH" || -z "$(ls -A "$DRAFT_MODEL_PATH" 2>/dev/null)" ]]; then
hf download "$DRAFT_MODEL" --local-dir "$DRAFT_MODEL_PATH"
fi
else
hf download "$MODEL"
export MODEL_PATH="$MODEL"
hf download "$DRAFT_MODEL"
DRAFT_MODEL_PATH="$DRAFT_MODEL"
fi

if [[ -n "$SLURM_JOB_ID" ]]; then
echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME"
fi

nvidia-smi

SERVER_LOG=/workspace/server.log

export VLLM_ENGINE_READY_TIMEOUT_S=3600
export VLLM_FLOAT32_MATMUL_PRECISION=high

if [ "${DP_ATTENTION}" = "true" ]; then
PARALLEL_ARGS="--tensor-parallel-size=1 --data-parallel-size=$TP --enable-expert-parallel"
elif [ "$EP_SIZE" -gt 1 ]; then
PARALLEL_ARGS="--tensor-parallel-size=$TP --enable-expert-parallel"
else
PARALLEL_ARGS="--tensor-parallel-size=$TP"
fi

# use 3 speculative tokens for all configs for now
NUM_SPEC_TOKENS=3

if [ "${EVAL_ONLY}" = "true" ]; then
setup_eval_context
MAX_MODEL_LEN="$EVAL_MAX_MODEL_LEN"
fi
start_gpu_monitor

set -x
vllm serve "$MODEL_PATH" --served-model-name "$MODEL" --host 0.0.0.0 --port $PORT \
$PARALLEL_ARGS \
--gpu-memory-utilization 0.90 \
--max-model-len $MAX_MODEL_LEN \
--block-size 128 \
--language-model-only \
--max-cudagraph-capture-size 2048 \
--max-num-batched-tokens "$((ISL * 2 ))" \
--speculative-config "{\"method\": \"eagle3\", \"model\": \"$DRAFT_MODEL_PATH\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS, \"attention_backend\": \"FLASH_ATTN\"}" \
--stream-interval 20 --no-enable-prefix-caching \
--trust-remote-code > $SERVER_LOG 2>&1 &

SERVER_PID=$!

wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID"

pip install -q datasets pandas

run_benchmark_serving \
--model "$MODEL" \
--port "$PORT" \
--backend vllm \
--input-len "$ISL" \
--output-len "$OSL" \
--random-range-ratio "$RANDOM_RANGE_RATIO" \
--num-prompts "$((CONC * 10))" \
--max-concurrency "$CONC" \
--result-filename "$RESULT_FILENAME" \
--result-dir /workspace/ \
--trust-remote-code \
--use-chat-template

if [ "${RUN_EVAL}" = "true" ]; then
run_eval --framework lm-eval --port "$PORT"
append_lm_eval_summary
fi

stop_gpu_monitor
set +x
9 changes: 9 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4183,3 +4183,12 @@
- "server_atom.sh: fix _MAX_CONC assignment before cudagraph size check; gate ATOM_MOE_GU_ITLV/AITER_BF16_FP8_MOE_BOUND on DeepSeek-V4-Pro only"
- "Search space: ISL=8192 and ISL=1024, 1P1D TP4, conc 1-512"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1927

- config-keys:
- minimaxm3-fp4-b300-vllm-mtp
description:
- "Add MiniMax-M3 NVFP4 (nvidia/MiniMax-M3-NVFP4) B300 single-node aggregated vLLM benchmark with EAGLE3 speculative decoding (spec-decoding: mtp, 3 draft tokens via Inferact/MiniMax-M3-EAGLE3)"
- "Image vllm/vllm-openai:vllm-minimax-m3-perf-x86_64-13.0.1-7a67223; benchmark script overlays vllm-project/vllm PR #46380 (MiniMax-M3 modelopt NVFP4 support, commit 6c08558) before serve; prompts routed through the chat template"
- "Target weights pre-staged read-only at /scratch/models/MiniMax-M3-NVFP4 (added MiniMax-M3-NVFP4 to launch_b300-nv.sh STAGED_MODELS); EAGLE3 draft downloaded to the writable /data/models; --block-size 128 (MSA), --language-model-only"
- "Sweeps tp 4/8 with and without EP and dp-attn at 1k1k and 8k1k, conc 1-512"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX

Check warning on line 4194 in perf-changelog.yaml

View check run for this annotation

Claude / Claude Code Review

Placeholder PR link XXX in perf-changelog 

The perf-changelog entry for `minimaxm3-fp4-b300-vllm-mtp` ends with `pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX` — the literal `XXX` placeholder was never filled in. Every other recent entry links to a real PR number (e.g. `/pull/1927` directly above); this should be `/pull/1929` to match this PR.

@claude claude Bot Jun 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The perf-changelog entry for minimaxm3-fp4-b300-vllm-mtp ends with pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX — the literal XXX placeholder was never filled in. Every other recent entry links to a real PR number (e.g. /pull/1927 directly above); this should be /pull/1929 to match this PR.

Extended reasoning...

What the bug is. perf-changelog.yaml line 4194 was added as part of this PR with the value pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX. The XXX is a literal placeholder string that the author clearly intended to replace with the actual PR number before pushing, but didn't. Every other recent entry in the file links to a real PR — the immediately preceding entry uses /pull/1927, and earlier ones use /pull/1762, /pull/1865, etc. This is a documentation/metadata oversight.\n\nHow it manifests. Anyone clicking the changelog link for the minimaxm3-fp4-b300-vllm-mtp entry after merge will hit GitHub's 404 page for /pull/XXX rather than landing on the PR that introduced the config. Tooling that walks the changelog looking up PRs (e.g. cross-referencing config additions back to their PR for release notes) will fail to resolve the placeholder.\n\nThe specific code path. In the diff applied to perf-changelog.yaml, the new YAML block ending at line 4194 contains:\n\nyaml\n- config-keys:\n - minimaxm3-fp4-b300-vllm-mtp\n description:\n - "Add MiniMax-M3 NVFP4 ..."\n ...\n pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX\n\n\nThe XXX is verbatim text, not a template variable that any downstream tool would substitute.\n\nWhy existing code doesn't prevent it. The changelog is hand-edited YAML; there is no schema validator or CI step that pattern-matches pr-link against /pull/\d+ to catch placeholders. The PR number isn't knowable at commit time (you only learn it after opening the PR), so it has to be filled in as an amendment — exactly the step that was skipped here.\n\nStep-by-step proof.\n1. Open perf-changelog.yaml after this PR merges.\n2. Scroll to the last entry — config minimaxm3-fp4-b300-vllm-mtp.\n3. Read line 4194: pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX.\n4. Click / curl the link → GitHub returns 404 (/pull/XXX is not a valid PR id).\n5. Compare with line 4185 directly above: pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1927 — resolves correctly.\n\nHow to fix. Replace XXX with 1929 on line 4194 so the entry reads pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1929, matching this PR's number.

1 change: 1 addition & 0 deletions runners/launch_b300-nv.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -362,6 +362,7 @@ else
MiniMax-M2.7
MiniMax-M2.7-NVFP4
MiniMax-M3
MiniMax-M3-NVFP4
Qwen3.5-397B-A17B
Qwen3.5-397B-A17B-FP8
Qwen3.5-397B-A17B-NVFP4
Expand Down