[ROCm] Tests confirmed skipped due to num_devices/num_gpus mismatch (nightly-log-verified)

[wjabbour]

Title

[ROCm] Tests confirmed skipped due to num_devices/num_gpus mismatch (nightly-log-verified)

Body

Per the ask on #50490 / #34994: list of tests confirmed skipped in the AMD nightly (vllm/amd-ci, "Full CI Run - nightly"), verified against real logs rather than inferred from config. All links below are to build #12360 (2026-08-25) — nightly builds rotate daily, so these specific job links will go stale; re-run against whatever's current if this sits for a while.

Root cause for all of these: the job's num_devices: (Buildkite provisioning) is lower than the test's own declared num_gpus requirement (multi_gpu_test/multi_gpu_marks/gpu_tier_mark), so the test is collected, attempted, and immediately self-skips every run. Shows green, never trips CI red.

Under-provisioned (job exists, doesn't give it enough GPUs)

Test	Job	Confirmed skip
tests/distributed/test_comm_ops.py (needs 4, has 2)	:amd: (MI300) Distributed Comm Ops	SKIPPED (Need at least 4 GPUs to run the test.) — 5 cases
tests/kernels/mamba/test_mamba_mixer2.py (needs 2, has 1)	:amd: (MI300) Mamba Kernels	SKIPPED (Need at least 2 GPUs to run the test.) — 3 cases
tests/plugins_tests/gguf/test_gguf_plugin_generate.py (needs 2, has 1)	:amd: (MI300) GGUF Plugin	SKIPPED (Need at least 2 GPUs to run the test.) — 2 cases
tests/lora/test_fused_moe_lora_kernel.py (needs 2, has 1)	:amd: (MI300) LoRA Shard 1	SKIPPED (Need at least 2 GPUs to run the test.) — 13 cases across shards
tests/lora/test_minicpmv_tp.py (needs 4, has 1)	:amd: (MI300) LoRA Shard 1	SKIPPED (Need at least 4 GPUs to run the test.)
tests/lora/test_qwen36_moe_lora.py (needs 2 and 4, has 1)	:amd: (MI300) LoRA Shard 1	SKIPPED (Need at least 2/4 GPUs to run the test.)
tests/lora/test_transformers_model.py (needs 4, has 1)	:amd: (MI300) LoRA Shard 2	SKIPPED (Need at least 4 GPUs to run the test.)
tests/models/multimodal/generation/test_common.py (llama4 case, needs 4, has 1)	:amd: (MI300) Multimodal Models (Extended Generation 3)	SKIPPED (Need at least 4 GPUs to run the test.) — 2 cases

Not run by any job at all

No job in the entire nightly build (#12360, 242 jobs) references these files:

• tests/distributed/test_kimi_linear_context_parallel.py (needs 2)
• tests/lora/test_deepseekv2_tp.py (needs 2 and 4)
• tests/lora/test_qwen3moe_tp.py (needs 2 and 4)
• tests/v1/e2e/test_replayssm_decode.py (needs 2)

Out of scope, not included above

• tests/v1/engine/test_engine_core.py — module-level pytest.skip(..., allow_module_level=True) gates it to CUDA only; it never reaches the GPU-count check on ROCm. Not a real ROCm gap, excluded.
• tests/models/kimi_k3/test_amd_latent_moe_runner.py / test_latent_moe_tail.py — only covering job is NVIDIA-only (:nvidia: (B200) Kimi K3), no AMD job exists at all. Can't verify from this pipeline; flagging as out of scope for a ROCm-specific list rather than guessing.

---

Separately: opened a draft PR with a static-analysis tool that catches this bug class before merge instead of finding it after the fact — happy to pursue that in parallel once this list is sorted, doesn't need to block on it.

[github-actions]

CC @hongxiayang @tjtanaa @vllmellm @giuseppegrossi for ROCm-related issue

[AndreasKaratzas]

Let's create some optional new test groups under test-amd.yaml for these. Good discovery :)
So the new test groups, should specifically target the tests that are skipped and use the correct number of GPUs for these tests.