[FR] GroupedTopKRouterFL missing method `_valid_grouping` — service crashes on DeepseekV3ForCausalLM (moonshotai/Moonlight-16B-A3B-Instruct)

[buzhengjing]

Bug Description

When loading any model whose architecture is resolved as DeepseekV3ForCausalLM (e.g., moonshotai/Moonlight-16B-A3B-Instruct) with vllm-plugin-FL installed, the vLLM service crashes unconditionally during the initial profiling run (determine_available_memory) with:

AttributeError: 'GroupedTopKRouterFL' object has no attribute '_valid_grouping'

The crash is not caused by any FlagGems operator. It occurs at the plugin framework level and cannot be worked around by disabling operators.

---

Environment

Item	Value
Model	moonshotai/Moonlight-16B-A3B-Instruct
Model Architecture	DeepseekV3ForCausalLM (MoE)
Hardware	NVIDIA H20-3e × 8
GPU Driver	570.158.01
vllm-plugin-FL	0.2.0
vLLM	0.20.2
FlagGems	5.0.2
PyTorch	2.11.0+cu130
CUDA	13.0
Python	3.12
Base Image	flagrelease_nvidia_vllm020plugin_base:0701

---

Steps to Reproduce

vllm serve moonshotai/Moonlight-16B-A3B-Instruct \
    --tensor-parallel-size 8 \
    --max-model-len 8192

The service crashes before processing any request — during the warmup / KV-cache profiling phase.

---

Root Cause Analysis

Call Chain

The crash occurs during service startup at _initialize_kv_caches → determine_available_memory → profile_run → _dummy_run, where a dummy forward pass is executed to measure GPU memory. For DeepseekV3ForCausalLM, this triggers the MoE router path:

vllm/v1/engine/core.py:128  __init__
  └─ _initialize_kv_caches
       └─ model_executor.determine_available_memory
            └─ vllm_fl/worker/worker.py:480  determine_available_memory
                 └─ model_runner.profile_run()
                      └─ vllm_fl/worker/model_runner.py:5588  _dummy_run
                           └─ deepseek_v2.py:1436  forward
                                └─ moe_runner.py:498  _apply_quant_method
                                     └─ base_router.py:284  select_experts
                                          └─ vllm_fl/ops/fused_moe/router.py:172  _compute_routing
                                               └─ self._valid_grouping(router_logits)  ← CRASH

The Defect

In vllm_fl/ops/fused_moe/router.py, line 172, GroupedTopKRouterFL._compute_routing calls:

if not self._valid_grouping(router_logits):

However, _valid_grouping is never defined in GroupedTopKRouterFL or any of its parent classes. This is an incomplete implementation — the method is referenced but never implemented.

Why Disabling Operators Cannot Fix This

The plugin calls register_router() at plugin load time, unconditionally replacing the vLLM MoE router with GroupedTopKRouterFL for DeepseekV3ForCausalLM. This monkeypatching happens regardless of FlagGems configuration.

Three independent startup attempts were made to rule out operator issues:

Attempt	Configuration	Result
1	Default (FlagGems fully enabled)	❌ Identical crash
2	USE_FLAGGEMS=0 (all FlagGems operators disabled)	❌ Identical crash
3	--enforce-eager (compilation disabled)	❌ Identical crash

All three produce exactly the same traceback. The crash is definitively in the plugin framework, not in any FlagGems operator.

---

Full Traceback

(EngineCore pid=1763) ERROR 07-30 00:45:00 [core.py:1136]
  File ".../vllm/v1/engine/core.py", line 1110, in run_engine_core
    raise e
  File ".../vllm/v1/engine/core.py", line 1110, in run_engine_core
    engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
  File ".../vllm/v1/engine/core.py", line 128, in __init__
    kv_cache_config = self._initialize_kv_caches(vllm_config)
  File ".../vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
    available_gpu_memory = self.model_executor.determine_available_memory()
  File ".../vllm/v1/executor/abstract.py", line 147, in determine_available_memory
    return self.collective_rpc("determine_available_memory")
  File ".../vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc
    result = run_method(self.driver_worker, method, args, kwargs)
  File ".../vllm_fl/dispatch/io_common.py", line 159, in wrapper
    return fn(*args, **kwargs)
  File ".../vllm_fl/worker/worker.py", line 480, in determine_available_memory
    self.model_runner.profile_run()
  File ".../vllm_fl/worker/model_runner.py", line 5897, in profile_run
    hidden_states, last_hidden_states = self._dummy_run(...)
  File ".../vllm_fl/worker/model_runner.py", line 5588, in _dummy_run
    outputs = self.model(...)
  File ".../vllm/model_executor/models/deepseek_v2.py", line 1436, in forward
    hidden_states = self.model(...)
  File ".../vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 498, in _apply_quant_method
    topk_weights, topk_ids = self.router.select_experts(...)
  File ".../vllm/model_executor/layers/fused_moe/router/base_router.py", line 284, in select_experts
    topk_weights, topk_ids = self._compute_routing(...)
  File ".../vllm_fl/ops/fused_moe/router.py", line 172, in _compute_routing
    if not self._valid_grouping(router_logits):
           ^^^^^^^^^^^^^^^^^^^^
AttributeError: 'GroupedTopKRouterFL' object has no attribute '_valid_grouping'

---

Expected Behavior

GroupedTopKRouterFL should either:

1. Implement _valid_grouping with the correct grouping validation logic for DeepseekV3ForCausalLM’s MoE routing configuration, or
2. Add a compatibility guard in register_router() to skip router replacement for unsupported MoE routing variants and fall back to the native vLLM router.

moonshotai/Moonlight-16B-A3B-Instruct is a standard DeepseekV3ForCausalLM MoE model and should be fully supported.

---

Impact

• Severity: Critical — service cannot start at all
• Scope: Any model resolving to DeepseekV3ForCausalLM with vllm-plugin-FL 0.2.0 installed
• Workaround: None available (cannot be bypassed by operator config, USE_FLAGGEMS=0, or --enforce-eager)