feat(vllm-0.24): add Qwen3.5 text-only runtime compatibility - #383
feat(vllm-0.24): add Qwen3.5 text-only runtime compatibility#383CherryLemon wants to merge 7 commits into
Conversation
| if input.stride(-1) != 1 or output.stride(-1) != 1: | ||
| return _torch_moe_sum(input, output) | ||
| return _flag_gems_moe_sum(input, output) | ||
|
|
There was a problem hiding this comment.
why not dispatch moe_sum via opmanager?
There was a problem hiding this comment.
Good point. The previous adapter bypassed the dispatch layer. I changed the bridge to use CachedOp("moe_sum"), so OpManager now owns backend selection, per-op policy, fallback, operator-list recording, and I/O-dump hooks. The monkey patch remains only because upstream vLLM calls vllm._custom_ops.moe_sum directly. The vendor.cuda implementation unwraps the original vLLM function to avoid recursive re-entry during fallback, and the new tests cover both dispatch and recursion prevention.
| class Qwen3_5TextModelArchConfigConvertor(ModelArchConfigConvertorBase): | ||
| """Normalize multimodal architecture names in text-only Qwen configs.""" | ||
|
|
||
| def get_architectures(self) -> list[str]: | ||
| architectures = super().get_architectures() | ||
| if not architectures: | ||
| default = _DEFAULT_ARCHITECTURES.get(self.hf_config.model_type) | ||
| normalized = [default] if default is not None else architectures | ||
| else: | ||
| normalized = [ | ||
| _CONDITIONAL_TO_CAUSAL.get(arch, arch) for arch in architectures | ||
| ] | ||
|
|
||
| # The runtime model loader consults hf_config.architectures again after | ||
| # ModelArchitectureConfig has been built. Keep both views synchronized | ||
| # so it cannot fall back to a stale VL architecture. | ||
| if normalized != architectures: | ||
| self.hf_config.architectures = normalized.copy() | ||
| return normalized | ||
|
|
||
|
|
||
| class Qwen3_5ForCausalLMConfig(Qwen3_5ForConditionalGenerationConfig): | ||
| """Use the upstream cache config checks and remove multimodal RoPE keys.""" | ||
|
|
||
| @staticmethod | ||
| def verify_and_update_config(vllm_config) -> None: |
There was a problem hiding this comment.
why need patch model architecture and rope param?
There was a problem hiding this comment.
Agreed. Those two patches were compatibility code for older checkpoint configs that used conditional-generation architecture names and retained multimodal RoPE fields. This PR now targets the canonical final text-only config, which declares the causal architecture directly. I removed the architecture convertor and RoPE mutation and reuse the upstream vLLM hybrid-cache verifier directly for the causal architectures.
| whitelist/blacklist. This supports arbitrary MoE top-k with FP32 accumulation | ||
| and handles empty-token batches without rebuilding or replacing vLLM's | ||
| `_moe_C` extension. | ||
|
|
There was a problem hiding this comment.
provide a general readme instead model-level readme
There was a problem hiding this comment.
Updated. I replaced the model-specific section with a general Runtime compatibility hooks section describing plugin-owned config/model registration and OpManager-based operator adapters. Model-specific implementation details remain in the code and PR description.
|
is this pr based on vllm-project/vllm#50210 ? |
Yes. The Qwen3.5 text-only model support is a plugin-level backport of vllm-project/vllm#50210 for vLLM 0.24.0. The model registration and hybrid GDN state hooks follow that PR. It is not a direct cherry-pick because the compatibility is installed through plugin runtime hooks, with additional plugin-specific checkpoint/FP8 mapping and MoE fixes. |
3 participants
PR Category
Core
PR Type
New Features / Bug Fixes
Description
Add Qwen3.5 text-only dense and MoE runtime support to the v0.3.0-dev branch,
which targets vLLM 0.24.0. The upstream vLLM release contains the causal model
classes but does not fully register their canonical text configs and causal
architectures, hybrid cache helpers, or VL-prefixed checkpoint mapping.
The compatibility layer is installed by the general plugin and leaves the
vLLM package and compiled
_moe_Cextension untouched.Related Issues
Related upstream implementations:
moe_sumreduction and support any topk vllm-project/vllm#46643Changes
architectures; do not rewrite model architectures or RoPE parameters.
model.language_model.*checkpointnames, including the shared
hf_to_vllm_mapperused by FP8 ignored-layermatching.
moe_sumthroughCachedOp("moe_sum"), keeping backendselection, fallback, per-op policy, operator-list recording, and I/O
diagnostics under OpManager. Empty batches and non-contiguous tensors retain
safe fallback handling.
quantization metadata with the unquantized FlagOS implementation.
dependency contract instead of duplicating version checks.
Out of scope for this PR:
the target branch through fix(worker): preserve DP GPU offset for independent engines #380 and fix(worker): account for CUDA graph memory in KV cache sizing #381.
Testing
pytest -qon the targeted registration,moe_sumadapter, OpManager,CUDA fallback, and fused-MoE-layer suites: 61 passed in a vLLM 0.24.0 /
Python 3.12 image.
covered VL-prefixed dense and FP8 MoE checkpoints: weight loading, health
checks, and text generation completed; this update does not change the model
shim or checkpoint mapper.
ruff check,ruff format --check, andgit diff --checkpass.Checklist