Skip to content

Commit d3ab6bf

Browse files
tengqmclaude
andcommitted
fix(metax): fall back to Triton cache ops when vllm C extensions are missing
When vllm is built with VLLM_TARGET_DEVICE=empty, torch.ops._C_cache_ops is not registered. The MetaX flash attention backend calls vllm._custom_ops.reshape_and_cache_flash which delegates to that C extension — causing AttributeError at runtime. vllm already ships a pure-Triton implementation (triton_reshape_and_cache_flash) for platforms without compiled extensions. This patch routes _custom_ops through the Triton fallback when _C_cache_ops is absent. The patch only fires when _C_cache_ops is missing — standard vllm wheels with compiled C extensions are unaffected. Note: _C activation ops (silu_and_mul etc.) do NOT need stubs here. The MetaX vendor backend (silu_and_mul_maca) already bypasses vllm's SiluAndMul/GeluAndMul classes and computes the activations directly with F.silu/F.gelu — no dependency on torch.ops._C. Tested on MetaX C550 (MACA 3.7.2.0, torch 2.8.0+metax): vllm 0.20.2 empty + vllm-plugin-FL + Qwen3-4B, eager + flash attn, inference succeeds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 1326a33 commit d3ab6bf

1 file changed

Lines changed: 24 additions & 0 deletions

File tree

  • vllm_fl/dispatch/backends/vendor/metax/patches

vllm_fl/dispatch/backends/vendor/metax/patches/__init__.py

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,3 +29,27 @@
2929
# TODO: remove when MetaX Triton support is available.
3030
import vllm.utils.import_utils as iu
3131
iu.has_triton_kernels = lambda: False
32+
33+
# --------------------------------------------------
34+
# When vllm is built with VLLM_TARGET_DEVICE=empty, the C extension modules
35+
# are not compiled. torch.ops._C_cache_ops.* are not registered, so the
36+
# MetaX flash attention backend's reshape_and_cache_flash call path fails.
37+
#
38+
# vllm ships a pure-Triton implementation (triton_reshape_and_cache_flash)
39+
# for platforms lacking the compiled extension. Route _custom_ops through
40+
# the Triton fallback when the C ops are missing.
41+
#
42+
# Safe for standard vllm wheels: the patch only fires when _C_cache_ops is
43+
# absent, and every op in the dispatch chain is covered by the Triton kernel.
44+
import torch
45+
46+
if not hasattr(torch.ops, "_C_cache_ops"):
47+
import vllm._custom_ops as _custom_ops
48+
from vllm.v1.attention.ops.triton_reshape_and_cache_flash import (
49+
triton_reshape_and_cache_flash,
50+
triton_reshape_and_cache_flash_per_token_head_quant,
51+
)
52+
_custom_ops.reshape_and_cache_flash = triton_reshape_and_cache_flash
53+
_custom_ops.reshape_and_cache_flash_per_token_head_quant = (
54+
triton_reshape_and_cache_flash_per_token_head_quant
55+
)

0 commit comments

Comments
 (0)