Skip to content

[Bug] triton 3.6.0: TritonILUVATARGPUMMAReduceThreadLocality pass causes PassManager::run failed on vllm kernel_unified_attention (BI-V150) #886

Description

@physics31415926

环境

组件 版本
硬件 Iluvatar BI-V150 × 8
CoreX Driver 4.5.0
torch 2.10.0+corex.4.5.0
flagtree 0.6.0+iluvatar3.6 (triton 3.6.0)
vllm 0.24.0+empty
vllm-plugin-fl 0.2.0

对比基准:triton 3.2.0+corex.4.5.0(无 flagtree)— 同硬件上全部测试通过。


问题 1:0 compatible backends for target (cuda)(已在 vllm-plugin-FL 侧修复)

torch._inductor 始终用 GPUTarget(backend='cuda') 构造 triton 目标,但 flagtree triton 3.6.0 注册的 backend 为 'iluvatar',且其 supports_target() 检查 target.backend == 'corex',导致:

RuntimeError: 0 compatible backends for target (cuda) ([]).
There should only be one.

已在 vllm-plugin-FL 侧修复(runtime probe supports_target() 确定正确 remap,将 'cuda''corex'):
physics31415926/vllm-plugin-FL@b1e7aed

不过,从 flagtree 角度,理想的做法是让 flagtree triton 的 supports_target() 也接受 'cuda',或者让 backend 注册时同时注册 'cuda' alias,避免下游框架需要 workaround。


问题 2:TritonILUVATARGPUMMAReduceThreadLocality pass 导致 PassManager::run failed(待修复)

应用问题 1 的修复后,triton 可以正常选择 backend 并开始编译,但在编译 vllm 的 kernel_unified_attention 时,flagtree 定制 pass TritonILUVATARGPUMMAReduceThreadLocality 触发 SSA dominance 检查失败。

完整报错

vllm/v1/attention/ops/triton_unified_attention.py:179:0: error: Failures have been detected while processing an MLIR pass pipeline
vllm/v1/attention/ops/triton_unified_attention.py:179:0: note: Pipeline failed while executing [`TritonILUVATARGPUMMAReduceThreadLocality` on 'builtin.module' operation]

vllm/v1/attention/ops/triton_attention_helpers.py:415:19: error: operand #0 does not dominate this use
vllm/v1/attention/ops/triton_unified_attention.py:539:44: note: called from
vllm/v1/attention/ops/triton_attention_helpers.py:415:19: note: operand defined here (op in the same block)

  File "vllm/v1/attention/ops/triton_unified_attention.py", line 1019, in unified_attention
RuntimeError: PassManager::run failed

出错代码

triton_attention_helpers.py:415

def softmax_step(S, M, L):
    m_j = tl.max(S, axis=1)
    m_j = tl.where(m_j > float("-inf"), m_j, 0.0)
    P = tl.exp(S - m_j[:, None])
    l_j = tl.sum(P, axis=1)
    alpha = tl.exp(M - m_j)   # ← line 415: operand M does not dominate this use
    L_new = L * alpha + l_j
    return m_j, L_new, P, alpha

M 是函数入参,在 TritonILUVATARGPUMMAReduceThreadLocality pass 的 IR 变换后,M 对应的 MLIR operand 不再支配 tl.exp(M - m_j) 处的 use,触发 SSA dominance 检查失败。triton 3.2.x 的同名 pass 无此问题。

调用链

kernel_unified_attention (triton_unified_attention.py:179)
  → softmax_step (triton_unified_attention.py:539)
    → tl.exp(M - m_j) (triton_attention_helpers.py:415)
      → TritonILUVATARGPUMMAReduceThreadLocality FAIL

期望修复

  • 方案 A(推荐):修复 TritonILUVATARGPUMMAReduceThreadLocality pass,使其能正确处理此类 IR 模式(函数入参在同 block 内的 use),与 triton 3.2.x 行为一致。
  • 方案 B:在 pass 中对此类 SSA dominance 情况做容错处理。

复现

# 需要 flagtree 0.6.0+iluvatar3.6 环境,Iluvatar BI-V150
import os
os.environ["VLLM_ALLOW_LONG_MAX_MODEL_LEN"] = "1"
os.environ["VLLM_PLUGINS"] = "fl"
from vllm import LLM, SamplingParams
llm = LLM(
    model="<any-attention-model>",
    tensor_parallel_size=4,
    max_model_len=4096,
    gpu_memory_utilization=0.60,
    enforce_eager=True,
)
outputs = llm.generate(["Hello"], SamplingParams(max_tokens=10, temperature=0.0))

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions