feat(ascend): port E2E-verified ascend support from 0.20.2 to v0.3.0-dev (vllm 0.24.0) by tengqm · Pull Request #387 · flagos-ai/vllm-plugin-FL

tengqm · 2026-08-17T12:07:03Z

Summary

Ports the E2E-verified Ascend (Huawei NPU) support from the vllm 0.20.2 baseline
(PR #307, commit 1326a33) onto v0.3.0-dev, which tracks vllm 0.24.0.

The 0.24.0 upgrade (#274) never touched the ascend directory: get_name()
still returned "ASCEND_FL", and 0.24.0's AttentionBackendEnum has no such
member — extensions are only accepted via the CUSTOM slot — so any Ascend
launch failed with ValueError: Unknown attention backend: 'ASCEND_FL'
(vllm/model_executor/layers/attention/attention.py:401). This PR restores a
working ascend backend on the 0.24.0 baseline.

PR upgrade vllm to 0.20.2 on ascend platform #307 port (1326a33, 13 files): the E2E-verified 0.20.2 ascend
implementation — CUSTOM-backend registration, get_supported_kernel_block_sizes → [128], supports_update_block_table, NPU-aware platform config, FLA/GDN
ops, fused_moe kernels, MMEncoder attention.
PR ascend: blacklist lift_fresh/_to_copy (coreDim=0 on scalar tensor init) #361 blacklist (baeafde): blacklist lift_fresh/lift_fresh_copy/
_to_copy in ascend.yaml (coreDim=0 crash on scalar tensor init);
E2E-required, merged into this branch.
One 0.24.0-specific fix in worker.py — see the dedicated section below.
Function-name pow blacklist (ascend.yaml): the Triton path crashed
compiling RoPE _compute_inv_freq (pow.Scalar -> pow_scalar) because
flag_gems 5.3.4 ascend pow.py uses chained or conditions the
triton_ascend 3.2.0 fork rejects. - pow was ineffective (config_filter
matches function name, not op name); replaced with the wrapper names
pow_tensor_tensor(_) / pow_tensor_scalar(_) / pow_scalar. E2E-verified
on a pristine pow.py — no container patch needed.

Test results (from PR #307, vllm 0.20.2 / ascend-cann9.0.0, Qwen3.6-27B-A3B and

Qwen3.6-35B-A3B, TP=2)

Test	Model	Type	Result
Text math (7×8)	27B	Text	✅ 56
Text knowledge (Capital of France)	27B	Text	✅ Paris
Text math (7×8)	35B	Text	✅ 56
Text knowledge (Capital of France)	35B	Text	✅ Paris
Image description	27B	Image	✅ Correct (black text, white background)
Image OCR	27B	Image	✅ "Hello"
Image description	35B	Image	✅ Correct (black text, white background)
Image OCR	35B	Image	✅ "Hello"
Concurrent text (8 requests)	27B	Text	✅ 8/8
Concurrent images (8 requests)	27B	Image	✅ 8/8
Concurrent mixed (8 requests)	27B	Mixed	✅ 8/8
Concurrent text (8 requests)	35B	Text	✅ 8/8
Concurrent images (8 requests)	35B	Image	✅ 8/8
Concurrent mixed (8 requests)	35B	Mixed	✅ 8/8
Prime numbers	27B	Text	✅ 2, 3, 5, 7, 11
Prime numbers	35B	Text	✅ 2, 3, 5, 7, 11
Code generation	27B	Text	✅ Valid Python
Code generation	35B	Text	✅ Valid Python

0.24.0 verification status

PASSED (2026-08-17, hw25, ascend-cann9.0.0, Qwen3-4B TP1, flagtree
path). Serve reached Application startup complete; OpManager dispatched
rms_norm / rotary_embedding to vendor.ascend and silu_and_mul to
default.flagos. Both smoke inferences returned coherent output
(knowledge → "Paris...", math → "56..."), system fingerprint
vllm-0.24.0-563743c8.

PASSED (2026-08-18, hw25, same stack, Triton path). With compiler triton
(PYTHONPATH=/opt/triton → triton 3.2.0 fork + triton_ascend 3.2.1 overlay),
serve reached Application startup complete and both smoke inferences
returned coherent output (knowledge → "Paris...", math → "56..."), same
fingerprint vllm-0.24.0-563743c8. The pow-compile blocker is fixed in this
PR via the function-name blacklist (item 4): flag_gems 5.3.4 ascend pow.py
uses chained or conditions the triton_ascend 3.2.0 fork rejects
(UnsupportedLanguageConstruct: chained boolean operators) while compiling
RoPE _compute_inv_freq (pow.Scalar -> pow_scalar). The previous - pow
entry was ineffective because config_filter() matches the impl function
name; the wrapper names now route pow to torch_npu (exact), verified E2E on
a pristine pow.py — no container patch needed. Upstream pow.py fix still
tracked in the 0.24.0 report (遗留 items).

Runtime stack: image flagos-runtime-ascend-cann9.0.0:2.1.2, Python 3.11.15,
torch 2.10.0+cpu, torch_npu 2.10.0, flag_gems 5.3.4, flagtree
0.6.1+ascend3.5 (default), side /opt/triton = triton_ascend 3.2.1
(covers triton as 3.2.0 fork), vllm 0.24.0+flagos, device Ascend910B4-1, CANN 9.0.0.

Non-fatal: flag_gems 5.3.4 index_select.py:45 logical and/or deprecation
warning (upstream, no correctness impact; tracked in the 0.24.0 report
遗留事项). GDN/hybrid models (Qwen3-Next) remain unsupported on 0.24.0
(see "0.24.0-specific customizations" §2).

0.24.0-specific customizations

Recorded here because the 0.24.0 baseline requires changes beyond a verbatim
port, per the porting plan.

worker.py — NPU profile branch restructure. In 0.24.0 the
profile_result-derived computations (torch_peak_increase, kv-cache budget,
cudagraph estimate) live at function level after the profiling with-block;
in 0.20.2 they sat inside the else branch. After cherry-pick the NPU branch
fell through into the profiling block (NameError on profile_result, then it
would have overwritten the NPU kv-cache budget with the profiling-derived
value). Fixed by moving the whole block inside else and defaulting
cudagraph_memory_estimate = 0 so the CUDA-graph warning tail stays safe on
NPU.
GDN/hybrid models (Qwen3-Next) are not yet supported on 0.24.0. vllm
0.24.0 restructured vllm/model_executor/layers/mamba/gdn_linear_attn.py
into a mamba/gdn/ package; qwen3_next.py lost its
causal_conv1d_fn/causal_conv1d_update/chunk_gated_delta_rule symbols;
and fla/ops/__init__.py no longer re-exports chunk_gated_delta_rule_fwd.
The GDN patches in patch.py target these symbols and silently no-op
(try/except-wrapped). Plain-attention models (Qwen3, Qwen2, Llama, ...) are
unaffected. Restructuring the GDN patches for the new layout is follow-up
work.
ascend.yaml license header. The v0.3.0-dev base file lacks the Apache
header that upstream/main carries — a merge-time cosmetic delta, not
introduced by this PR.

Merge-back notes

All 9 files under vllm_fl/dispatch/backends/vendor/ascend/ are byte-identical
to upstream/main, so the eventual v0.3.0-dev → main merge will be
conflict-free for the whole ascend directory.
git diff --check reports two pre-existing "blank line at EOF" warnings in
patch.py / mm_encoder_attention.py, inherited verbatim from upgrade vllm to 0.20.2 on ascend platform #307
(byte-identical to upstream/main); intentionally left untouched.

This PR was written in part with the assistance of generative AI.

vllm serve crashes during EngineCore startup on Ascend 910B4 with "coreDim is invalid (value 0)". The crash originates from register_buffer("_k_scale", torch.tensor(1.0)) during model construction: torch.tensor(scalar) materializes the constant via aten::lift_fresh, which FlagGems intercepts. Its copy kernel computes a zero grid for a scalar/tiny tensor, and the NPU rejects the launch with coreDim=0. Blacklist lift_fresh, lift_fresh_copy, and _to_copy so these trivial copies fall back to torch_npu (lossless). Verified on ascend-cann9.0.0: serve reaches "Application startup complete" and inference is coherent (Qwen3-4B, TP=1).

In vllm 0.24.0 the profile_result-derived computations (torch_peak_increase, kv cache budget, cudagraph estimate) live at function level after the with-block. 1326a33's 0.20.2 ascend port had them inside the else branch, so after cherry-pick the NPU branch fell through and raised NameError on profile_result, then would have overwritten the NPU kv-cache budget with the profiling-derived value. Move the whole block inside the else and default cudagraph_memory_estimate to 0 so the CUDA-graph warning tail stays safe on NPU.

flag_gems ascend pow kernels use `if (A or B or C)` which the triton_ascend 3.2.0 fork frontend rejects (UnsupportedLanguageConstruct: chained boolean operators), so the Triton path crashed compiling the RoPE _compute_inv_freq `10000.0 ** t` (pow.Scalar -> pow_scalar). config_filter() matches on the impl function name (item[1].__name__), so the previous `- pow` entry was ineffective. List the wrapper names (pow_tensor_tensor(_) / pow_tensor_scalar(_) / pow_scalar) instead; falling back to torch_npu.pow is exact. Verified E2E on hw25 (ascend-cann9.0.0, vllm 0.24.0): Triton path serve + inference with pristine flag_gems pow.py, no container patch needed. Co-Authored-By: Claude <noreply@anthropic.com>

…s None) flag_gems cumsum kernels call get_num_sms(device_idx) -> torch_device_fn.get_device_properties(idx).multi_processor_count when N > 16384. On torch_npu 2.9.0 / triton_ascend 3.2.0 (cann8.5.0) the properties object lacks multi_processor_count -> None * 4 -> TypeError, killing EngineCore on the first decode step (vllm topk_topp_sampler torch.cumsum(probs_sort, dim=-1, out=probs_sort), batch < 8 -> pytorch path). hw25 (cann9.0.0, torch_npu 2.10.0) has a working multi_processor_count, so this is cann8.5.0-specific. config_filter() matches on the impl function name (item[1].__name__), so both wrapper names are listed: "cumsum" (aten::cumsum) and "cumsum_out" (aten::cumsum.out). Falling back to torch_npu.cumsum is exact. E2E verified on hw26 (cann8.5.0) flagtree path: Qwen3-4B serve + knowledge/math requests, no cumsum/coreDim crash markers.

… stride crash torch_npu._npu_rotary_embedding (ATB path) calls repeat_interleave to expand the cos/sin cache for GQA. Under triton_ascend 3.2.0 (cann8.5.0) the flag_gems repeat_interleave pointwise copy kernel (rank-3, t2048) fails to compile: ConvertTritonIRToLinalgIR 'strides must not be zero' -> MLIRCompilationError. The exception raises inside the vendor.ascend rotary impl on the first decode call; CachedOp's silent failure path marks the impl failed (no warning log) and falls back to the flag_gems rope kernel, which hangs/garbage-collects the first request under the triton compiler. Blacklisting repeat_interleave falls back to the exact torch_npu native implementation. This PR was written in part with the assistance of generative AI.

flag_gems linear (aten::linear wrapper) kernel spins forever on device under triton_ascend 3.2.0 for decode (M=1) shapes: AICore pinned at ~111% while EngineCore stalls in get_current_stream (driver.py:219). Fresh SIGUSR1 stack dump of the hung EngineCore pinned the hang inside flag_gems/ops/linear.py:173 linear -> triton runner -> get_current_stream. flagtree 0.6.0+ascend3.2 compiles the same kernel fine. mm/addmm already blacklisted; linear was the remaining GEMM path in the hot decode loop. Falling back to torch_npu.linear is exact. E2E-verified on hw26 (cann8.5.0): Qwen3-4B both requests HTTP 200, coherent output, no crash. Co-Authored-By: Claude <noreply@anthropic.com>

tengqm · 2026-08-18T05:17:16Z

Ascend CANN 8.5.0 (hw26) — dual-compiler E2E verification for vllm 0.24.0

Verified 2026-08-18 on hw26 (ascend-cann8.5.0, 910B4, Qwen3-4B TP1).

Both compiler paths pass E2E — serve reaches Application startup complete, inference coherent (knowledge "Paris" / math "56"), crash markers 0:

flagtree 0.6.0+ascend3.2 (default)
triton_ascend 3.2.0 via compiler triton

Four blacklist commits added on this branch for the triton path under triton_ascend 3.2.0:

pow (dd5b76e) — impl wrapper name mismatch
cumsum (679085e) — get_num_sms returns None
repeat_interleave (36ec4e4) — MLIR stride crash
linear (cf8998c) — flag_gems linear spins forever for decode (M=1) shapes: AICore pinned at ~111%, EngineCore stalls in get_current_stream (root-caused via SIGUSR1 stack dump). flagtree 0.6.0+ascend3.2 compiles the same kernel correctly. Routing aten::linear to torch_npu unblocks the decode loop.

The container ascend.yaml was verified byte-identical to the branch at cf8998c.

Environment: torch 2.9.0+cpu / torch_npu 2.9.0 / flag_gems 5.3.4 / vllm 0.24.0+flagos (cp311 aarch64 empty wheel).

cyber-pioneer and others added 3 commits August 17, 2026 19:52

github-actions Bot added the core label Aug 17, 2026

tengqm mentioned this pull request Aug 17, 2026

docs(vllm-repack): record ascend CANN 9.0.0 0.24.0 E2E verification flagos-ai/build-infra#423

Merged

tengqm mentioned this pull request Aug 18, 2026

docs(vllm): record ascend 0.24.0 Triton-path pow blacklist verification flagos-ai/build-infra#426

Merged

tengqm and others added 3 commits August 18, 2026 11:11

This was referenced Aug 18, 2026

feat(vllm-plugin-wheel): add plugin_repo input for pre-merge fork ports flagos-ai/build-infra#432

Merged

feat(sunrise): port attention backend to vLLM 0.24.0 (CUSTOM registration + ptpu memory_stats shim) #391

Open

Navigation Menu

Uh oh!

feat(ascend): port E2E-verified ascend support from 0.20.2 to v0.3.0-dev (vllm 0.24.0) - #387

feat(ascend): port E2E-verified ascend support from 0.20.2 to v0.3.0-dev (vllm 0.24.0)#387
tengqm wants to merge 7 commits into
flagos-ai:v0.3.0-devfrom
tengqm:feat/ascend-v024

tengqm commented Aug 17, 2026 •

edited

Loading

Uh oh!

tengqm commented Aug 18, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

tengqm commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Contents

Test results (from PR #307, vllm 0.20.2 / ascend-cann9.0.0, Qwen3.6-27B-A3B and

0.24.0 verification status

0.24.0-specific customizations

Merge-back notes

Uh oh!

tengqm commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Ascend CANN 8.5.0 (hw26) — dual-compiler E2E verification for vllm 0.24.0

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tengqm commented Aug 17, 2026 •

edited

Loading

tengqm commented Aug 18, 2026 •

edited

Loading