[RFC]: Using Helion for Selected vLLM CustomOps

[yushangdi]

Motivation.

TL;DR

1. We propose to use Helion by default for some CustomOps.

2. We show that Helion kernels have superior performance compared to the CUDA kernels in vLLM.

   Across broad shape sweeps, the three Helion kernels we benchmarked achieve 1.382–1.785x geomean speedups on H100, 1.556–1.804x on B200, and 1.022–1.427x on MI350X.

   On Qwen3-8B-FP8, this translates to 2.59%–3.49% higher end-to-end throughput on H100, 3.77%–7.44% on B200, and 3.41%–8.76% on MI350X across the tested concurrency levels.

3. We propose to let vLLM take a permanent Helion dependency.

Helion is a Python-embedded DSL that compiles PyTorch-like kernel code to lower-level DSLs, such as Triton. As demonstrated in the PyTorch blog post Portable vLLM Model Inference Kernels in Helion, kernels under vllm/kernels/helion/ are competitive with—and often faster than—their CUDA counterparts in vLLM, delivering end-to-end gains on Qwen3 FP8 models running on H100 and B200 GPUs. We have also observed end-to-end gains on AMD MI350X GPUs.

This RFC proposes an integration model for using Helion kernels in vLLM CustomOps when CUDA graphs are enabled. The initial rollout will cover the following kernels:

1. per_token_group_fp8_quant
2. silu_and_mul_per_block_quant
3. rms_norm_per_block_quant

The earlier Helion kernel-authoring RFC, [RFC]: Add Helion integration in vLLM (#32219), remains the design reference for implementing and registering kernels. This RFC proposes the following additional integration changes:

1. Make Helion a pinned vLLM dependency.
2. Use Helion by default for some CustomOps.

Comparison with Helion integration in SGLang. Helion is integrated into SGLang as an optional backend for Kimi Delta Attention. Like the integration proposed here, it uses pre-tuned Helion configs and requires no runtime autotuning. The main difference is that SGLang provides separate kernel implementations and fixed configurations for decode and prefill workloads; our integration uses a single kernel implementation for each CustomOp and selects among pre-tuned configurations based on the input shape range. In SGLang, for Kimi-Linear-48B-A3B-Instruct (TP=2, 2xGB200), we observe a geomean 5.04%–5.27% improvement in tok/s compared to the default kernels.

Performance

When measuring performance, we used a single config for each Helion kernel on H100 and up to six configs for each Helion kernel on B200.

Standalone Kernel Performance with Extensive Shape Sweeps on H100 and B200

A sweep uses physical dimensions from public model configurations and TP-local dimensions for TP sizes 1, 2, 4, and 8. The shape matrix includes RNJ1, Gemma 3 (270M through 27B), dense and MoE Qwen3 variants, MiniMax-M2, DeepSeek-V3, and Kimi-K2.

• H100 artifact: https://gist.github.com/yushangdi/592fc18a79459198504737f019811b4b
• B200 artifact: https://gist.github.com/yushangdi/850731446a7215c6a72978683d973455

H100

B200

Kernel	Number of shapes
per_token_group_fp8_quant	1,173
rms_norm_per_block_quant	255
silu_and_mul_per_block_quant	918

Smaller number of configs used by each Helion kernel. Higher performance gains are possible with additional configurations. For example, the existing configuration set achieves a 2.313x geomean speedup (range: 1.54–3.57x) for rms_norm_per_block_quant.

In this RFC, we intentionally trade some performance for a smaller, more manageable configuration set—at most six configs for each Helion kernel. Users seeking further performance improvements can autotune the Helion kernel for their specific workloads.

End-to-End Model Performance on Qwen/Qwen3-8B-FP8 with Helion CustomOps

H100

Settings:

• Linear backend: DeepGEMM with float32 ceiling E8M0 scales
• Input/output lengths: 512/600, with EOS ignored
• Concurrency: 8, 16, 32, and 64
• Per observation: one full-concurrency warmup wave, then four measured waves

Concurrency	Native tok/s	Helion tok/s	Helion vs. native
8	1,273.46	1,317.89	+3.489%
16	2,352.54	2,432.04	+3.379%
32	4,008.54	4,129.85	+3.026%
64	6,292.87	6,455.52	+2.585%

B200

Settings:

• Linear backend: CUTLASS. We do not use the DeepGEMM linear backend for the B200 experiments because it does not use all three kernels.
• Input/output lengths: 512/600, with EOS ignored
• Concurrency: 8, 16, 32, and 64
• Per observation: one full-concurrency warmup wave, then four measured waves

Concurrency	Native tok/s	Helion tok/s	Helion vs. native
8	1,804.0	1,938.1	+7.44%
16	3,135.6	3,296.8	+5.14%
32	5,633.5	5,992.3	+6.37%
64	9,159.9	9,505.6	+3.77%

Proposed Change.

The proposed integration is prototyped in #48995. If this RFC is accepted, we will submit the changes as a series of polished PRs.

• [existing] @register_kernel exposes a kernel as torch.ops.vllm_helion.<kernel_name>.
• [existing] ConfigManager loads checked-in Helion configurations from configs/<kernel>/<platform>.json.
• [existing] Each kernel's deterministic pick_config() chooses one pre-tuned config. There is no runtime autotuning.
• [new] Helion kernels being used will JIT-compile a selected variant during CUDA-graph capture.
• [new] Gate CustomOp routing to Helion kernels with VLLM_USE_HELION_KERNELS, defaulting to 1. Setting it to 0 falls back to the alternative torch.ops._C CUDA kernel.

The runtime behavior is:

Condition	Behavior
Supported op during CUDA-graph capture	Helion
VLLM_USE_HELION_KERNELS feature disabled	Same as current
No CUDA graph	Same as current; compiled routing pass is not installed
Unsupported platform (no Helion config file for it)	Same as current
Op executed outside CUDA-graph capture	Same as current

Routed Custom Op

vLLM defines torch.ops.vllm_helion.routed_<kernel_name>. Its schema and mutation contract match the corresponding Helion op. Its implementation is conceptually:

def routed_impl(*args):
    if torch.cuda.is_current_stream_capturing():
        return helion_op(*args)  # torch.ops.vllm_helion.<name>
    return native_op(*args)  # torch.ops._C.<name>

During CUDA-graph capture, the Helion launch is recorded; replay executes the recorded kernel without re-entering the Python dispatcher. Outside capture, the same routed op calls the native implementation.

If either the Helion op or the native op is unavailable—for example, because the native build does not contain the CUDA op or the platform has no Helion configs—the routed op is not registered.

Direct Call

per_token_group_fp8_quant has a direct eager call site in QuantFP8.forward_cuda. Conceptually, we write:

if (
    VLLM_USE_HELION_KERNELS
    and not torch.compiler.is_compiling()
    and x.is_contiguous()
    and torch.cuda.is_current_stream_capturing()
):
    # This calls the routed custom op registered above.
    return helion_per_token_group_fp8_quant(...)
return native_per_token_group_fp8_quant(...)

The not torch.compiler.is_compiling() condition short-circuits before the capture check while Dynamo is tracing. The compiled graph therefore still contains the native quant op, allowing RMSNorm/quant fusion to match normally; the post-grad routing pass handles any supported op that remains afterward. Only genuinely eager execution being captured into a CUDA graph takes the direct Helion path.

Handling the torch.compile Path

Some models still use torch.compile by default, and we integrate Helion into them by using a post-grad Inductor pass in the compiled path. The routed Helion custom ops are inserted into the torch.compiled graph through the Inductor pass.

We add an additional post-grad FX pass, HelionFusionRoutingPass. It retargets direct calls to these native ops and remaining auto_functionalized wrappers whose inner target is a native op. Running it last therefore preserves vLLM's existing fusion, lowering, functionalization, and copy-elimination behavior.

Extending Support to AMD

Although this RFC initially targets NVIDIA sm90 and sm100, the same integration can support AMD GPUs by adding platform-specific tuned configurations. On ROCm, torch.cuda.is_current_stream_capturing() reports the HIP stream-capture state, allowing the routed custom op to work without platform-specific logic.

The AMD results below demonstrate the resulting performance gains.

Helion compared with the corresponding AITER kernels on MI350X:

Artifact: https://gist.github.com/yushangdi/80dcca98fb7a0652af22f6be7240beaf

The RMSNorm gains are concentrated in large-token, wide-hidden shapes rather than spread uniformly across the matrix:

Shape regime	Number of shapes	Geomean
hidden <= 4096, all token counts	153	1.001x
hidden > 4096, tokens <= 256	70	1.005x
hidden > 4096, tokens >= 272	32	1.169x

End-to-End Model Performance on Qwen/Qwen3-8B-FP8 on MI350X

Max concurrency	Prompts	AITER output tok/s	Helion output tok/s	Throughput change	AITER TPOT (ms)	Helion TPOT (ms)	TPOT change
8	64	1,358.20	1,463.29	+7.74%	5.580	5.432	-2.65%
16	128	2,589.50	2,816.32	+8.76%	5.892	5.633	-4.40%
32	512	4,489.52	4,642.60	+3.41%	6.641	6.387	-3.82%
64	512	8,479.87	8,840.22	+4.25%	7.459	7.160	-4.01%

Feedback Period.

2 weeks

CC List.

@xiaohongchen1991 @zou3519

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[github-actions]

CC @hongxiayang @tjtanaa @vllmellm @giuseppegrossi for ROCm-related issue

[jahnclawdmonet]

Data point from a GPU class outside your benchmark matrix: RTX PRO 6000 Blackwell (SM120, CC 12.0, 96 GB), canonical platform name nvidia_rtx_pro_6000_blackwell_server_edition, on nightly 0.26.1rc1.dev1177+ga9a17e709 with helion 1.4.0 / triton 3.7.1 / torch 2.13.0+cu130, single GPU.

Three observations relevant to the default-on proposal:

1. Out of the box, every Helion kernel is disabled on this platform. vllm/kernels/helion/configs/ ships configs for nvidia_b200, nvidia_h100, nvidia_h200 only, so registration logs Helion kernel '<name>' is disabled: No configs available ... on platform 'nvidia_rtx_pro_6000_blackwell_server_edition' for all eight kernels and execution falls back to the CUDA ops. Default-on would currently be a silent no-op for SM120 (workstation Blackwell) and any other platform without shipped configs.

2. Local autotuning (scripts/autotune_helion_kernels.py --autotune-effort quick) on the three RFC kernels was rough on this platform: 52 of 126 shape configs succeeded overall. Per kernel: per_token_group_fp8_quant 0/42 (every attempt ended in helion.exc.NoConfigFound), silu_and_mul_per_block_quant 21/42, rms_norm_per_block_quant 31/42 (that one alone took 53 minutes). So opt-in autotuning as a distribution story is currently a partial answer here: one of the three RFC kernels cannot produce a config at quick effort at all.

3. Where configs did land, the performance case holds. scripts/benchmark_helion_kernels.py --baseline cuda with cudagraphs on the autotuned kernels:

kernel	cases timed	geomean vs CUDA	min	max	faster than CUDA
silu_and_mul_per_block_quant	42	1.50x	0.34x	8.82x	39/42
rms_norm_per_block_quant	42	3.29x	0.89x	14.00x	39/42
per_token_group_fp8_quant	no config produced, fell back to CUDA

The silu geomean (1.50x) sits in the same band as your H100 numbers (1.556x to 1.804x). The rms_norm geomean lands higher (3.29x), driven mostly by small-token cases where the CUDA op is launch-bound (up to 14x at 1-4 tokens) while large-token cases are roughly at parity. So SM120 benefits at least as much as the datacenter parts once a config exists. The gap is distribution: shipped per-platform configs or a more robust autotune path for platforms outside the current three. If shipped configs is the intended model, happy to contribute the generated nvidia_rtx_pro_6000_blackwell_server_edition config files for the two kernels that tuned successfully as a PR.

Caveats: quick autotune effort rather than full (full may well fix the per_token failures and improve the other two), single GPU, numerics checked against the eager reference by the benchmark script's default path. The per-candidate failure logs were truncated on my side, so I cannot yet say whether the per_token failures are resource limits or something else; can rerun with full logging if useful.

[jahnclawdmonet]

Follow-up on item 2, the per_token_group_fp8_quant autotune failures: full verbose logs show all 42 shapes die identically at generation 0.

For every shape the sequence is: Failed to transfer compiler seed config 1: (the reason string after the colon is empty), fallback to the default config which becomes the entire initial population (Initial population: 1 total), that single candidate fails, and LFBOTreeSearch (quick effort, initial_population=FROM_BEST_AVAILABLE, max_generations=5) raises NoConfigFound from population_statistics without further exploration.

So on this platform the blocker for that kernel is the seed/default config path rather than a broad search failure: the other two kernels have working seeds on SM120 and tuned fine. The empty failure reason on the seed transfer line also looks like a logging gap in Helion worth an upstream note, since it hides exactly the information needed to fix the seed. Full log available if useful.