Skip to content

perf(nvidia): add opt-in Hopper long-context routing - #384

Merged
ceci3 merged 1 commit into
v0.3.0-devfrom
perf/h100-long-context-routing
Aug 17, 2026
Merged

perf(nvidia): add opt-in Hopper long-context routing#384
ceci3 merged 1 commit into
v0.3.0-devfrom
perf/h100-long-context-routing

Conversation

@CherryLemon

Copy link
Copy Markdown
Collaborator

Summary

  • add an architecture-specific dispatch profile for NVIDIA Hopper
  • route full-attention to the CUDA vendor backend so FlashAttention-3 is selected
  • blacklist FlagGems mm and mm_out so dense projections use native CUDA GEMM
  • keep the optimization disabled by default; enable it with VLLM_FL_HOPPER_LONG_CONTEXT_OPT=1 for direct A/B testing
  • preserve the existing NVIDIA configuration on non-Hopper devices and when capability probing is unavailable

Motivation

Nsight Systems analysis of a 16k-input serving workload showed two independent Hopper bottlenecks:

  • Triton unified-attention full prefill was about 11x slower than FA3 for the profiled head shape and accounted for 76.2% of the GPU-time gap.
  • FlagGems Hopper host-TMA aten::mm.out was slower than native dense GEMM for the projection shapes.

The measured output throughput moved from 535.340 tok/s to 691.895 tok/s with the attention route alone, and to 785.631 tok/s with both routes enabled, reaching 96.09% of the native baseline.

Usage

# baseline behavior, also the default
unset VLLM_FL_HOPPER_LONG_CONTEXT_OPT

# Hopper optimized route
export VLLM_FL_HOPPER_LONG_CONTEXT_OPT=1

Validation

  • 44 dispatch/config/FlagGems unit tests passed in the vLLM 0.24.0 stack.
  • On an H100 process, flag off selected nvidia.yaml, FlagOS attention, and no mm/mm_out blacklist.
  • On the same H100 process, flag on selected nvidia_hopper.yaml, vendor attention, and the mm/mm_out blacklist.
  • Ruff and git diff --check passed.

@ceci3
ceci3 merged commit 64a8b06 into v0.3.0-dev Aug 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants