Skip to content

fix(gdn): keep packed decode beta in fp32 - #385

Open
CherryLemon wants to merge 2 commits into
v0.3.0-devfrom
fix/gdn-packed-beta-precision
Open

fix(gdn): keep packed decode beta in fp32#385
CherryLemon wants to merge 2 commits into
v0.3.0-devfrom
fix/gdn-packed-beta-precision

Conversation

@CherryLemon

@CherryLemon CherryLemon commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • move the packed GDN decode precision fix into vllm-plugin-FL instead of modifying the installed vLLM package
  • replace the vulnerable packed decode Triton kernel at general-plugin registration time
  • keep sigmoid(b) in FP32 throughout the recurrent state update
  • make the hook capability-based and idempotent: patch any vendor build carrying the vulnerable kernel, while no-oping when the FLA module/kernel is unavailable or already fixed

Root cause

The vLLM 0.24.0 packed GDN decode kernel computes sigmoid(b) in FP32, rounds it to the input dtype, and converts it back to FP32. That round-trip perturbs every recurrent state update, so the error compounds over long decode sequences. The standard GDN update path keeps the gate in FP32.

This is an implementation-level issue rather than an NVIDIA-specific one. Vendor images that omit this FLA kernel or route GDN through a different implementation are left untouched by module/symbol/source detection. H100 is the reproduction and validation platform for this PR.

Validation

For the serving head shape H=16, HV=48, K=128, V=128, an H100 packed-vs-standard comparison over 1000 recurrent steps produced:

state dtype metric before plugin patch
FP32 state relative L2 1.612e-3 7.530e-8
FP32 output relative L2 2.694e-3 1.691e-5
BF16 state relative L2 3.016e-3 2.085e-5
BF16 output relative L2 3.061e-3 3.487e-5

A 42-case serving regression previously improved strict-loop incidence from 20/42 to 1/42 and answer closure from 8/42 to 39/42 with this numerical fix. It is a necessary precision fix, not a guarantee that every long sampled generation will converge.

  • 5 patch lifecycle/capability tests passed, including missing-module and vendor-agnostic registration cases.
  • The plugin-owned kernel compiled and ran on H100 for the 1000-step comparison.
  • Ruff, Python compilation, and git diff --check passed.

Comment thread vllm_fl/__init__.py Outdated
Comment on lines +147 to +150
from vllm_fl.patches.gdn_packed_decode import patch_vllm_packed_gdn_beta

patch_vllm_packed_gdn_beta()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only nvidia need gdn patch,also need in other vendor?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not NVIDIA-specific; it is tied to this vulnerable packed GDN kernel implementation. Any vendor build that ships and uses the same kernel needs the fix, while some vendor vLLM images may omit the FLA kernel entirely or route GDN through a different implementation. Therefore the hook remains vendor-agnostic and capability-based rather than checking vendor_name. I hardened the loader in 59e2b63 so a missing FLA/patch module safely no-ops before import, while the existing symbol/source checks no-op when the target kernel is absent or already fixed. I also added missing-module and vendor-agnostic registration tests; all 5 targeted tests pass. H100 is the reproduction and validation platform here, not the applicability condition.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants