fix(gdn): keep packed decode beta in fp32 - #385
Conversation
| from vllm_fl.patches.gdn_packed_decode import patch_vllm_packed_gdn_beta | ||
|
|
||
| patch_vllm_packed_gdn_beta() | ||
|
|
There was a problem hiding this comment.
only nvidia need gdn patch,also need in other vendor?
There was a problem hiding this comment.
This is not NVIDIA-specific; it is tied to this vulnerable packed GDN kernel implementation. Any vendor build that ships and uses the same kernel needs the fix, while some vendor vLLM images may omit the FLA kernel entirely or route GDN through a different implementation. Therefore the hook remains vendor-agnostic and capability-based rather than checking vendor_name. I hardened the loader in 59e2b63 so a missing FLA/patch module safely no-ops before import, while the existing symbol/source checks no-op when the target kernel is absent or already fixed. I also added missing-module and vendor-agnostic registration tests; all 5 targeted tests pass. H100 is the reproduction and validation platform here, not the applicability condition.
Summary
sigmoid(b)in FP32 throughout the recurrent state updateRoot cause
The vLLM 0.24.0 packed GDN decode kernel computes
sigmoid(b)in FP32, rounds it to the input dtype, and converts it back to FP32. That round-trip perturbs every recurrent state update, so the error compounds over long decode sequences. The standard GDN update path keeps the gate in FP32.This is an implementation-level issue rather than an NVIDIA-specific one. Vendor images that omit this FLA kernel or route GDN through a different implementation are left untouched by module/symbol/source detection. H100 is the reproduction and validation platform for this PR.
Validation
For the serving head shape
H=16, HV=48, K=128, V=128, an H100 packed-vs-standard comparison over 1000 recurrent steps produced:A 42-case serving regression previously improved strict-loop incidence from 20/42 to 1/42 and answer closure from 8/42 to 39/42 with this numerical fix. It is a necessary precision fix, not a guarantee that every long sampled generation will converge.
git diff --checkpassed.