[Bug]: Qwen3.5 GDN — ~1.9x long-prefill throughput regression and ~1.15 GiB extra device memory vs 0.1.dev19754

[jhsmith409]

[Bug] Qwen3.5 GDN: ~1.9x long-prefill throughput regression and ~1.15 GiB extra memory use vs. dev19754

Summary

Comparing two vLLM builds on the same machine, same weights, same flags, same
torch, same FlashInfer, the newer build shows two independent regressions:

1. Long-prefill throughput regression ~1.9x. 176,425-token prefill drops
   from 1415 tok/s to 755 tok/s. Short prefill (~15.5k) is unaffected.
2. ~1.15 GiB additional device memory at identical configuration, which
   makes --max-num-batched-tokens=8192 fatally OOM on 16 GB cards where the
   older build runs it comfortably.

There is also a robustness bug: a worker-side OOM kills the whole engine and
surfaces as an unrelated KeyError instead of failing the single request (§5).

1. Environment

	"ship" (works)	"nightly" (regressed)
vLLM	0.1.dev19754+g3a0914114	0.26.1rc1.dev1177+ga9a17e709
image	vllm/vllm-openai@sha256:d392f621bb3e…	vllm/vllm-openai@sha256:3578c1fa6a96…
torch	2.13.0+cu130	2.13.0+cu130
flashinfer	0.6.17	0.6.17
transformers	5.15.0	5.15.1

• GPUs: 2x RTX 5070 Ti (16 GB, SM120 / compute 12.0), TP=2
• Model: unsloth/Qwen3.8-27B-NVFP4 (Qwen3_5ForConditionalGeneration,
  model_type: qwen3_5), compressed-tensors NVFP4, --language-model-only
• Attention backend on both: FLASHINFER (out of ['FLASHINFER','TRITON_ATTN'])
• Model dims: 64 layers, full_attention_interval=4,
  linear_num_value_heads=48, linear_value_head_dim=128, head_dim=256

Serving flags (identical except where noted):

--tensor-parallel-size=2 --kv-cache-dtype=fp8 --max-model-len=188160
--gpu-memory-utilization=0.95 --kv-cache-memory-bytes=<pin> --max-num-seqs=4
--max-num-batched-tokens=<4096|8192>
--mamba-cache-dtype=bfloat16 --mamba-ssm-cache-dtype=bfloat16
--compilation-config={"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,2,4]}
--enable-prefix-caching --language-model-only --trust-remote-code

2. Regression A — long-prefill throughput

Identical prompt construction, max_tokens=1, median of 3 reps, same 176,425
prompt tokens on both sides.

Build	max-num-batched-tokens	tok/s	seconds
ship	8192	1442	122.3
ship	4096	1415	~125
nightly	4096	755	233.6
nightly	8192	fatal OOM	—

bt-matched comparison is 1415 -> 755 = 1.87x slower. Reps are extremely
tight (755/755/755; 233.81/233.62/233.64 s), so this is not noise.

Short prefill is unaffected, which is the diagnostic part:

Build	~15.5k prefill, C=1	C=4
ship (bt=4096)	2179	2191
nightly (bt=4096)	2183	2195

Decode is also fine (nightly marginally faster): C=1 52.7 vs 51.7,
C=3 142.5 vs 138.1, C=4 189.0 vs 186.5 combined tok/s.

So the penalty appears only at long context, and only in prefill.

What I ruled out

• Not FlashInfer version — 0.6.17 on both.
• Not torch — 2.13.0+cu130 on both.
• Not attention backend — both log
  Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN'].
• Not the FLA chunk kernels — third_party/flash_linear_attention/ops/chunk_o.py
  is byte-identical between the two images. Only fused_recurrent.py (decode
  path, adds a SPLIT_BATCH_HEAD_GRID for B*HV > 65535) and index.py
  (adds a gpu_sync_allowed() guard) differ.
• Not GDN prefill backend dispatch — the selection function in
  model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py (the
  triton|flashinfer|cutedsl chooser) is byte-identical between builds.
• Not FlashInfer autotuner fallback — the bt=4096 run logged zero
  No tuned config covers warnings.

I was not able to isolate the root cause further from the outside.

3. Regression B — ~1.15 GiB additional memory

Measured post-boot, idle, same day, same machine:

Build	KV pool	idle free / card
ship (bt=8192)	193,602 tok	1440 MiB
nightly (bt=4096)	188,937 tok	330 MiB

The nightly has a smaller KV pool (~79 MiB less) yet ~1110 MiB less free
memory — roughly 1.15 GiB more consumed at equivalent configuration.

4. Consequence — bt=8192 OOMs on the nightly

With --max-num-batched-tokens=8192, the first ~16k prefill dies:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 48.00 MiB.
GPU 1 has a total capacity of 15.51 GiB of which 31.31 MiB is free.

File ".../vllm/third_party/flash_linear_attention/ops/chunk_o.py", line 168, in chunk_fwd_o
    o = torch.empty_like(v)
  via chunk_gated_delta_rule_fwd  (.../ops/chunk.py:72)

The 48.00 MiB is exactly the GDN output buffer:

v = [max_num_batched_tokens, linear_num_value_heads / TP, linear_value_head_dim]  bf16
  = [8192, 48/2, 128] * 2 bytes = 50,331,648 B = 48.00 MiB   # exact match

Ship allocates the identical 48 MiB and is fine — it has 1440 MiB free. The
allocation didn't change; the available headroom did.

Relaxing --kv-cache-memory-bytes does not help. Freeing 64 MiB moved free
memory only 31.31 -> 33.31 MiB, because the caching allocator absorbed it:
reserved-but-unallocated rose 37.92 -> 112.92 MiB. (expandable_segments is
already enabled via PYTORCH_ALLOC_CONF.)

--max-num-batched-tokens=4096 halves the buffer to 24 MiB and works — but
costs long-prefill throughput, and the nightly is already 1.87x slower there.

Note on the memory-profiling guardrail

With --kv-cache-memory-bytes set, vLLM logs skipped memory profiling, so it
never verifies that what remains after the pin can cover prefill activations.
The server boots healthy and then dies on the first real request. A startup-time
check that the post-pin headroom covers the GDN buffer
(max_num_batched_tokens * H_v/TP * D_v * dtype_size) would turn a fatal
runtime crash into an actionable boot error.

5. Separate bug — worker OOM kills the engine and masks itself

The OOM is raised in the worker, but the failure surfaces as:

File ".../vllm/v1/core/sched/scheduler.py", line 1854, in update_from_output
    req_index = model_runner_output.req_id_to_index[req_id]
KeyError: 'chatcmpl-95f443c92d6709b3-a7e574bd'
EngineCore encountered a fatal error.
...
vllm.v1.engine.exceptions.EngineDeadError

The scheduler indexes req_id_to_index for a request whose worker died, so the
real cause (torch.OutOfMemoryError) is masked by an unrelated KeyError and
the entire engine dies rather than the one request failing. This reproduces
deterministically and is independent of whether the OOM itself is legitimate.

Suggested handling: treat a missing req_id in req_id_to_index as a failed
request (fail that request with the worker's original exception) instead of
raising KeyError and taking down EngineCore.

6. Possible fix for the allocation itself

chunk_fwd_o already supports a caller-provided output buffer:

# third_party/flash_linear_attention/ops/chunk_o.py
if core_attn_out is not None:
    assert core_attn_out.numel() >= v.numel(), ...
    o = core_attn_out[: v.numel()].view(*v.shape)
else:
    o = torch.empty_like(v)          # <-- 48 MiB transient, taken on this path

core_attn_out is plumbed through chunk.py end-to-end (both builds), but
no caller in model_executor/ passes it — grep -rn core_attn_out vllm/model_executor/ returns nothing in either image. Threading the model's
existing output buffer into chunk_gated_delta_rule would remove a transient
max_num_batched_tokens * H_v/TP * D_v * dtype_size allocation per
linear-attention layer. Happy to prepare that PR if maintainers agree it is the
right shape.

7. Reproduction

docker run --gpus all --ipc host -p 8005:8005 \
  -v /path/to/qwen3.8-27b-nvfp4:/models/m:ro \
  vllm/vllm-openai@sha256:3578c1fa6a9676e1de068b9d75c777cc865d251fadfbe6175ae82278739c6674 \
  --model=/models/m --tensor-parallel-size=2 --language-model-only \
  --kv-cache-dtype=fp8 --max-model-len=188160 --max-num-seqs=4 \
  --max-num-batched-tokens=8192 --gpu-memory-utilization=0.95 \
  --mamba-cache-dtype=bfloat16 --mamba-ssm-cache-dtype=bfloat16

# then send any ~16k-token prompt -> engine dies.
# with --max-num-batched-tokens=4096 it survives; time a ~176k prefill
# (max_tokens=1) and compare against dev19754.

[dhakshin32]

Hi, I'd like to take a look at §5 — the unhandled KeyError at scheduler.py:1854 where update_from_output indexes model_runner_output.req_id_to_index with a req_id that isn't present, crashing the entire EngineCore instead of failing the individual request.

I'm not planning to tackle the throughput regression (§2) or the memory increase (§3) in the same PR — those look like separate
root causes that need bisection.

[jahnclawdmonet]

Independent attempt on different SM120 hardware, using your exact image digests.

Environment deltas from your setup, stated upfront: single RTX PRO 6000 Blackwell 96 GB (SM120, CC 12.0), so TP=1 instead of TP=2, and no 16 GB memory pressure. Same model (unsloth/Qwen3.8-27B-NVFP4), same images by digest (ship = sha256:d392f621bb3e..., tag qwen38-x86_64-cu130; nightly = sha256:3578c1fa6a96...), same flags otherwise (--kv-cache-dtype=fp8 --max-model-len=188160 --max-num-seqs=4 --max-num-batched-tokens=4096 --mamba-cache-dtype=bfloat16 --mamba-ssm-cache-dtype=bfloat16, PIECEWISE cudagraphs, prefix caching on, --language-model-only). Prompts: 3 distinct ~140k-token prompts (unique leading header per rep so prefix caching cannot serve a rep from cache), max_tokens=1, temperature 0.

build	rep	prompt tokens	seconds	prefill tok/s
ship	0	140,571	28.21	4,983.7
ship	1	140,484	27.54	5,101.8
ship	2	140,699	27.64	5,090.0
nightly	0	140,571	28.37	4,954.4
nightly	1	140,484	27.64	5,082.8
nightly	2	140,699	27.77	5,067.4

Device memory at identical config: consumed memory (weights + non-torch) 20.96 GiB ship vs 21.09 GiB nightly; "kv cache memory in use" 61.89 vs 61.76 GiB; nvidia-smi during serving 87091 vs 86793 MiB.

Result: the regression does not reproduce here. Mean prefill is 5058.5 tok/s on ship vs 5034.9 tok/s on nightly, a 0.5 percent gap, smaller than the rep-to-rep spread within either build (2.4 percent across ship reps). Memory is flat as well. One incidental difference: engine init was faster on nightly (125.5 s vs 146.0 s, compilation 56.7 s vs 76.0 s).

Since the same digests show a 1.9x prefill gap on your 2x RTX 5070 Ti setup but none at TP=1 on a single 96 GB card, the delta likely lives in something this config does not exercise: the TP=2 path, or behavior under the near-full memory conditions of 16 GB cards (consistent with your Regression B being a fatal OOM).

Caveats: TP=1 vs your TP=2 means any regression that lives in the TP overlap path would not show here; my prompt corpus is synthetic word filler rather than your workload; single-GPU 96 GB removes the OOM dimension entirely, so this says nothing about your Regression B fatal OOM other than the memory delta number above.

[jhsmith409]

I'll send a more rigorous answer tomorrow. This error did not reproduce on my RTX 6000 Pro either (lots of VRAM, so nothing gets really strained). I was able to reproduce the OOM on an RTX 5090 and I've been seeing a similar OOM error since one of the nightlies between v0.25 and v0.26 - I previously posted an issue (Qwen3.6 instead of 3.8) on that an bisected every single nightly to narrow down to the exact day it was introduced. Will post the details tomorrow. See #50780 and #49476