Summary
Reproduced the exact wedge family described in #45406's second trigger (FCFS + async KV-load parked requests stranded behind an unschedulable queue head), but with SimpleCPUOffloadConnector (eager mode, CPU backend) on a recent nightly — providing an independent production data point with py-spy stacks and live /metrics from the wedged state.
Environment
- vLLM:
0.26.1rc1.dev1214+gf6130145c (nightly wheel, 2026-08-26)
- Hardware: 2× RTX 3090, TP=2, 128 GB host RAM, Ubuntu 22.04
- Model: hybrid linear-attention/full-attention 27B (GDN-style, 64 layers,
full_attention_interval=4), fp8 KV cache, block_size=1648
- Server flags (relevant):
--kv-offloading-size 60 --kv-offloading-backend native (→ SimpleCPUOffloadConnector via VLLM_USE_SIMPLE_KV_OFFLOAD=1, eager mode), --max-num-seqs 6, --max-num-batched-tokens 4096, --enable-prefix-caching --enable-chunked-prefill, MTP-style speculative decoding (dflash, 7 tokens), VLLM_PREFIX_CACHE_RETENTION_INTERVAL=1648
- Load: sustained agentic traffic (~270 requests/hour, short-to-medium prompts, heavy prefix reuse — external prefix cache hit rate 84.7% right before the wedge)
Local patch disclosure: we carry three small local patches to vllm/v1/simple_kv_offload/manager.py (store-side hash registration — the fix for our own report #53498 — plus a final-flush-on-request-finish). None of them touch the load path, the scheduler, or the promotion logic. Notably, the hash-registration patch enables CPU-cache lookups that always miss on stock main (#53498), which is what makes the async-load path (get_num_new_matched_tokens → WAITING_FOR_REMOTE_KVS) reachable in the first place on this connector. The stranding mechanism itself is scheduler-side and reachable with other connectors on stock main per #45406's FCFS reproduction with OffloadingConnector.
Symptom
Engine wedged silently at 17:51 local time: no ERROR/WARNING lines, /metrics still served, /health green. Live metrics from the wedged state:
vllm:num_requests_running{...} 0.0
vllm:num_requests_waiting{...} 3.0
vllm:num_requests_waiting_by_reason{...,reason="capacity"} 1.0
vllm:num_requests_waiting_by_reason{...,reason="deferred"} 2.0
Running: 0, Waiting: 3 (1 capacity + 2 deferred) — the deferred pair matches WAITING_FOR_REMOTE_KVS-style parked async loads (they had just matched CPU-resident prefix blocks), the capacity request is the unschedulable head. GPU KV usage was high (~80%+) right before the collapse after a burst of long-prompt requests.
py-spy on EngineCore main thread, two samples 10 s apart (engine alive, not deadlocked on a lock — busy-looping):
Thread 153859 (active): "MainThread"
_process_engine_step (vllm/v1/engine/core.py:1477) # the `time.sleep(0.001)` yield branch
run_busy_loop (vllm/v1/engine/core.py:1413)
run_engine_core (vllm/v1/engine/core.py:1354)
i.e. step_fn() keeps returning model_executed=False while scheduler.has_requests() is true — schedule() produces nothing schedulable every step, forever. Workers spin in shm_broadcast.acquire_read. Only a restart recovers; after restart with identical config and continued traffic the engine is healthy again.
Analysis
Same structural pattern as #45388/#45406: the waiting-queue traversal hits the capacity-blocked head and breaks; the two parked async-KV-load requests behind it are never reached by _try_promote_blocked_waiting_request(), so their finished_recving/load-completion signals are never consumed; nothing is running, so no blocks are ever freed; schedule() returns an empty step indefinitely while the engine reports healthy. #44560's admission gate doesn't cover this because the load requests were already admitted and re-queued via skipped_waiting when the async load was issued.
We did not capture VLLM_LOGGING_LEVEL=DEBUG in the incident window (production); the py-spy dumps and metric snapshots above were captured live during the wedge and are available in full if useful.
Suggested direction
Merge #45406 (defense-in-depth skip-past-unschedulable-head when self.running is empty), and consider a watchdog: if num_requests_running == 0 while num_requests_waiting > 0 with reason="deferred" persists for N seconds, log at WARNING/ERROR (and/or expose it in /health) — a healthy-idle engine has waiting=0; this combination is always a wedge.
Related
Before submitting:
Summary
Reproduced the exact wedge family described in #45406's second trigger (FCFS + async KV-load parked requests stranded behind an unschedulable queue head), but with
SimpleCPUOffloadConnector(eager mode, CPU backend) on a recent nightly — providing an independent production data point with py-spy stacks and live/metricsfrom the wedged state.Environment
0.26.1rc1.dev1214+gf6130145c(nightly wheel, 2026-08-26)full_attention_interval=4), fp8 KV cache,block_size=1648--kv-offloading-size 60 --kv-offloading-backend native(→SimpleCPUOffloadConnectorviaVLLM_USE_SIMPLE_KV_OFFLOAD=1, eager mode),--max-num-seqs 6,--max-num-batched-tokens 4096,--enable-prefix-caching --enable-chunked-prefill, MTP-style speculative decoding (dflash, 7 tokens),VLLM_PREFIX_CACHE_RETENTION_INTERVAL=1648Local patch disclosure: we carry three small local patches to
vllm/v1/simple_kv_offload/manager.py(store-side hash registration — the fix for our own report #53498 — plus a final-flush-on-request-finish). None of them touch the load path, the scheduler, or the promotion logic. Notably, the hash-registration patch enables CPU-cache lookups that always miss on stock main (#53498), which is what makes the async-load path (get_num_new_matched_tokens→WAITING_FOR_REMOTE_KVS) reachable in the first place on this connector. The stranding mechanism itself is scheduler-side and reachable with other connectors on stock main per #45406's FCFS reproduction withOffloadingConnector.Symptom
Engine wedged silently at 17:51 local time: no ERROR/WARNING lines,
/metricsstill served,/healthgreen. Live metrics from the wedged state:Running: 0, Waiting: 3 (1 capacity + 2 deferred)— the deferred pair matchesWAITING_FOR_REMOTE_KVS-style parked async loads (they had just matched CPU-resident prefix blocks), the capacity request is the unschedulable head. GPU KV usage was high (~80%+) right before the collapse after a burst of long-prompt requests.py-spy on EngineCore main thread, two samples 10 s apart (engine alive, not deadlocked on a lock — busy-looping):
i.e.
step_fn()keeps returningmodel_executed=Falsewhilescheduler.has_requests()is true —schedule()produces nothing schedulable every step, forever. Workers spin inshm_broadcast.acquire_read. Only a restart recovers; after restart with identical config and continued traffic the engine is healthy again.Analysis
Same structural pattern as #45388/#45406: the waiting-queue traversal hits the capacity-blocked head and
breaks; the two parked async-KV-load requests behind it are never reached by_try_promote_blocked_waiting_request(), so theirfinished_recving/load-completion signals are never consumed; nothing is running, so no blocks are ever freed;schedule()returns an empty step indefinitely while the engine reports healthy. #44560's admission gate doesn't cover this because the load requests were already admitted and re-queued viaskipped_waitingwhen the async load was issued.We did not capture
VLLM_LOGGING_LEVEL=DEBUGin the incident window (production); the py-spy dumps and metric snapshots above were captured live during the wedge and are available in full if useful.Suggested direction
Merge #45406 (defense-in-depth skip-past-unschedulable-head when
self.runningis empty), and consider a watchdog: ifnum_requests_running == 0whilenum_requests_waiting > 0withreason="deferred"persists for N seconds, log at WARNING/ERROR (and/or expose it in/health) — a healthy-idle engine has waiting=0; this combination is always a wedge.Related
Before submitting: