ZmqEventPublisher.publish() blocks EngineCore thread indefinitely when event queue is full, causing shm_broadcast deadlock

[ilmarkov]

Summary

ZmqEventPublisher.publish() calls Queue.put(events) with no timeout on a bounded queue (maxsize=max_queue_size, default 100 000). Under sustained high concurrency, the queue fills faster than the publisher thread can drain it, causing publish() to block the EngineCore's step loop indefinitely. This prevents the APIServer from receiving the step-completion signal, so it never writes the next batch to the shm_broadcast slot, deadlocking all DP ranks.

Affected file

vllm/distributed/kv_events.py — ZmqEventPublisher

# constructor
self._event_queue = Queue[EventBatch | None](maxsize=max_queue_size)

# publish() — called synchronously from EngineCore step loop
def publish(self, events: EventBatch) -> None:
    ...
    self._event_queue.put(events)   # ← blocks indefinitely when queue is full

Root cause

queue.Queue.put(item) with a non-zero maxsize blocks the calling thread until a slot is available. The publisher thread drains the queue at ZMQ send speed, which under bursty conditions (see below) can lag behind event production. Once the queue reaches max_queue_size, every call to publish() from the EngineCore hangs.

The ZMQ PUB socket already has its own hwm to handle backpressure at the network layer (by dropping messages). The Python queue should mirror this behaviour — drop, not block.

How the queue fills in practice

With enable_kv_cache_events=True and a SimpleCPUOffloadConnector active:

• Each GPU→CPU block offload emits two events per block per KV-cache-spec group: BlockRemoved(medium='GPU') + BlockStored(medium='CPU')
• At high concurrency (e.g. DP-8, 256 concurrent requests, ~150 k-token shared prefixes), a single scheduler step across 8 DP ranks can enqueue thousands of EventBatch objects in one shot.
• The publisher thread (_publisher_thread) encodes and sends these serially via send_multipart. When the ZMQ socket's HWM is hit (messages dropped), the thread stays fast — but encoding latency and any shared ZMQ-context contention (Nixl RDMA, DP coordinator) can cause momentary lag.
• Over many minutes of inference the queue drifts toward max_queue_size. A final large CPU-offload burst pushes it over the edge and put() blocks.

Observed symptom

In a disaggregated prefill/decode deployment on 8× DP ranks:

1. All 8 EngineCore_DPx processes log simple_kv_offload/manager.py CPU-store events at the same timestamp.
2. Immediately after, all 8 ranks enter the shm_broadcast poll loop and emit repeated Poller timed out (every 5 s), eventually emitting:

   No available shared memory broadcast block found in 60 seconds.
   

3. The APIServer never schedules the next batch; health endpoints remain responsive but no inference progress is made.
4. DPSupervisor eventually marks ranks unhealthy and kills the pod.

The hang is 100% reproducible above a threshold concurrency (~256 for 150 k-token prompts) and absent at lower concurrency, consistent with a gradual queue fill.

This issue is not present when enable_kv_cache_events=False (uses NullEventPublisher).

Proposed fix

Replace the blocking put() with a non-blocking put_nowait() and drop events when the queue is full — mirroring the ZMQ HWM semantics that already exist on the socket:

def publish(self, events: EventBatch) -> None:
    if not self._running:
        raise RuntimeError("Publisher is closed")
    if events.data_parallel_rank is None:
        events.data_parallel_rank = self._data_parallel_rank
    try:
        self._event_queue.put_nowait(events)
    except queue.Full:
        logger.warning(
            "KV event queue full (%d items), dropping event batch",
            self._event_queue.maxsize,
        )

Dropped batches are acceptable: the ZMQ PUB HWM already allows lossy delivery, and subscribers (e.g. EPP's precise-prefix-cache) are designed to tolerate missing events by treating unknown blocks as cache misses.

Environment

• vLLM v0.27.2rc1.dev77+gac7509e2b
• DP size 8, two LWS pods x 4 GPUs each
• SimpleCPUOffloadConnector with cpu_bytes_to_use=42949672960
• MooncakeStoreConnector also active
• kv_events_config: hwm=100000, max_queue_size=100000, buffer_steps=10000