processor: undersized urgent_consensus queue drops gossip bursts (missed attestations on high-footprint operators)

[shane-moore]

Summary

A large mainnet operator (~75 committees, 57 of 128 subnets, v1.2.3) is missing attestations because the processor's single urgent_consensus queue drops inbound gossip during sub-second slot-boundary bursts, sometimes the node's own committee messages. The default depth (1000) is an underived placeholder. This issue is scoped to one focused change: raise that default to a measured, sane value (4096). Work-item expiry and the queue split are tracked in #254. (The affected operator is already unblocked via the existing --work-queue-size override; this fixes the default so the next high-footprint operator does not silently hit the same wall.)

Problem / impact

• Bursts of Processor queue full / Unable to pass message to message receiver on urgent_consensus, hundreds within single seconds at slot boundaries, quiet between.
• Not CPU-starved (~2.6 of 8 cores during the drops, idle cores available); load is bursty, not sustained.
• Most dropped messages are other operators' relay traffic (harmless), but a fraction are the node's own consensus/partial-signature messages, which causes the misses.

Root cause

Every inbound gossip message is one work item on the single urgent_consensus queue (default 1000, processor/src/lib.rs:78), drained by num_cpus workers each running the full validate-and-route (message_receiver/src/manager.rs:78). Past 1000 queued, the newest is dropped. RSA verify (message_validator/src/lib.rs:698) is lock-free and parallel, so throughput is bounded by the worker pool and the 1000-slot queue is the only burst buffer.

go-ssv validates inside go-libp2p's pubsub pipeline, which buffers up to ValidateThrottle = 8192 in-flight validations before dropping, so it absorbs ~8x larger slot-boundary bursts on comparable hardware. Anchor's 1000 was never derived from traffic (placeholder from #57).

Measured on mainnet (2026-06-14, read-only)

From the mainnet Anchor canaries' /metrics (4 active nodes, up to 80 days uptime):

• Per-item validation service time ~165-430 us (anchor_processor_worker_time{type="message_receiver"}, exact mean via sum/count). Drain ~2.5k msg/s/core, ~20k/s at 8 cores.
• Average arrival ~50-125 msg/s/node; message_receiver is 99.7% of all urgent_consensus traffic.
• Zero drops and zero queue depth on every canary over full uptime, so 1000 is adequate at single-cluster footprint and only the high-footprint operator's bursts overflow it. Average CPU ~2% of a core; this is burst-vs-buffer, not CPU.
• The operator overshoots 1000 by "hundreds" per burst (true peak >= ~1500, clipped by the drop), and runs clean on go-ssv's ~8192 ceiling.

Proposed fix

Raise the urgent_consensus default from 1000 to 4096 (processor/src/lib.rs:78), with a comment recording the derivation, plus tests (default size; burst-absorption past the old cap). Keep the queue name (it is the --work-queue-size CLI contract); value-only change, operator overrides preserved.

Why 4096 (not 8192): it covers the observed overshoot with ~2.7x margin and is modest enough to be safe without work-item expiry. A deeper default (toward go-ssv's 8192) buffers ~8x more before dropping, which on a sustained-overloaded node means grinding through proportionally more stale messages, so it would need expiry as a guard. 4096 sidesteps that, keeps the change to a single value, and is a strict improvement; if production shows it is short, the escalation below raises it with the proper guard. Peak held memory ~= depth x real msg size (~4 MB at 4096); tokio mpsc allocates lazily so idle cost is ~0.

Going deeper (if 4096 is not enough)

Recorded so a future maintainer can escalate without re-deriving; every trigger is a processor metric already exported. None needed now.

1. Deeper default (toward 8192). More burst headroom if send_error{type="message_receiver"} keeps climbing after 4096. A deeper buffer alone only delays drops, so under sustained overload it needs a shedding policy (next), not just a bigger backlog.
2. Priority / queue split (Processor: Split into multiple queues #254), the real overload lever. Shed other operators' relay traffic before our own duty-critical/outbound work; ownership answers "still useful to us?" far better than age. Trigger: send_error{type="signer"|"sender"} fires (own outbound dropped) or multi-second urgent_consensus backlog.
   • Why not time-based expiry: the dominant traffic is attestation QBFT / partial-signature messages, which stay useful for ~the attestation inclusion window (up to 32 slots / 1 epoch; a late-but-within-epoch attestation still earns the target reward, the largest flag component). A correct duty-aware TTL is therefore epoch-scale, so expiry would essentially never fire (a node 30+ slots behind is already dead) for near-zero benefit, while a naive ~1-slot TTL would wrongly drop still-rewardable work. Ownership-based shedding is the better mechanism; expiry might help only for genuinely short-window types (e.g. sync-committee, included the next slot).
3. Decouple accept-from-validate. Dispatch validation off the worker pool (go-libp2p style) so queue depth stops being the only burst buffer.
4. Per-subnet-scaled depth. Size the queue from the node's subnet count so benefit and memory cost land on the same high-footprint nodes.

Considered and not doing: un-hiding --work-queue-size. Its values are queue names slated to change in #254 (lib.rs:58 TODO) and an unknown name is a hard startup failure (config.rs:308-310), so documenting it now is a rename footgun.

Scope

• In: raise the urgent_consensus default (value) + tests.
• Deferred to Processor: Split into multiple queues #254: work-item expiry, the queue split, and the deeper escalation above.

Related

• Processor: Split into multiple queues #254 (processor: split into multiple queues), the structural follow-up and home for the escalation path.
• Do not subscribe to subnets for liquidated clusters #1084 (do not subscribe to liquidated-cluster subnets), complementary load reduction (~57 -> ~38 subnets for the affected operator), not a substitute.

[shane-moore]

Updated the issue body with read-only measurements from the mainnet canaries' /metrics (no operator data needed).

• Real per-item validation time is ~165-430 us (worker_time{type=message_receiver}, exact mean via sum/count), ~10x the pure-crypto microbench floor (~32 us). That is the real drain cost, so it is the sizing input.
• Average arrival ~50-125 msg/s/node; message_receiver is 99.7% of urgent_consensus. Zero drops / zero queue depth on every canary over up to 80 days, so 1000 is fine at small footprint and only the high-footprint operator overflows it.

Net: N* is bracketed to (1000, 8192] (operator overshoots 1000 by "hundreds" -> floor ~1500; runs clean on go-ssv's 8192-in-flight ceiling). A full 8192 queue drains in ~410 ms at 8 cores, well inside a slot, so it clears between bursts. Recommending 8192 + ~1-slot expiry + keep the drop alarm, refined later from the operator's post-deploy queue_length/send_error.

Dropped the planned synthetic burst-sweep: it can't pin the depth because the operator's burst size is the unknown it would need as input.

[shane-moore]

Narrowed the scope after the measurements + discussion: this issue is now just the default bump 1000 -> 4096, not 8192 + expiry.

• 4096, not 8192: it covers the observed overshoot with margin and is safe without expiry; a deeper buffer needs a shedding guard to avoid regressing sustained-overloaded nodes, which is more than this fix should carry.
• Expiry downgraded in "going deeper": the dominant traffic is attestation QBFT/partial-sigs, which stay useful for ~the inclusion window (up to ~1 epoch; a late-but-within-epoch attestation still earns the target reward). So a correct TTL is epoch-scale and would essentially never fire, while a short TTL would drop still-rewardable work. The real overload lever is ownership-based shedding via the Processor: Split into multiple queues #254 split (drop other operators' relay traffic before our own), so expiry + split both live in Processor: Split into multiple queues #254.

The affected operator is unblocked in the meantime via --work-queue-size urgent_consensus=....