Skip to content

Do not subscribe to subnets for liquidated clusters #1084

Description

@shane-moore

Description

Subnet subscriptions are computed from every cluster the operator is a member of, with no filter on liquidation status:

  • SubnetGenerator::committees_from_network_state (subnet_service/src/subscriptions.rs) maps get_own_clusters() directly to subnets. ClusterLiquidated only flips cluster.liquidated (database/src/pending_updates.rs), and a cluster only leaves the own-clusters set when its last validator is removed, so liquidated clusters keep their subnets subscribed indefinitely. The same logic exists in all releases (v1.2.3: subnet_service/src/lib.rs).
  • Expected message rates for gossipsub topic scoring (get_committee_info_for_subnet in subnet_service/src/scoring.rs) also count validators from liquidated clusters, network-wide, inflating expected rates even though liquidated clusters generate no traffic.

Everywhere else, Anchor already treats liquidation as "do no work" (#558, #264): duties are filtered in the validator store and metadata service, index sync skips liquidated clusters, and exit processing skips them. The gossip layer is the remaining gap.

go-ssv never subscribes for liquidated clusters: subscriptions happen per started validator (Validator.Start()), and validators are only initialized/started from shares filtered with ByNotLiquidated. One asymmetry worth knowing: on mid-run liquidation go-ssv stops the validator without unsubscribing the topic (onShareStop does not call p2pNetwork.Unsubscribe), so it sheds the stale subscription only on restart. The fix proposed here unsubscribes live, which is slightly stronger than go-ssv but consistent with Anchor's reactive subscription architecture.

Impact

Every inbound message on a subscribed subnet is fully validated on the urgent_consensus processor queue before "not interested" filtering, so unnecessary subnet subscriptions translate directly into wasted validation CPU and queue pressure (#254).

This is not hypothetical: while investigating a mainnet operator's sustained Processor queue full / Unable to pass message to message receiver errors, we found roughly a third of their registered validators sit in liquidated clusters. Their committees map to 57 of 128 subnets today; filtering liquidated clusters would drop that to 38, i.e. about a third less mainnet gossip to validate. Operators in many public clusters accumulate liquidated clusters over time and never shed the subscriptions.

Proposed solution

Filter liquidated clusters at the two read sites in subnet_service:

  1. SubnetGenerator::committees_from_network_state: skip clusters with cluster.liquidated when building the committee set.
  2. get_committee_info_for_subnet (scoring): skip liquidated clusters when computing expected message rates, so topic score parameters reflect actual traffic.

No event plumbing is needed: ClusterLiquidated/ClusterReactivated already publish a NetworkState change through the db watch channel, which triggers handle_subnet_changes; the existing subscribe/unsubscribe diffing then handles gossipsub topics, peer-manager join/leave, and the ENR subnet bitfield. This unsubscribe path is already exercised in production when a cluster's last validator is removed. Set semantics keep a subnet if a non-liquidated cluster shares it.

The filter belongs at the read sites, not inside get_own_clusters(): other consumers of cluster state (e.g. exit-event verification) intentionally still need to see liquidated clusters.

Testing: the subnet_service test harness already exercises the full run() loop against an in-memory database and captures emitted TopicEvents; add cases asserting an Unsubscribe after liquidating a cluster (when no other own cluster shares the subnet), a Subscribe after reactivation, and subnet retention when a non-liquidated cluster shares the subnet.

Edge cases considered

  • Zero expected rates: filtering can drive a topic's expected message rate to 0; the score-param generation already guards this (expected_messages_per_decay_interval > 0.0 checks in topic_score_config.rs, mesh-delivery weight disabled), so no degenerate params result.
  • Message validation and publishing are unaffected: both message_validator and message_sender use the pure committee-to-subnet computation, never the subscription state.
  • Reactivation race: resubscription is immediate on ClusterReactivated, but mesh grafting takes a few heartbeats, so the first duty after reactivation may ride on lazy gossip rather than eager mesh push. go-ssv has the same exposure, and the event already lags by the EL follow distance.
  • In-flight QBFT instances for a committee liquidated mid-slot stop receiving messages; their output is already unusable since duty gating and signing refusal apply from the same event.
  • Historical replay: an own cluster liquidated and later reactivated in history can produce one transient unsubscribe/resubscribe during initial sync (state updates are published per block batch and watch notifications coalesce, so same-batch transitions are no-ops). Bounded by own liquidation history.
  • Mesh density: shedding these subnets stops relaying for other committees sharing them; the go-ssv fleet already sheds them on restart, so this does not push the network below the dominant client's baseline.
  • Filtering on validator activity (exited/pending validators), as go-ssv's started-validator gating effectively also does, is deliberately out of scope: it is metadata-driven rather than contract-event-driven and has riskier edge cases (e.g. pending-activation validators must keep their subnet warm).

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions