XDPoS v2: orphaned peers serving a stale higher-round QC cause an unrecoverable 'missing trie node' wedge during recovery

[AnilChinchawale]

Summary

During recovery of a wedged XDPoS v2 (HotStuff BFT) devnet (chainId 5151, 6-node masternode set), we found that orphaned/follower nodes still running an abandoned chain segment can indefinitely re-poison a freshly-recovered validator set by gossiping a stale, higher-round QuorumCert whose block/state no longer exists on the canonical chain. The recovered validators adopt that QC as highestQC (HotStuff picks the highest round), then attempt to propose on top of a block whose world state has been pruned/never re-executed, and wedge permanently with:

ERROR Failed to create mining context  err="missing trie node <root> (path ) state <root> is not available, not found"

This turned an otherwise-mechanical rollback recovery into a multi-hour firefight: every time we rebuilt the validators from a clean snapshot, they re-acquired the stale QC from a follower we hadn't yet found, and re-wedged.

Environment

• Client: XDC v2.7.0-devnet (XDC-v270, 48c4c4be, go1.26.4)
• Consensus: XDPoS v2 (round-based BFT), epoch 900, gap 450, V2 switch block 1800, period 2s
• Topology: 6 enrolled masternodes (4× v2.7.0 + 1× xone + 1× geth-fork), plus read-only follower nodes (besu / erigon / reth) syncing the same networkId for cross-client parity work
• Trigger: a producer crash left an uncommitted tail (blocks ~15242–15444) that only some nodes retained

What happens

1. Network wedges; committed head across the honest majority is block B (e.g. 15241/14835 after rollback). An abandoned segment up to block P (15443/15444) survives only on a few nodes, carrying a QC at round R_p where R_p > R_b.
2. Operators rebuild the validator set from a clean snapshot at B (correct committed head).
3. A still-running follower on the abandoned segment dials in to a validator (inbound connections are accepted even with --nodiscover) and gossips SyncInfo containing the QC at R_p.
4. The validator adopts highestQC = QC(P @ R_p) because R_p is the highest round it has seen — even though it does not have block P's world state (pruned under --gcmode full, or the block was never re-executed).
5. On its proposing turn the validator tries to build P+1 on P, calls state.New(P.Root) → missing trie node → cannot create mining context → never proposes → round times out forever. Quorum can never be reached. Permanent wedge.

Observed evidence

# validator a, after clean rebuild to committed head, re-poisoned by a follower:
[yourturn] Yes, it's my turn based on parent block round=14399 \
    ParentHash=0x3e3fae264...785859f ParentBlockNumber=15443
ERROR Failed to create mining context err="missing trie node a290079b... is not available, not found"

# no node on the canonical chain even has block 15443 anymore:
eth_getBlockByNumber(0x3c53) => null   # on all rebuilt validators
# yet highestQC still references it (round 14399 > canonical rounds ~13700)

# the source, found via admin_peers on the wedged validator:
"name":"erigon/v3.4.2-..."  @ 95.217.112.125     # follower on the abandoned chain
"name":"besu/v26.7-develop-..."                    # follower on the abandoned chain

Killing all orphaned followers (besu/erigon/reth) fleet-wide was the only way to stop re-poisoning; after that the rebuilt validators converged and resumed producing immediately.

Why this is a bug (not just operator error)

HotStuff safety says never commit against a higher QC — but here the node adopts a highestQC for a block it cannot materialize (no state), and there is no fallback: it neither (a) requests or re-executes the missing state from the QC-advertising peer, nor (b) refuses a highestQC whose block body/state is unavailable, nor (c) falls back to proposing on the highest block it can build on. The result is a liveness deadlock that is invisible to the operator (the node logs "Yes, it's my turn" and looks healthy) and self-perpetuating across restarts because the QC is re-gossiped.

Suggested fixes (any one breaks the deadlock)

1. Guard highestQC adoption: when receiving a QC/SyncInfo for a block whose header/body/state is not locally retrievable and cannot be fetched from the sender, do not silently set it as highestQC; log loudly and either fetch-or-reject.
2. Proposer fallback: if createMiningContext(highestQC.block) fails with a missing-state error, fall back to proposing on the highest block with available state (and surface a WARN), rather than looping "Not my turn / my turn" forever.
3. Reject QCs for pruned/orphaned blocks older than the last committed block or off the canonical chain during SyncInfo processing.
4. Operational: document that read-only followers must be stopped or reset before/during a coordinated validator rollback, because they re-serve abandoned QCs to --nodiscover validators via inbound connections.

Impact

High for any coordinated recovery of a v2 network that also runs follower/observer nodes (multi-client parity fleets, explorers, RPC followers). A single un-reset follower can prevent the entire validator set from recovering, with no obvious error pointing at the follower.