Skip to content

leios: a synced dingo node can't serve historical endorser blocks to a downstream peer — "endorser block not found" raised as a protocol violation that tears down the NtN connection #2662

Description

@chadaibbhot-git

Summary

On musashi (magic 164), a dingo relay that has fully synced to tip cannot re-serve
historical endorser blocks to a downstream dingo node syncing from it. When the downstream
requests an EB the relay no longer has, the relay raises protocol error: leios endorser block not found: <slot>.<hash>, which is treated as a protocol violation and tears down the whole
NtN connection
— tripping the downstream's short-lived-connection backoff (escalates to ~2m).
Net effect: a downstream dingo node (e.g. an operator's block producer) can't obtain the Leios
region from its own upstream dingo relay; it must fetch EBs from the IOG prototype relay directly.

There are two issues bundled here: (a) the node lacks/doesn't backfill historical EB bodies to
serve them, and (b) a not-found EB is escalated to a fatal protocol violation rather than handled
gracefully.

Current behavior

Both nodes dingo main (cbd517f, incl. #2654), --run-mode leios, core storage. Upstream
synced to tip (~block 121k); downstream a fresh/behind node catching up over NtN. Upstream
(serving) logs, ~6 teardowns in 40 min during catch-up:

ERROR "unexpected connection failure: protocol error: leios endorser block not found: 1893186.057b3488c0f56ffede6f63dd74e4491ba7ebc0785d9adfca21a7772fb2dc58de"
ERROR "unexpected connection failure: protocol error: leios endorser block not found: 1914738.7c62df9b0867b20955f5e14f7445d781ab40c3927b131735c525c5edf83c0455"

Downstream side: connection resets right after selecting the relay as best peer, then
short-lived connection detected, applying backoff (8s → 32s → 2m8s). The downstream still
advances via chainsync between resets (and through EB-light stretches), so catch-up limps along —
but it cannot rely on the relay for the EB region.

Analysis

EBs are served from an in-memory cache; historical EBs the node didn't recently handle are not
present and are not backfilled-to-serve, so the lookup returns not-found. Two problems:

  1. Availability/retention: a synced node should be able to serve the EBs of the chain it holds
    to its downstream peers (this is the serving-side mirror of the fetch-side gap in leios: endorser-block backfill stalls a from-genesis sync at the EB-region onset ("leios-fetch served no new transactions within tail budget") #2656).
  2. Robustness: a missing EB is escalated to a protocol violation that kills the connection,
    instead of a graceful "don't have it" response — this trips the peer's instability backoff and
    compounds the slowdown.

Distinct from #2630 (verifying dingo serving its own forged EBs) — this is about re-serving
historical EBs already on-chain. Same EB-availability root as #2656 (fetch side).

Reproduction

  1. Node A (dingo main, --network musashi --run-mode leios) synced to tip.
  2. Node B (empty/behind) syncs from node A over NtN through the Leios region.
  3. On node A observe protocol error: leios endorser block not found: <slot>.<hash> and the
    connection torn down; on node B observe repeated short-lived connection detected, applying backoff against node A.

Acceptance

  • A synced dingo node can serve historical EBs of its held chain to downstream peers (backfill /
    retain as needed), so a downstream can sync the Leios region from an upstream dingo relay.
  • A not-found EB is handled gracefully (no fatal protocol violation / connection teardown).

Code touchpoints

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Fields

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions