You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
leios: a synced dingo node can't serve historical endorser blocks to a downstream peer — "endorser block not found" raised as a protocol violation that tears down the NtN connection #2662
On musashi (magic 164), a dingo relay that has fully synced to tip cannot re-serve
historical endorser blocks to a downstream dingo node syncing from it. When the downstream
requests an EB the relay no longer has, the relay raises protocol error: leios endorser block not found: <slot>.<hash>, which is treated as a protocol violation and tears down the whole
NtN connection — tripping the downstream's short-lived-connection backoff (escalates to ~2m).
Net effect: a downstream dingo node (e.g. an operator's block producer) can't obtain the Leios
region from its own upstream dingo relay; it must fetch EBs from the IOG prototype relay directly.
There are two issues bundled here: (a) the node lacks/doesn't backfill historical EB bodies to
serve them, and (b) a not-found EB is escalated to a fatal protocol violation rather than handled
gracefully.
Current behavior
Both nodes dingo main (cbd517f, incl. #2654), --run-mode leios, core storage. Upstream
synced to tip (~block 121k); downstream a fresh/behind node catching up over NtN. Upstream
(serving) logs, ~6 teardowns in 40 min during catch-up:
Downstream side: connection resets right after selecting the relay as best peer, then short-lived connection detected, applying backoff (8s → 32s → 2m8s). The downstream still
advances via chainsync between resets (and through EB-light stretches), so catch-up limps along —
but it cannot rely on the relay for the EB region.
Analysis
EBs are served from an in-memory cache; historical EBs the node didn't recently handle are not
present and are not backfilled-to-serve, so the lookup returns not-found. Two problems:
Robustness: a missing EB is escalated to a protocol violation that kills the connection,
instead of a graceful "don't have it" response — this trips the peer's instability backoff and
compounds the slowdown.
Distinct from #2630 (verifying dingo serving its own forged EBs) — this is about re-serving historical EBs already on-chain. Same EB-availability root as #2656 (fetch side).
Reproduction
Node A (dingo main, --network musashi --run-mode leios) synced to tip.
Node B (empty/behind) syncs from node A over NtN through the Leios region.
On node A observe protocol error: leios endorser block not found: <slot>.<hash> and the
connection torn down; on node B observe repeated short-lived connection detected, applying backoff against node A.
Acceptance
A synced dingo node can serve historical EBs of its held chain to downstream peers (backfill /
retain as needed), so a downstream can sync the Leios region from an upstream dingo relay.
A not-found EB is handled gracefully (no fatal protocol violation / connection teardown).
Code touchpoints
leios-fetch EB serving / lookup (in-memory cache lookupLeiosEndorserBlock; backfill in ouroboros/leios_backfill.go) — origin of leios endorser block not found.
The path that escalates a not-found EB to a connection-killing protocol violation.
Summary
On
musashi(magic 164), a dingo relay that has fully synced to tip cannot re-servehistorical endorser blocks to a downstream dingo node syncing from it. When the downstream
requests an EB the relay no longer has, the relay raises
protocol error: leios endorser block not found: <slot>.<hash>, which is treated as a protocol violation and tears down the wholeNtN connection — tripping the downstream's short-lived-connection backoff (escalates to ~2m).
Net effect: a downstream dingo node (e.g. an operator's block producer) can't obtain the Leios
region from its own upstream dingo relay; it must fetch EBs from the IOG prototype relay directly.
There are two issues bundled here: (a) the node lacks/doesn't backfill historical EB bodies to
serve them, and (b) a not-found EB is escalated to a fatal protocol violation rather than handled
gracefully.
Current behavior
Both nodes dingo
main(cbd517f, incl. #2654),--run-mode leios, core storage. Upstreamsynced to tip (~block 121k); downstream a fresh/behind node catching up over NtN. Upstream
(serving) logs, ~6 teardowns in 40 min during catch-up:
Downstream side: connection resets right after selecting the relay as best peer, then
short-lived connection detected, applying backoff(8s → 32s → 2m8s). The downstream stilladvances via chainsync between resets (and through EB-light stretches), so catch-up limps along —
but it cannot rely on the relay for the EB region.
Analysis
EBs are served from an in-memory cache; historical EBs the node didn't recently handle are not
present and are not backfilled-to-serve, so the lookup returns not-found. Two problems:
to its downstream peers (this is the serving-side mirror of the fetch-side gap in leios: endorser-block backfill stalls a from-genesis sync at the EB-region onset ("leios-fetch served no new transactions within tail budget") #2656).
instead of a graceful "don't have it" response — this trips the peer's instability backoff and
compounds the slowdown.
Distinct from #2630 (verifying dingo serving its own forged EBs) — this is about re-serving
historical EBs already on-chain. Same EB-availability root as #2656 (fetch side).
Reproduction
main,--network musashi --run-mode leios) synced to tip.protocol error: leios endorser block not found: <slot>.<hash>and theconnection torn down; on node B observe repeated
short-lived connection detected, applying backoffagainst node A.Acceptance
retain as needed), so a downstream can sync the Leios region from an upstream dingo relay.
Code touchpoints
lookupLeiosEndorserBlock; backfill inouroboros/leios_backfill.go) — origin ofleios endorser block not found.