leios: from-genesis sync on Musashi is very slow through the tx-dense region (~15 h to near-tip) — CPU-pegged "utxo not found" repair loop

[chadaibbhot-git]

Environment

• Dingo ghcr.io/blinklabs-io/dingo:0.65.0 (commit 1f6b6a4) and :0.64.0 (commit 420c9ecf) — reproduced on both official releases. (v0.65.1, released the same day, does not touch the block-application/UTxO path — its only ledger/ change is a chainsync rollback-loop fix — so this is expected to persist.)
• network musashi (Leios testnet, magic 164), serve --network musashi --run-mode leios, fresh from genesis (empty DB), core storage.
• Reference: a fully-synced IOG prototype cardano-node 11.1.0.164 on the same network syncs this chain to tip without issue.

Summary

A dingo node syncing Musashi from genesis takes 15+ hours and struggles to reach tip. Almost all of that time is spent in the tx-dense region (epochs ~6–12, produced during network load testing), where dingo enters a CPU-pegged per-block loop and applies blocks at a tiny fraction of its early-epoch rate. The IOG Haskell node syncs the same blocks fine, so the chain is valid — this is a dingo block-application performance problem.

Evidence (real numbers, this run)

Two speed regimes, genesis sync of the same chain:

region	rate	note
genesis → ~epoch 6 (empty blocks)	~4,600 slots/s	flies
epochs ~6–12 (tx-dense)	~24 blocks/min	grinds; multi-minute stalls per block

• Genesis → epoch 6 in ~5 min; epochs 6–12 then took ~6+ hours; total wall time to near-tip ~15 h and counting (still ~20 h behind tip after ~18 h).
• During the grind: CPU pegged ~108–148%, and a flood of
  msg="skipping unrecoverable transaction input utxo repair" error="utxo not found"
  (~1,300–2,000 lines per 30 s) while the ledger tip is frozen on a single block.
• v0.65.0's eb fetch stall failover (fix(leios): eb fetch stall failover #2844) did not help this grind — expected, since fix(leios): eb fetch stall failover #2844 targets the fetch stall, whereas this is a block-application / UTxO-resolution grind. The v0.64.0...v0.65.0 diff touches only EB fetch/backfill, not the ledger apply path.
• Near tip, the node also hit the restart-recoverable blockfetch stall described in leios: from-genesis catch-up stalls for minutes when a relay is slow to serve a referenced endorser block; add per-request fetch deadline and failover #2819 — the ledger tip froze (~20 h behind) with no repair-loop activity, and a container restart let it resume. (Referenced for context, not re-filed.)

Controlled A/B: dingo vs IOG Haskell on identical hardware

To rule out hardware/config/network as the cause, we ran both nodes side by side:

• Two identical VMs on one hypervisor host (same CPU, 4 vCPU, 16 GB each).
• Byte-identical Musashi config on both (md5-verified: topology.json + all 5 genesis files + config.json + peer-snapshot.json).
• Started from genesis at the same instant. dingo:0.65.0 (core) vs IOG cardano-node 11.1.0.164 (prototype-2026w28).

Per-epoch sync time (wall seconds to complete each epoch):

epoch	Haskell	dingo	dingo slower
0–5 (empty)	16–33 each	32–65 each	~2×
6	107	599	5.6×
7	214	1,551	7.2×
8	168	2,027	12×
9	314	(still grinding at cutoff)
10	335
11	2,135
12	2,860
13	321

Headline: the IOG Haskell node completed genesis → tip in 123 minutes (peak 2.3 GB RAM, 0 errors). At that same 123-minute mark, dingo was still on epoch 9 of 16 — it had not yet finished the tx-dense region, with its densest epochs (10–12, where Haskell spent 36 and 48 min) still ahead. Dingo is ~2× slower even on the empty early epochs, then 6–12× slower per tx-dense epoch, and its per-epoch time keeps climbing as density rises.

Resources / errors over the run:

	Haskell	dingo
RAM (steady)	~1.0–1.4 GB	~3.7 GB (≈3.4×)
errors	0	utxo_not_found (repair loop, climbing), blockfetch_timeout

Same blocks, same hardware, same config — Haskell applies them cleanly; dingo enters the CPU-pegged utxo not found repair loop. This isolates the problem to dingo's block-application path.

Why this matters

When a dingo node has to sync from genesis (a fresh node, or after the chain is reset), this slow behavior is currently the only option — there is no working fast-bootstrap to fall back on:

• Mithril (dingo's built-in fast-bootstrap) can't be used here: a self-hosted aggregator's chain observer can't handshake the Leios prototype node (PallasChainObserver … handshake version not accepted, tested on mithril 0.8.47 and 0.9.17), so no snapshot can be produced for this network.
• A raw dingo DB copy works within a single dingo version but is not portable across dingo releases (badger/sqlite schema), so it isn't a durable fallback.

Reproduction

1. dingo:0.65.0, fresh genesis, serve --network musashi --run-mode leios.
2. Watch the tip advance quickly to ~epoch 6, then crawl for hours through the tx-dense epochs with CPU pegged and utxo not found / unrecoverable transaction input utxo repair flooding the log.

Asks

1. Speed up block application through the tx-dense region — the per-input utxo not found "repair" loop appears pathological (the IOG node applies the same blocks without it). This is the primary fix.
2. Ideally, a version-stable fast-bootstrap for Musashi — e.g., Mithril chain-observer support for the Leios prototype (so a self-hosted aggregator can snapshot the immutable DB and dingo can import it). Happy to test.

Happy to provide full logs, timing traces, or run a patched build against the live network.