stream realtime: canonical block fataled with 'unknown parent' — parent was the canonical block at N-1 (no reorg involved)

[dzhelezov]

Observation (long-running stream-mode soak, chain 1 / ethereum)

The stream-realtime client hit its fail-loud unknown-parent path and exited (75/TEMPFAIL, by design; the supervisor restarted it and crash recovery resumed cleanly ~11s later, gap backfilled):

Error: Portal realtime: streamed block 25455459 (0xd481da308324da8bfbde16a9b99975d90c977e76e10d54dba037d82d0fb914cd) has an unknown parent 0x49f95d8bcca2c957d569df3e82f3588f8de68357a8d1a722409edd34e1f6e0ef — a reorg deeper than the unfinalized window, or a skipped block. Cannot reconcile safely; restart to re-sync from the finalized head.

Third-party check: no reorg — both blocks are canonical

Queried a public RPC (ethereum-rpc.publicnode.com) minutes after the event:

• eth_getBlockByNumber(25455458) → hash 0x49f95d8bcca2c957d569df3e82f3588f8de68357a8d1a722409edd34e1f6e0ef — exactly the "unknown parent".
• eth_getBlockByNumber(25455459) → hash 0xd481da308324da8bfbde16a9b99975d90c977e76e10d54dba037d82d0fb914cd, parent 0x49f95d… — exactly the streamed block.

So the error message's first hypothesis (deep reorg) is ruled out for this instance: the stream delivered the canonical N, and the canonical N-1 was missing from the client's unfinalized window. Either the stream never delivered N-1 (server-side skip / delivery gap) or the client received and then lost it (window-state bug: eviction, finalize consuming it early, or the redelivery handshake missing a case).

Impact

Availability, not correctness: the fail-loud path worked as designed, restart + crash recovery healed the gap with no data loss observed. But every such event is a full process restart; a recurring skip shape would turn into a restart churn.

Notes

• PR fix(portal): stream-realtime correctness wave — same-block child logs, finality anchor, reorg prune, head pin, parent txs #26 rewrites exactly this seam (finality anchor, redelivery handshake, wire state machine) — this instance is useful field evidence for its review: whatever the root cause, the reworked window logic should make this shape either impossible or reconcilable without a restart (e.g. by fetching the missing parent by hash before declaring the window unreconcilable).
• The same process had the issue stream realtime: sleep() leaks an abort listener per call on the long-lived stream signal (MaxListenersExceededWarning storm) #28 abort-listener leak storm running in the minutes before the fatal (three signals at ~24k/3.6k/1.7k listeners) — believed unrelated to the parent gap, noted for completeness.
• Single instance observed so far (~2h into that process's uptime); will track recurrence.

[dzhelezov]

Second occurrence — same instance, ~24 minutes (120 blocks) after the first, again on ethereum mainnet:

• streamed block: 25455579 0x35fd94cd29e396049940f36c40170597d8731f1e43528e346a2770c88c70589d
• rejected "unknown" parent: 0x31bee29d3ff459ea71de5c7171c8bc2198c48cf0d0f4c8ed1a0d471ba7b9fb0d
• third-party check (publicnode, by number): 25455579 is canonical with exactly that hash, and canonical 25455578 is the rejected parent — identical shape to the first instance: no reorg involved, the client was simply missing canonical N-1.
• fatal site identical (portal-realtime.ts:246 unknown-parent guard); clean crash-recovery + gap backfill on restart, as before.

Two instances 24 minutes apart upgrades this from rare anomaly to a recurring pattern: whatever drops canonical N-1 from the window (stream skip or window-state loss) happens at a rate that matters for stream-mode availability. PR #26 rewrites exactly this seam — its landing priority just went up.

[dzhelezov]

Third instance, same shape (2026-07-04 01:46:29Z):

• streamed block: ethereum 25455961 0x9d6e209b09afab68f9ce0a646014f19d79a4f4126f6488bb6d3e2602ce67f14d
• "unknown parent": 0xb6c6be3fe33b609eb10ecf941b99c8f6fb0cb656ee605cdee9a6190abe74e32f
• third-party check (publicnode eth_getBlockByNumber): the streamed block IS canonical at 25455961, and the "unknown" parent IS the canonical block at 25455960. No reorg involved — the client was again missing canonical N-1.

Spacing between instances: #1 block 25455459, #2 25455579 (+120 blocks, ~24 min), #3 25455961 (+382 blocks, ~77 min) — recurring but not periodic, so a strictly cadence-driven trigger looks unlikely.

As with the prior two, the pre-crash log shows the issue #28 abort-listener storm (≈15,150 listeners on one AbortSignal at ~77 min uptime, ≈3.3/s — above the 1/s poll-only estimate, consistent with multiple sleep() call sites on the hot path). Recovery was clean again (crash-recovery resume, gap backfilled from the finalized head).

Three fatals in ~1.7 h on one chain upgrades this from anomaly to an availability defect for stream mode. PR #26's window/redelivery rework touches exactly this seam; if the unknown-parent path survives that rework, a bounded fetch-parent-by-hash reconciliation (fail-loud on mismatch) looks like the natural follow-up.

[dzhelezov]

Fourth instance, same shape (2026-07-04 07:29:52Z) — and the first on a build that includes the PR #26 stream rework (built from 19b1d58), so the rework narrows but does not eliminate this shape (fatal raised at portal-realtime.ts:401 via portal-realtime-wire.ts:461):

• streamed block: ethereum 25457676 0x0882e002418514469c84dfee8460fbb3056c493ed7152fca6d91db9ce88fb84c
• "unknown parent": 0x3d9c37292f268240d898e23c0695614af676ac5a84b848ba7877300f32cea33d
• third-party check (publicnode eth_getBlockByNumber): the streamed block IS canonical at 25457676, and the "unknown" parent IS the canonical block at 25457675. Same no-reorg-visible-on-the-canonical-chain shape as instances 1–3.

New data point — the client indexed a block at N−1 twenty-four seconds before the fatal: Indexed block chain=ethereum number=25457675 at 07:29:28Z, with blocks arriving at a steady ~12s cadence, and the finalized head at 25457591 (84 blocks below N−1) — so finalize-time eviction of the parent cannot be the mechanism. The indexed N−1's hash is not recoverable post-recovery (crash recovery rewinds to the finalized head). But if stream-mode indexing implies the block passed the same parent-link check that fataled here, then the window's 25457675 must have been a non-canonical sibling of the canonical one — i.e. the stream delivered an orphaned N−1, then the canonical N, without (re)delivering the canonical N−1. That would make this a rollback/redelivery gap at a 1-block micro-reorg rather than a lost-block bug. Verifying that implication against the current code is the next root-cause step.

Recovery was again clean (exit 75 → supervisor restart → crash recovery from cache, gap backfilled, no data loss observed). Spacing: instance 3 → 4 ≈ 5h43m.

Fix-shape note: whatever the root cause, the fatal path should dump the window's entry at N−1 (present/absent + hash) and the last few delivered block hashes before exiting — a cheap diagnostic that would let the next instance pin the mechanism definitively. Root-cause investigation in progress.

[dzhelezov]

Root cause identified: number-only resume cursor omits parentBlockHash — the client opts out of the Portal's fork negotiation

The instance-4 inference above is confirmed, and the mechanism is now pinned (high confidence, live-verified against the production Portal):

Crux proof. In the post-#26 code, every path that yields a block event (and hence logs Indexed block) is parent-linked into the window (portal-realtime.ts:87 append check; :81/:88/:92 anchor/window-entry checks; INV-10 tripwire at :423-435; the blind-append hole at :79 is unreachable because the wire always passes the startup anchor, portal-realtime-wire.ts:471-476). So the indexed 25457675 was a non-canonical sibling of canonical 25457675 — both children of canonical 25457674.

Mechanism. The /stream request body (portal-realtime.ts:195-219) carries only fromBlock + filters; resume state is cursor = header.number + 1 (:277). Per the SQD EVM API contract, an unfinalized-range consumer should send parentBlockHash so the server can detect a fork and answer HTTP 409 with previousBlocks (the canonical replacement chain). Verified live against the ethereum-mainnet Portal:

• POST /stream {fromBlock: H−2, parentBlockHash: 0x…deadbeef} → 409 {"previousBlocks":[{number, hash}, …]} ending at fromBlock−1.
• The same request without parentBlockHash → 200, data served — the server has no way to know the client's history is orphaned.
• At tip, responses are bounded and the connection closes in ~0.1s, so every inter-block boundary is a number-only resume boundary.

Instance-4 sequence, fully forced: (1) conn serves the orphan 25457675′ → indexed, cursor := 25457676, conn closes; (2) canonical 25457675 (0x3d9c…) replaces the orphan; the client polls fromBlock=25457676 → head is 25457675 < fromBlock → 204 — the only interval in which the canonical N−1 could be delivered is excluded by the cursor; (3) canonical 25457676 lands (~24s after the orphan — matches the log timing exactly) → parent ∉ window → the gap fatal at :401. Same cursor logic pre- and post-#26 (the #26 rework never touched it), which is why the rework didn't change the shape. Chain-1-only recurrence also fits: 4 events over ~2,217 tip blocks ≈ 0.18%, the order of ethereum's real 1-block orphan rate, while Base/Arbitrum single-sequencer feeds essentially never serve a same-height losing block.

Alternative mechanisms are excluded by code trace: no in-band rollback marker exists to mishandle (fork signaling is request-side via 409 — note the current deterministic-4xx branch at :240 would even treat a 409 as fatal-config); finalize eviction was 84 blocks below N−1; #26's reorg prune touches childAddresses, not the window; duplicate requires exact tip-hash equality.

Fix (PR incoming): protocol-native heal, confined to the I/O shell — keep a delivered-hash ring (last ~2048 heights, seeded with the startup anchor), send parentBlockHash: ring[cursor−1] on every request, handle 409 by rewinding the cursor to the highest previousBlocks entry matching the ring at/above the finalized floor (bounded step-down fallback; fatal below finality or after N consecutive 409s). Redelivered canonical blocks then flow through the existing reconcile as a normal reorg + appends — no new window machinery, #26's anchor/finalize-guard/redelivery handshake untouched, and the gap fatal stays as the terminal backstop so no block is ever accepted unlinked. A 1-block orphan at tip becomes reorg + 2 appends with no restart. The fatal path also gains a diagnostic dump (window entry at N−1, anchor, last ring entries, cursor, last 409 body) so any future instance is self-identifying. Regression tests will simulate the exact orphan-then-canonical sequence with per-connection body assertions.

[dzhelezov]

Instance 5 — 2026-07-04 10:18:33Z. Same fatal shape, but a new sub-variant that confirms the root cause from the opposite direction.

• Fataled block: streamed 25458514 0xc5b8ca9904683d440e42fd0fa4364d23d98383b9ca2a53f2cca2a0ae4d0b10a7, claiming parent 0xf27000f45de59aa8b47a2f2c5efa6ff93f3efce3b7e588cfc289ef5d5bbb29c9 (unknown to the client).
• Third-party check (publicnode): canonical 25458514 is 0x9c64fb2dadfc76c647ccd0bf9a2ab7e2ccfcfbf5603df7a7dd912319d9ab80ed, whose parent is canonical 25458513 0x05ac3f2472775d265f2b1e863f280da0d27663e3a427050fcc9e295ee08c3c49.
• So unlike instance 4 (a canonical block whose canonical parent was missing from the client window), here the streamed block and its claimed parent are both non-canonical: the stream handed the client an orphan-branch block whose parent it had never delivered. The client had indexed a block at 25458513 ~27s earlier (10:18:06Z); the resume at 25458514 was served from a different branch.

This is the mirror image of instance 4 and closes the residual mechanism question from the root-cause analysis above: with number-only resume, branch selection at every resume boundary is entirely server-side, in both directions — the client can be left behind on an orphan (instance 4) or handed one it never asked for (instance 5). Neither is healable without a hash anchor on resume.

PR #46's parentBlockHash + 409 negotiation covers both shapes: the client's anchor either matches the serving branch (append continues) or the server responds 409 → rewind to the fork point → re-stream, converging on whatever branch the server currently serves, with a follow-up 409/rewind if it switches back. A transient reorg instead of a process exit.

Recovery as with all prior instances: clean crash recovery on restart (~11s), no data loss observed.