cross-validation: RPC-realtime store missing on-chain logs of long-established addresses at scattered recent blocks — (block,address)-scoped, third-party confirmed; Portal store complete

[dzhelezov]

Found by the hourly A/B differ (the strict onlyB class added in #13): on chain 1, 89 log rows + 18 block rows are present in the Portal-leg store but absent from the RPC-realtime-leg store, inside the finalized overlap. Per-block sampling over [25455000, 25456100] shows 10 affected heights with clusters of 3–6 logs each.

Decisive third-party-confirmed sample (publicnode eth_getTransactionReceipt):

• tx 0xea181850e9fcd5c0ad7ae89ca92c10472ffd2d58918342f003183f9f7dc009fe, block 25455946
• the on-chain receipt carries logs at indices 989/990/992 from 0x61129032f2309d0bb73e00cc3ae411e4cbd9d95f (topic0s match the Portal-leg rows byte-for-byte)
• the RPC-realtime store has the SAME transaction's logs at indices 997–1006 (other addresses) but not 989/990/992 — the loss is (block, address)-scoped and cuts through the middle of a single receipt.

This is not new-child discovery lag. Every affected address is long-established in BOTH stores — e.g. 0x611290… first seen at block 24,986,184 with 745 logs stored (739 in the lossy leg); two others first seen at 22,681,265 with ~6.2–6.4k logs each. At another affected height (25455045) the RPC-realtime leg has ZERO rows while the chain carries 6 relevant logs from two of these established addresses.

Mechanism open, two candidates:

1. Provider-served incomplete realtime responses — the same leg's provider previously produced realtime (RPC mode): deterministic crash when a block's full-block eth_getLogs response exceeds viem's 10MiB body cap #23 (>10MiB full-block eth_getLogs bodies) and realtime (RPC mode): access_list stored as NULL for realtime-ingested txs — including txs with real access lists; Portal /stream preserves them #27 (accessList stripped from full-tx blocks, provider-confirmed), so response-integrity issues on this path have precedent.
2. A client-side realtime handling defect (refetch/reorg path dropping part of a block's logs).

The Portal leg is chain-true at every sampled point (its rows match the on-chain receipt exactly). Notably, the same-block-child log-loss class that #26 just fixed in the fork's stream path shows that per-address realtime log loss is an easy defect class to have; whatever the mechanism here, the RPC-realtime path has no integrity check that would catch it — it trusts the provider's response completeness. Cross-validation is currently the only detector.

Repro (against two independently-synced stores):

-- per-block log-count diff over a recent window, chain 1
select block_number, count(*) from ponder_sync.logs
where chain_id=1 and block_number between 25455000 and 25456100 group by 1;
-- run on both stores and diff; then eth_getTransactionReceipt any missing row's tx hash

Follow-ups:

• the hourly differ will (correctly) FAIL on this class until it is either tolerated-with-issue-link (mirroring the realtime (RPC mode): access_list stored as NULL for realtime-ingested txs — including txs with real access lists; Portal /stream preserves them #27 tolerated-class pattern: measured, per-chain, loud, removable) or the leg is repaired/re-backfilled;
• mechanism investigation (provider response replay vs client trace) to split candidates 1/2;
• VALIDATION.md entry: fabricated-empty (cross-validation finding: single tx with fabricated-empty access_list ([]) in one of two independently-synced stores (chain 42161) #32) and dropped-row (this) classes are both invisible to single-store checks — cross-validation catches what presence checks cannot.

[dzhelezov]

Root cause identified — fork-side (the deployed build predates two merged fixes), not the RPC provider. Correcting this issue's framing: the loss is also not RPC-realtime-specific — the same blind state flowed through a Portal catch-up backfill on the affected deployment.

Leg A (the lossy store) runs 0.16.6-sqd.1. That build has two interacting defects, both already fixed on main:

1. Discovered factory children were never persisted to the sync store — leg A's ponder_sync.factory_addresses has zero rows for the lifetime of the store (physical stats: zero tuples ever inserted). Fixed by fix(portal): persist discovered factory children to the sync store #2 (merged ~24h after leg A's build was cut).
2. The restart re-discovery scan floor latches to the uncached tail's chunk boundary, not the factory deploy block — with intervals cached to near-head, a restart re-discovers children only from the tail chunk's floor. Fixed by fix(portal): correctness wave 2A — head-clamp, discovery floor, extend-local checks, trace receipts, head-pin #15 (wave-2A FIX 2; regression test portal/portal.test.ts:2142).

Consequence: at leg A's crash restart (Jul 3 20:48Z — the #23 crash), its in-memory child registry = (empty-store seed) ∪ (scan from the 500k chunk floor):

• chain 1: floor 25,000,000 → blind to 810 of 851 children
• chain 8453: floor 48,000,000 → blind to all active children
• chain 42161: floor 480,000,000 → blind to all 137

Evidence this is client-side (high confidence):

• The kept/lost partition is exact on child registration block: every lost address registered <25,000,000 (e.g. 0x611290… @ 24,986,184); every kept address registered ≥25,051,936. The gap between highest-lost and lowest-kept registration contains exactly one multiple of the 500k discovery chunk: 25,000,000. Response truncation cannot produce a partition keyed on registration block.
• The same blind registry drives both transports: the post-restart Portal catch-up backfill (one Postgres transaction) rewrote blocks 25454365/25454369 dropping exactly the addresses leg B holds. A provider artifact cannot reproduce identically across two independent transports.
• Leg B acquired every affected row via its live stream at head (Postgres txid ordering interleaves with chain time); no later repair involved.
• Leg A's own log shows one clean single-pass index of each affected block ~3s after chain time with event_count = exactly the kept subset — no retries, no reorg handling. The short set was manufactured client-side by the blind matcher.

True blast radius (the hourly differ under-reported it): chain 1: 95 logs / 19 blocks; chain 8453: ~476 logs / 139 blocks; chain 42161: ~33 logs / 8 blocks — growing while leg A stays blind. The differ's finalized-overlap window pins to min(max(A), max(B)) − margin, so when the lossy leg stops persisting, the window freezes and divergence above it is invisible (chain 8453 reported PASS for ~17h while leg A stored nothing new). Filed separately as a differ hardening issue.

Initial suspicion of provider-served partial responses is refuted for this issue. (The same deployment does log phantom-reorg lines on chains that effectively never reorg — a separate RPC-quality observation, unrelated to this loss mechanism.)

Repair: a plain restart cannot heal leg A — the blind-synced ranges are marked complete in ponder_sync.intervals, and the app tables indexed the incomplete event set. Repair = rebuild the store on a fixed build, or surgically delete interval coverage above the last-good block per chain (chain 1: 25,454,299 / chain 8453: 48,139,256 / chain 42161: 480,064,117) and re-backfill. Leg A's data past those points is unusable as evidence until repaired.

User-facing: the published 0.16.6-sqd.1 and 0.15.17-sqd.1 revisions carry this behavior — factory-config apps silently lose child-address coverage after any restart with a warm interval cache. Fixed on main (#2 + #15); cutting a fixed release revision is the follow-up.

Cross-validation earning its keep: this class is invisible to single-store checks, to the process's own logs, and to spot receipts — only the A/B diff caught it.

[dzhelezov]

Progression update from the hourly finalized-overlap cross-validation (2026-07-04 12:22Z run) — the loss is progressing and now surfaces in a third table.

Chain 1 (ethereum):

• B-only rows (present in the Portal-stream store, absent from the RPC-realtime store) grew 239 → 382 this run: 336 logs + 46 whole blocks, heights 25454313–25458699.
• New: the transactions table. 30 transactions in the Portal store are absent from the RPC-realtime store. Verified directly against both stores: at 27 affected heights the RPC-realtime store has zero blocks, zero logs, and zero transactions, while the Portal store holds all of them (27 blocks / 238 logs / 30 txs). All 30 fall inside the same lost span (25456456–25458699), whose upper bound coincides exactly with the logs/blocks loss above — one event, three tables.

Context: the RPC-realtime leg's persistence stalled and then recovered during this window (newest-row skew went from ~3 h back to ~90 s); the missing heights sit inside the stall/recovery window, so the loss pattern is consistent with rows being skipped rather than backfilled during recovery. The Portal-stream store remains self-consistent and complete over the same span.

Housekeeping: the cross-validation tolerance for this issue currently covers logs and blocks only, so the transactions facet reported as an unexpected divergence this run. Extending the tolerance to transactions with a strict predicate (only txs whose block is wholly absent from the RPC-realtime store) so the hourly verdict keeps its signal for genuinely new divergence classes — the underlying finding stays open and tracked here.

[dzhelezov]

Progression update (differ run ~13:50Z, 2026-07-04): the loss is not bounded by the stall/recovery window — leg A is still dropping isolated blocks after row persistence recovered (~12:22Z).

• Chain 1 B-only (wholly-absent-in-A) span now reaches 25459346, past the previously recorded upper bound 25458699. Count: 51 blocks (min 25454313).
• The new missing heights — 25458877, 25458946, 25459121, 25459288, 25459346 — are scattered single blocks deep inside leg A's populated range (A head 25459480 at probe time), at heights leg A indexed after its recovery. Observed rate since recovery ≈ 5 blocks / 90 min.
• Facet counts this run: logs 354, blocks 51, transactions 35 (the transaction facet now folds into this class via fix(harness): ab-diff — extend the issue-#36 tolerance to transactions (wholly-absent-block predicate) #49's wholly-absent-block predicate; a B-only tx whose block exists in A would still fail loud).

So this is an ongoing, low-rate loss mode on the RPC-realtime leg, not a one-off recovery artifact. Will keep tracking growth hourly.

[dzhelezov]

Disposition for the release wave — root cause fixed on main, closing

The root cause here is fully identified and fork-side / client-side (confirmed in the analysis above), not a provider artifact: leg A ran 0.16.6-sqd.1, which predates two interacting fixes. With factory intervals cached to near-head, a restart re-seeded the in-memory child registry only from the tail chunk's floor, so every child registered below that floor went unmatched — its logs/blocks/txs were silently dropped while the interval was still marked complete. The kept/lost partition is exact on child registration block (a response-truncation artifact cannot produce that), and the same blind registry drove both transports (the post-restart Portal catch-up backfill reproduced the loss in a single Postgres transaction), which rules out a provider cause.

Fix set — merged on main and carried by this release cut (0.16.6-sqd.2 / 0.15.17-sqd.2 / 0.16.7-sqd.1):

• fix(portal): persist discovered factory children to the sync store #2 — discovered factory children are persisted to the sync store in the same transaction that marks the interval synced (INV-15), including on the RPC finality-delegation path.
• fix(portal): correctness wave 2A — head-clamp, discovery floor, extend-local checks, trace receipts, head-pin #15 — the restart re-discovery floor is pinned from spec.factories at construction time, so it no longer latches to the lowest uncached interval's chunk boundary; a warm-cache restart re-discovers from the factory deploy block.
• INV-15 hardening test (factory-range gate dependence) + FIX 2 floor-refinement overscan — follow-ups from PR #15 review #21 — factory-range gate-pin floor refinement, closing the early-fetch race where a shared chunk fetched by another filter's interval could run with the floor unset.

Regression tests (portal/portal.test.ts, green on 0.15.17 / 0.16.6 / 0.16.7):

• construction-time discovery-floor pin — the BUG-A scenario (child registered below the tail floor, its event never fetched) fails before the fix and passes after;
• INV-15 × INV-9 — children pre-discovered by the wide Portal scan are persisted even when their interval is delegated to RPC (the fallback dedupe-skips them, so the portal-side flush must run);
• restart seeded only with persisted children still fetches child logs from cached factory intervals (no re-discovery needed).

The dense-source / factory-child discovery kill-and-resume path is additionally re-accepted under a Poisson chaos campaign: store logically byte-identical to a clean baseline across 33 verified resumes, zero invariant violations (VALIDATION.md §4.4).

Affected-store guidance: 0.16.6-sqd.1 / 0.15.17-sqd.1 deployments with factory / account configs should upgrade to the -sqd.2 (or 0.16.7-sqd.1) revision. A store that was blind-synced after a warm-cache restart cannot self-heal by a plain restart — the incomplete ranges are marked complete in ponder_sync.intervals; repair means rebuilding on the fixed revision, or surgically deleting interval coverage above the last-good block per chain and re-backfilling.

Root cause understood, fix + regression tests on main and in the release cut, guidance documented. Closing. Cross-validation earned its keep on this one — the class is invisible to single-store checks, to the process's own logs, and to spot receipts.