fix(portal): unbounded sources anchor the backfill floor at genesis (C10, ports #8)#10
Merged
Merged
Conversation
…C10, ports #8) An undefined `fromBlock` means "from genesis" (ponder's runtime reads `filter.fromBlock ?? 0`), but `compileFetchSpec`'s `backfillStart` took `Math.min` over only the DEFINED fromBlocks. A config mixing one bounded source (e.g. fromBlock 15M) with one unbounded source therefore computed backfillStart=15M; `chunkRange` clamped every chunk fetch to >=15M, and the unbounded source's [0, 15M) intervals were assembled EMPTY and marked synced -- a permanent silent gap on the production historical path. Fix: if ANY filter omits fromBlock the floor is 0, symmetric with the existing backfillEnd rule (any undefined toBlock => unbounded). Re-expressed for the post-refactor compileFetchSpec; catalogued as INV-16. Ports the backfill-floor finding from PR #8 (@mo4islona); the other four stream-realtime findings stay with that PR. Tests (both mutation-verified to fail on the pre-fix source): - portal-filters.test.ts INV-16/C10: backfillStart is 0 for undefined, min for all-defined, 0 for mixed. - portal.test.ts C10 regression: two sources (one fromBlock 15M, one undefined) -- the undefined source's early-range log IS fetched/inserted. Gates green on both compat.tested versions (0.16.6, 0.15.17): 134 tests each. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 3, 2026
mo4islona
added a commit
that referenced
this pull request
Jul 3, 2026
…s 4–7) Rebased onto post-refactor main. The backfill-floor fix (finding 3 / C10) landed separately as #10 (INV-16), so this PR carries the four stream-realtime hardenings. Each ships a regression test; INVARIANTS.md is updated where a contract changes. - reconnect on child discovery (finding 4): a `logsRevision` bumped on each discovered child; `streamHotBlocks` snapshots it at open and re-opens with the widened server-side filter the moment it advances. `ndjsonLines` gained a `finally { void reader.cancel() }` (fire-and-forget) so the early break closes the old connection without slowing the historical hot path. - refuse unsupported stream mode (finding 5): `assertStreamModeSupported` rejects non-log or receipt-requiring sources under PORTAL_REALTIME=stream. Asserted at process start (wiring's getLocalSyncProgress) AND at realtime-generator start. - finalized-head probe fatal (finding 6, INV-9): in stream mode the RPC fallback is suppressed, so a persistently unknown Portal head can't locate the historical ↔realtime boundary. Both seams now fail loud — clampFinalizedToPortalHead retries then throws, and portal.ts throws on an unknown head — instead of passthrough / returning []. Changes the prior INV-9 "warn + empty" contract, deliberately. - unknown-parent gap fatal (finding 7, INV-10): a reorg deeper than the unfinalized window (or a skipped block) now throws instead of silently clearing the window, which left already-indexed unfinalized blocks on the wrong fork with no rollback. Changes the prior INV-10 "gap resets" contract, deliberately. INVARIANTS.md: INV-9/INV-10 rewritten; G4 (finding 6) and G5 (finding 7) added to the fixed-gaps list. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dzhelezov
added a commit
that referenced
this pull request
Jul 3, 2026
…me (#8) * fix(portal): close the four stream-realtime silent-gap paths (findings 4–7) Rebased onto post-refactor main. The backfill-floor fix (finding 3 / C10) landed separately as #10 (INV-16), so this PR carries the four stream-realtime hardenings. Each ships a regression test; INVARIANTS.md is updated where a contract changes. - reconnect on child discovery (finding 4): a `logsRevision` bumped on each discovered child; `streamHotBlocks` snapshots it at open and re-opens with the widened server-side filter the moment it advances. `ndjsonLines` gained a `finally { void reader.cancel() }` (fire-and-forget) so the early break closes the old connection without slowing the historical hot path. - refuse unsupported stream mode (finding 5): `assertStreamModeSupported` rejects non-log or receipt-requiring sources under PORTAL_REALTIME=stream. Asserted at process start (wiring's getLocalSyncProgress) AND at realtime-generator start. - finalized-head probe fatal (finding 6, INV-9): in stream mode the RPC fallback is suppressed, so a persistently unknown Portal head can't locate the historical ↔realtime boundary. Both seams now fail loud — clampFinalizedToPortalHead retries then throws, and portal.ts throws on an unknown head — instead of passthrough / returning []. Changes the prior INV-9 "warn + empty" contract, deliberately. - unknown-parent gap fatal (finding 7, INV-10): a reorg deeper than the unfinalized window (or a skipped block) now throws instead of silently clearing the window, which left already-indexed unfinalized blocks on the wrong fork with no rollback. Changes the prior INV-10 "gap resets" contract, deliberately. INVARIANTS.md: INV-9/INV-10 rewritten; G4 (finding 6) and G5 (finding 7) added to the fixed-gaps list. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: drop stray .idea/ IDE files; gitignore .idea/ Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Dmtirii Zhelezov <dzhelezov@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports the backfill-floor fix (the one historical-path finding) from open PR #8 by @mo4islona onto current
main. The other four findings in #8 are Portal stream-realtime hardening and stay with that PR.The bug
An undefined
fromBlockmeans "from genesis" — ponder's runtime readsfilter.fromBlock ?? 0throughout. ButcompileFetchSpec'sbackfillStarttookMath.minover only the definedfromBlocks:So a config mixing one bounded source (
fromBlock: 15_000_000) with one unbounded source (nofromBlock) computedbackfillStart = 15M.chunkRange(portal-chunks.ts) then clamps every chunk's low bound to>= 15M, so the unbounded source's[0, 15M)intervals are assembled empty and marked synced -- a permanent silent gap on the production historical path, losing the whole pre-15M history of the unbounded source.The fix
If any filter omits
fromBlock, the floor is0-- symmetric with the existingbackfillEndrule directly below it (any undefinedtoBlock=> unbounded):Re-expressed for the post-refactor
compileFetchSpec(PR #8 predates the invariant-first re-architecture, where this lived inline increatePortalHistoricalSync). Added to the invariant catalog as INV-16 (Backfill-floor genesis semantics) inportal/INVARIANTS.md.Tests (mutation-verified -- both fail on the pre-fix source)
portal-filters.test.ts-- INV-16/C10:backfillStartis0for an undefinedfromBlock,minwhen all defined,0for a mixed config.portal.test.ts-- C10 regression (ported from fix(portal): close silent-gap paths in backfill floor + stream realtime #8's C10 test): two log sources, onefromBlock: 15M, one undefined; the undefined source's block-100 log is fetched and inserted.Mutation proof -- reverting only the source fix to the pre-fix
Math.min-over-defined-only and re-running the two files in the built tree:Restoring the fix returns both files to green (29 passed).
Gate results
Full Portal suite via
scripts/sync-upstream.sh <ver> --test(biome check + fresh clone + build + vitest), bothcompat.testedversions:0.16.60.15.17biome check portal/ --diagnostic-level=error: clean (32 files, 0 errors).grep -rnE 'prt_int_|prt_euler_|sqd_rpc_[A-Za-z0-9]{10}'): no matches.Credit: original fix by @mo4islona in #8; this PR ports just the C10 backfill-floor finding, with a focused unit test added on
compileFetchSpec.Generated with Claude Code