Skip to content

fix(portal): unbounded sources anchor the backfill floor at genesis (C10, ports #8)#10

Merged
dzhelezov merged 1 commit into
mainfrom
fix/backfill-floor-c10
Jul 3, 2026
Merged

fix(portal): unbounded sources anchor the backfill floor at genesis (C10, ports #8)#10
dzhelezov merged 1 commit into
mainfrom
fix/backfill-floor-c10

Conversation

@dzhelezov

Copy link
Copy Markdown
Collaborator

Ports the backfill-floor fix (the one historical-path finding) from open PR #8 by @mo4islona onto current main. The other four findings in #8 are Portal stream-realtime hardening and stay with that PR.

The bug

An undefined fromBlock means "from genesis" — ponder's runtime reads filter.fromBlock ?? 0 throughout. But compileFetchSpec's backfillStart took Math.min over only the defined fromBlocks:

const froms = fs.map((f) => f.fromBlock).filter((b) => b != null);
const backfillStart = froms.length ? Math.min(...froms) : 0;

So a config mixing one bounded source (fromBlock: 15_000_000) with one unbounded source (no fromBlock) computed backfillStart = 15M. chunkRange (portal-chunks.ts) then clamps every chunk's low bound to >= 15M, so the unbounded source's [0, 15M) intervals are assembled empty and marked synced -- a permanent silent gap on the production historical path, losing the whole pre-15M history of the unbounded source.

The fix

If any filter omits fromBlock, the floor is 0 -- symmetric with the existing backfillEnd rule directly below it (any undefined toBlock => unbounded):

const froms = fs.map((f) => f.fromBlock);
const backfillStart =
  froms.length && froms.every((b) => b != null)
    ? Math.min(...(froms as number[]))
    : 0;

Re-expressed for the post-refactor compileFetchSpec (PR #8 predates the invariant-first re-architecture, where this lived inline in createPortalHistoricalSync). Added to the invariant catalog as INV-16 (Backfill-floor genesis semantics) in portal/INVARIANTS.md.

Tests (mutation-verified -- both fail on the pre-fix source)

  • portal-filters.test.ts -- INV-16/C10: backfillStart is 0 for an undefined fromBlock, min when all defined, 0 for a mixed config.
  • portal.test.ts -- C10 regression (ported from fix(portal): close silent-gap paths in backfill floor + stream realtime #8's C10 test): two log sources, one fromBlock: 15M, one undefined; the undefined source's block-100 log is fetched and inserted.

Mutation proof -- reverting only the source fix to the pre-fix Math.min-over-defined-only and re-running the two files in the built tree:

x portal-filters.test.ts > INV-16/C10 ...
  -> expected 15000000 to be +0
x portal.test.ts > regression: an undefined fromBlock is genesis ... (C10) ...
  -> expected [] to have a length of 1 but got +0   <- the silent gap
 Test Files  2 failed (2)
      Tests  2 failed | 27 passed (29)

Restoring the fix returns both files to green (29 passed).

Gate results

Full Portal suite via scripts/sync-upstream.sh <ver> --test (biome check + fresh clone + build + vitest), both compat.tested versions:

Version Result
0.16.6 15 files, 134 tests passed
0.15.17 15 files, 134 tests passed
  • biome check portal/ --diagnostic-level=error: clean (32 files, 0 errors).
  • Secret scan (grep -rnE 'prt_int_|prt_euler_|sqd_rpc_[A-Za-z0-9]{10}'): no matches.

Credit: original fix by @mo4islona in #8; this PR ports just the C10 backfill-floor finding, with a focused unit test added on compileFetchSpec.

Generated with Claude Code

…C10, ports #8)

An undefined `fromBlock` means "from genesis" (ponder's runtime reads
`filter.fromBlock ?? 0`), but `compileFetchSpec`'s `backfillStart` took
`Math.min` over only the DEFINED fromBlocks. A config mixing one bounded
source (e.g. fromBlock 15M) with one unbounded source therefore computed
backfillStart=15M; `chunkRange` clamped every chunk fetch to >=15M, and the
unbounded source's [0, 15M) intervals were assembled EMPTY and marked
synced -- a permanent silent gap on the production historical path.

Fix: if ANY filter omits fromBlock the floor is 0, symmetric with the
existing backfillEnd rule (any undefined toBlock => unbounded). Re-expressed
for the post-refactor compileFetchSpec; catalogued as INV-16.

Ports the backfill-floor finding from PR #8 (@mo4islona); the other four
stream-realtime findings stay with that PR.

Tests (both mutation-verified to fail on the pre-fix source):
- portal-filters.test.ts INV-16/C10: backfillStart is 0 for undefined,
  min for all-defined, 0 for mixed.
- portal.test.ts C10 regression: two sources (one fromBlock 15M, one
  undefined) -- the undefined source's early-range log IS fetched/inserted.

Gates green on both compat.tested versions (0.16.6, 0.15.17): 134 tests each.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@dzhelezov dzhelezov merged commit 05f34f0 into main Jul 3, 2026
4 of 8 checks passed
mo4islona added a commit that referenced this pull request Jul 3, 2026
…s 4–7)

Rebased onto post-refactor main. The backfill-floor fix (finding 3 / C10) landed
separately as #10 (INV-16), so this PR carries the four stream-realtime hardenings.
Each ships a regression test; INVARIANTS.md is updated where a contract changes.

- reconnect on child discovery (finding 4): a `logsRevision` bumped on each
  discovered child; `streamHotBlocks` snapshots it at open and re-opens with the
  widened server-side filter the moment it advances. `ndjsonLines` gained a
  `finally { void reader.cancel() }` (fire-and-forget) so the early break closes
  the old connection without slowing the historical hot path.

- refuse unsupported stream mode (finding 5): `assertStreamModeSupported` rejects
  non-log or receipt-requiring sources under PORTAL_REALTIME=stream. Asserted at
  process start (wiring's getLocalSyncProgress) AND at realtime-generator start.

- finalized-head probe fatal (finding 6, INV-9): in stream mode the RPC fallback
  is suppressed, so a persistently unknown Portal head can't locate the historical
  ↔realtime boundary. Both seams now fail loud — clampFinalizedToPortalHead retries
  then throws, and portal.ts throws on an unknown head — instead of passthrough /
  returning []. Changes the prior INV-9 "warn + empty" contract, deliberately.

- unknown-parent gap fatal (finding 7, INV-10): a reorg deeper than the unfinalized
  window (or a skipped block) now throws instead of silently clearing the window,
  which left already-indexed unfinalized blocks on the wrong fork with no rollback.
  Changes the prior INV-10 "gap resets" contract, deliberately.

INVARIANTS.md: INV-9/INV-10 rewritten; G4 (finding 6) and G5 (finding 7) added to
the fixed-gaps list.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dzhelezov added a commit that referenced this pull request Jul 3, 2026
…me (#8)

* fix(portal): close the four stream-realtime silent-gap paths (findings 4–7)

Rebased onto post-refactor main. The backfill-floor fix (finding 3 / C10) landed
separately as #10 (INV-16), so this PR carries the four stream-realtime hardenings.
Each ships a regression test; INVARIANTS.md is updated where a contract changes.

- reconnect on child discovery (finding 4): a `logsRevision` bumped on each
  discovered child; `streamHotBlocks` snapshots it at open and re-opens with the
  widened server-side filter the moment it advances. `ndjsonLines` gained a
  `finally { void reader.cancel() }` (fire-and-forget) so the early break closes
  the old connection without slowing the historical hot path.

- refuse unsupported stream mode (finding 5): `assertStreamModeSupported` rejects
  non-log or receipt-requiring sources under PORTAL_REALTIME=stream. Asserted at
  process start (wiring's getLocalSyncProgress) AND at realtime-generator start.

- finalized-head probe fatal (finding 6, INV-9): in stream mode the RPC fallback
  is suppressed, so a persistently unknown Portal head can't locate the historical
  ↔realtime boundary. Both seams now fail loud — clampFinalizedToPortalHead retries
  then throws, and portal.ts throws on an unknown head — instead of passthrough /
  returning []. Changes the prior INV-9 "warn + empty" contract, deliberately.

- unknown-parent gap fatal (finding 7, INV-10): a reorg deeper than the unfinalized
  window (or a skipped block) now throws instead of silently clearing the window,
  which left already-indexed unfinalized blocks on the wrong fork with no rollback.
  Changes the prior INV-10 "gap resets" contract, deliberately.

INVARIANTS.md: INV-9/INV-10 rewritten; G4 (finding 6) and G5 (finding 7) added to
the fixed-gaps list.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: drop stray .idea/ IDE files; gitignore .idea/

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Dmtirii Zhelezov <dzhelezov@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant