fix(portal): unbounded sources anchor the backfill floor at genesis (C10, ports #8) (#10)

An undefined `fromBlock` means "from genesis" (ponder's runtime reads
`filter.fromBlock ?? 0`), but `compileFetchSpec`'s `backfillStart` took
`Math.min` over only the DEFINED fromBlocks. A config mixing one bounded
source (e.g. fromBlock 15M) with one unbounded source therefore computed
backfillStart=15M; `chunkRange` clamped every chunk fetch to >=15M, and the
unbounded source's [0, 15M) intervals were assembled EMPTY and marked
synced -- a permanent silent gap on the production historical path.

Fix: if ANY filter omits fromBlock the floor is 0, symmetric with the
existing backfillEnd rule (any undefined toBlock => unbounded). Re-expressed
for the post-refactor compileFetchSpec; catalogued as INV-16.

Ports the backfill-floor finding from PR #8 (@mo4islona); the other four
stream-realtime findings stay with that PR.

Tests (both mutation-verified to fail on the pre-fix source):
- portal-filters.test.ts INV-16/C10: backfillStart is 0 for undefined,
  min for all-defined, 0 for mixed.
- portal.test.ts C10 regression: two sources (one fromBlock 15M, one
  undefined) -- the undefined source's early-range log IS fetched/inserted.

Gates green on both compat.tested versions (0.16.6, 0.15.17): 134 tests each.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: dzhelezov

portal/INVARIANTS.md

@@ -3,7 +3,7 @@
 The `portal/` layer is organised around **explicit, provable invariants**. Every module has a functional
 core (pure, unit-tested) behind an imperative shell; the core asserts these invariants at runtime
 (`invariant("INV-…", …)` from `portal-invariant.ts`) and the test suite proves them with property-based
-tests (`fast-check`, seed-pinned for deterministic CI). IDs are stable (INV-1…INV-15) and grep-able
+tests (`fast-check`, seed-pinned for deterministic CI). IDs are stable (INV-1…INV-16) and grep-able
 across **doc ⟷ code ⟷ test**.
 
 **Runtime modes** (`PORTAL_CHECKS`, default `on`): `off` disables all checks; `on` runs the O(1) checks
@@ -30,6 +30,7 @@ Legacy `C#` tags from the pre-refactor comments are mapped where they existed.
 | **INV-12** | **Exactly-once insertion / stash lifecycle.** Per interval: logs inserted once (range-data), blocks/txs/receipts/traces once (block-data); a stash entry is created then consumed exactly once; `delegated` entries consumed once; stash stays bounded. An upstream range RETRY legitimately re-creates an entry: production (`on`) keeps the pre-refactor overwrite semantics (debug-logged); `strict` makes the double-set loud. | Double insertion corrupts; a leaked stash entry leaks memory; but a benign upstream retry must not crash production. | `portal.ts` inserts logs in range-data, block-data in block-data; stash `set` then `delete`. | `invariantStrict("INV-12", …)` on set (strict mode only); `delete` on take. | `portal-shell.test.ts` — "INV-12: the stash is consumed exactly once — a second syncBlockData returns undefined without re-inserting", "INV-12: 'on' mode keeps overwrite semantics for an upstream range retry; 'strict' makes it loud"; `portal.test.ts` (insert-exactly-once round-trips). |
 | **INV-13** | **Progress & cache bounds.** The stream cursor strictly advances BY CONSTRUCTION (`cursor = last + 1 ≥ cursor + 1` per batch; a 204 terminates — no runtime guard could ever fire, so none exists); the chunk cache never exceeds the in-service span + READAHEAD; **a rejected chunk promise is evicted immediately** so a later interval retries instead of replaying the cached rejection (fixes **G1**); **a FRONTIER chunk truncated at a then-lower finalized head is EXTENDED (tail-only stream + merge) before being served past its `coveredTo`** — never served stale (head-boundary bug found in review; fixed on main by PR #5/802ceed and RE-IMPLEMENTED here with behavior parity — main's own #5 regression test passes unchanged). A failed extend evicts the whole entry — never leaves the optimistic high-water in place. | A cached rejection permanently poisons a chunk; an unbounded cache OOMs; a blind idx cache hit over a head-truncated chunk marks an interval synced over a silent gap. | `readAheadPlan`/`evictionPlan` bound the cache (pure); `dataChunk` records `coveredTo` per cache entry and extends via the shared `runStreams` (append-only merge; rows accounted through the same per-fetch token); `p.catch`/`extended.catch` evict + free exactly once. | `dataChunk`'s INV-1/INV-9/INV-3 asserts bracket every fetch; `portal-client.ts` documents progress-by-construction at the cursor advance. | `portal-chunks.test.ts` — "INV-13: read-ahead plan is bounded (≤ readahead), within raEnd, and depth-1-only when saturated", "eviction plan…"; `portal-client.test.ts` — "INV-13: the cursor strictly advances and the stream terminates"; `portal.test.ts` — "regression: frontier chunk truncated at a lagging Portal head is EXTENDED…" (main's #5 test, verbatim); `portal-shell.test.ts` — "G1: a failed chunk is evicted (rows freed) and a later call refetches…" and "frontier extend streams ONLY the newly-finalized tail…". |
 | **INV-14** | **Config validity.** All `PORTAL_*` env is parsed ONCE into a frozen, validated `PortalConfig`; garbage values fail fast with actionable messages (previously `Number("abc") → NaN` silently poisoned the chunk grid). | A silent NaN in the chunk grid corrupts every downstream computation. | `loadPortalConfig` validates + `Object.freeze`s once. | `PortalConfigError` at parse. | `portal-config.test.ts` — "INV-14: garbage numeric → loud PortalConfigError (not silent NaN)" + the defaults/bounds/checks matrix. |
+| **INV-16** (C10) | **Backfill-floor genesis semantics.** `compileFetchSpec`'s `backfillStart` — the LOW-side clamp `chunkRange` (portal-chunks.ts) applies to every chunk fetch — is `0` whenever ANY source omits `fromBlock` (⇒ genesis), and only the `Math.min` of the DEFINED `fromBlock`s when they are ALL defined. Symmetric with `backfillEnd`, where any undefined `toBlock` ⇒ unbounded. | A source with no `fromBlock` starts at genesis (ponder's runtime reads `filter.fromBlock ?? 0`). Taking `Math.min` over only the DEFINED `fromBlock`s let a bounded source (e.g. 15M) raise the floor past an unbounded source's start; `chunkRange` then clamped every fetch to `≥ 15M` and the unbounded source's `[0, 15M)` intervals assembled EMPTY and were marked synced — a permanent silent gap on the PRODUCTION historical path. | `compileFetchSpec` sets `backfillStart = froms.every(defined) ? Math.min(froms) : 0` (mirrors `backfillEnd`); `portal.ts` closes over the frozen spec and `chunkRange(idx, chunkBlocks, backfillStart, end)` uses it as the low bound. | — (pure floor; the empty-assembly it prevents has no safe runtime signal, so it is proven by test). | `portal-filters.test.ts` — "INV-16/C10: an undefined fromBlock anchors backfillStart at genesis (0)" (undefined ⇒ 0, all-defined ⇒ min, all-undefined ⇒ 0); `portal.test.ts` — "regression: an undefined fromBlock is genesis — the [0, min) prefix of an unbounded source is NOT silently skipped (C10, ports #8 by @mo4islona)". Ported from PR #8 (@mo4islona). |
 | **INV-15** | **Factory-children persistence.** A factory interval marked cached ⇒ its children are persisted in the sync store: newly-discovered children are queued and flushed via `syncStore.insertChildAddresses` inside the SAME `syncBlockRangeData` whose transaction ponder commits with `insertIntervals`; the flush is scoped to children whose creation block ∈ the interval (cross-interval children stay queued for their owning interval); a failed flush restores the queue and fails the interval loud; restart-preloaded children re-discover at `prev ≤ bn` and are NEVER re-queued (no double-insert). Ported from PR #2 into this architecture. | Ponder's core marks `requiredFactoryIntervals` cached after every interval and on startup loads children ONLY from the store — an unpersisted child means every restart of a factory app silently drops that child's events. | `portal-discovery.ts` queues inside the min-merge guard (`prev === undefined \|\| prev > bn`) and exposes `takePendingInRange`/`restorePending`; `portal.ts` flushes after the interval's chunks resolve, before read-ahead/assemble/insert. | A flush failure rethrows (ponder rolls the interval back). | `portal.test.ts` — "regression: discovered factory children are persisted via insertChildAddresses in the SAME syncBlockRangeData call", "regression: a failing insertChildAddresses fails LOUD and the children re-flush on the next call…", "restart: a new sync seeded ONLY with the persisted children still fetches child logs…", "regression: each syncBlockRangeData persists ONLY its interval's children…", "regression: a post-flush insertLogs failure fails the interval LOUD…", "regression: a child already known from the store is NOT re-flushed when discovery re-runs live". |
 
 ## The fixed gaps
@@ -39,3 +40,4 @@ Legacy `C#` tags from the pre-refactor comments are mapped where they existed.
 - **G3 → INV-7.** Buffered-row accounting only counted completed chunks; in-flight NDJSON arrays were invisible to backpressure. Now the client's `onRows` hook registers rows per arriving batch against a per-fetch token. Mutation-verified: a no-op hook fails the mid-chunk test.
 - **Head-boundary staleness → INV-13** (found in review; fixed on main by PR #5, merged as 802ceed — re-implemented here with behavior parity incl. the `extends` metrics counter; main's #5 regression test passes unchanged). A frontier chunk fetched while the Portal head sat inside its grid range was cached truncated forever; when the head advanced, later intervals in the same chunk were served from the truncated cache and marked synced over a silent gap. Fixed with per-entry `coveredTo` + tail-only EXTEND. One deliberate divergence: on a FAILED extend, main leaves a cached rejection (its pre-existing G1 gap); here the entry is evicted and its rows freed — the G1-consistent recovery this PR introduces.
 - **Restart child loss → INV-15** (fixed on main by PR #2; ported into this architecture with all six of its regression tests passing unchanged).
+- **Backfill-floor silent gap (C10) → INV-16** (found + fixed by PR #8, @mo4islona; ported here onto `compileFetchSpec`). `backfillStart` took `Math.min` over only the DEFINED `fromBlock`s, so one bounded source (15M) raised the floor past an unbounded source and `chunkRange` clamped its `[0, 15M)` history out — assembled empty and marked synced. Now any undefined `fromBlock` ⇒ floor `0` (symmetric with `backfillEnd`). Mutation-verified: reverting the floor to `Math.min` over the defined-only list fails both the `portal-filters.test.ts` unit assertion (mixed ⇒ 0) and the `portal.test.ts` C10 regression (the unbounded source's block-100 log goes un-inserted).

portal/portal-filters.test.ts

@@ -194,6 +194,35 @@ test('INV-1 support: compileFetchSpec is deterministic in structure and frozen',
   expect(a.logQuery()).toEqual(b.logQuery());
 });
 
+test('INV-16/C10: an undefined fromBlock anchors backfillStart at genesis (0)', () => {
+  // A missing fromBlock means "from genesis" (ponder runtime reads `filter.fromBlock ?? 0`). If ANY
+  // source omits it the floor MUST be 0, else chunkRange clamps every fetch past the earliest DEFINED
+  // start and the unbounded source's [0, min) history is assembled empty and marked synced (silent gap).
+  const allDefined = compileFetchSpec(
+    [
+      { filter: logFilter({ fromBlock: 15_000_000 }) },
+      { filter: logFilter({ fromBlock: 8_000_000 }) },
+    ],
+    new Map(),
+  );
+  expect(allDefined.backfillStart).toBe(8_000_000); // all defined ⇒ min
+
+  const mixed = compileFetchSpec(
+    [
+      { filter: logFilter({ fromBlock: 15_000_000 }) },
+      { filter: logFilter({ fromBlock: undefined }) }, // genesis source
+    ],
+    new Map(),
+  );
+  expect(mixed.backfillStart).toBe(0); // ANY undefined ⇒ 0 (this line fails on the pre-fix Math.min)
+
+  const allUndefined = compileFetchSpec(
+    [{ filter: logFilter({ fromBlock: undefined }) }],
+    new Map(),
+  );
+  expect(allUndefined.backfillStart).toBe(0);
+});
+
 test('INV-5 support: the trace request carries NO trace filter (fetch-all)', () => {
   const trace: any = {
     type: 'trace',

portal/portal-filters.ts

@@ -356,10 +356,16 @@ export function compileFetchSpec(
 
   // the chain's actual backfill window, from the filters — used to bound chunk fetches so a bounded
   // backfill (or the tail) never over-fetches. Fully automatic; no client tuning.
-  const froms = fs
-    .map((f) => f.fromBlock)
-    .filter((b): b is number => b != null);
-  const backfillStart = froms.length ? Math.min(...froms) : 0;
+  // A source with NO fromBlock starts at genesis (ponder's runtime reads `filter.fromBlock ?? 0`); if
+  // ANY source omits it the floor MUST be 0 — symmetric with backfillEnd's undefined-⇒-unbounded rule
+  // below. Taking Math.min over only the DEFINED fromBlocks would clamp every chunk fetch past the
+  // earliest DEFINED start (chunkRange, portal-chunks.ts) and mark the [0, min) prefix of an unbounded
+  // source synced over a permanent silent gap — the whole pre-min history lost. (INV-16 / C10)
+  const froms = fs.map((f) => f.fromBlock);
+  const backfillStart =
+    froms.length && froms.every((b) => b != null)
+      ? Math.min(...(froms as number[]))
+      : 0;
   const tos = fs.map((f) => f.toBlock);
   const backfillEnd =
     tos.length && tos.every((t) => t != null)

portal/portal.test.ts

@@ -1653,3 +1653,128 @@ test('regression: frontier chunk truncated at a lagging Portal head is EXTENDED
     srv.close();
   }
 });
+
+test('regression: an undefined fromBlock is genesis — the [0, min) prefix of an unbounded source is NOT silently skipped (C10, ports #8 by @mo4islona)', async () => {
+  // f1 has NO fromBlock (⇒ genesis); f2 starts at 15M. The bug took Math.min over only the DEFINED
+  // fromBlocks → backfillStart=15M → chunkRange clamped chunk 0's fetch to start at 15M, so f1's block-100
+  // log was never streamed and its [0,15M) history was silently marked synced. After the fix the floor is
+  // 0 (any undefined ⇒ genesis, symmetric with backfillEnd). PORTAL_CHUNK_FIXED=1 + head=2e9 come from the
+  // beforeEach: chunkBlocks stays 500k → block 100 ∈ chunk 0, and [100,100] is well under the head.
+  const T1 = '0x' + '11'.repeat(32);
+  const A1 = '0x' + 'aa'.repeat(20); // f1 — unbounded (genesis) source
+  const A2 = '0x' + 'bb'.repeat(20); // f2 — fromBlock 15M
+  const BN1 = 100;
+  const TXA = '0x' + 'a1'.repeat(32);
+  const mkBlock = () => ({
+    ...FIXTURE_BLOCK,
+    header: {
+      ...FIXTURE_BLOCK.header,
+      number: BN1,
+      hash: '0x' + BN1.toString(16).padStart(64, '0'),
+    },
+    logs: [
+      {
+        address: A1,
+        topics: [T1],
+        data: '0x',
+        transactionHash: TXA,
+        transactionIndex: 0,
+        logIndex: 0,
+      },
+    ],
+    transactions: [
+      {
+        ...FIXTURE_BLOCK.transactions[0],
+        transactionIndex: 0,
+        hash: TXA,
+        from: A1,
+        to: A1,
+      },
+    ],
+  });
+  const srv = http.createServer((req, res) => {
+    let body = '';
+    req.on('data', (c) => {
+      body += c;
+    });
+    req.on('end', () => {
+      if (req.url?.includes('finalized-head')) {
+        res.writeHead(200, { 'content-type': 'application/json' });
+        res.end(JSON.stringify({ number: 1_000_000_000 }));
+        return;
+      }
+      const q = body ? JSON.parse(body) : {};
+      const from = q.fromBlock ?? 0;
+      const to = q.toBlock ?? 1e12;
+      const wantsA1 = (q.logs ?? []).some((s: any) =>
+        (s.address ?? [])
+          .map((x: string) => x.toLowerCase())
+          .includes(A1.toLowerCase()),
+      );
+      if (from <= BN1 && to >= BN1 && wantsA1) {
+        res.writeHead(200, { 'content-type': 'application/x-ndjson' });
+        res.end(JSON.stringify(mkBlock()) + '\n');
+        return;
+      }
+
+      res.writeHead(204).end();
+    });
+  });
+  const p: number = await new Promise((r) =>
+    srv.listen(0, () => r((srv.address() as AddressInfo).port)),
+  );
+  try {
+    const inserted = { logs: [] as any[] };
+    const syncStore: any = {
+      insertLogs: (x: any) => inserted.logs.push(...x.logs),
+      insertBlocks: () => {},
+      insertTransactions: () => {},
+      insertTransactionReceipts: () => {},
+      insertTraces: () => {},
+      insertChildAddresses: () => {},
+    };
+    const mkFilter = (
+      sourceId: string,
+      address: string,
+      topic0: string,
+      fromBlock: number | undefined,
+    ): any => ({
+      type: 'log',
+      chainId: 1,
+      sourceId,
+      address,
+      topic0,
+      topic1: null,
+      topic2: null,
+      topic3: null,
+      fromBlock,
+      toBlock: undefined,
+      hasTransactionReceipt: false,
+      include: [],
+    });
+    const f1 = mkFilter('s1', A1, T1, undefined); // no fromBlock ⇒ genesis
+    const f2 = mkFilter('s2', A2, '0x' + '22'.repeat(32), 15_000_000);
+    const sync = createPortalHistoricalSync({
+      common: {
+        logger: { debug() {}, info() {}, warn() {}, error() {}, trace() {} },
+      },
+      chain: { id: 1, name: 'mainnet', portal: `http://localhost:${p}` },
+      childAddresses: new Map(),
+      eventCallbacks: [{ filter: f1 }, { filter: f2 }],
+    } as any);
+    const iA: [number, number] = [BN1, BN1];
+    const logs = await sync.syncBlockRangeData({
+      interval: iA,
+      requiredIntervals: [{ interval: iA, filter: f1 }],
+      requiredFactoryIntervals: [],
+      syncStore,
+    } as any);
+    // BEFORE FIX: floor=15M → chunk 0's fetch clamped to [15M, …] → block 100 never streamed → 0 logs.
+    // AFTER FIX:  floor=0 → chunk 0's fetch [0, 499_999] reaches block 100 → f1's log present.
+    expect(inserted.logs).toHaveLength(1);
+    expect(inserted.logs[0].topics[0].toLowerCase()).toBe(T1.toLowerCase());
+    expect(logs).toHaveLength(1);
+  } finally {
+    srv.close();
+  }
+});