Skip to content

fix(portal): persist discovered factory children to the sync store#2

Merged
mo4islona merged 1 commit into
mainfrom
fix/persist-factory-child-addresses
Jul 3, 2026
Merged

fix(portal): persist discovered factory children to the sync store#2
mo4islona merged 1 commit into
mainfrom
fix/persist-factory-child-addresses

Conversation

@mo4islona

Copy link
Copy Markdown
Collaborator

The bug (critical: silent event loss after restart)

The portal historical sync discovers factory children with its wide scan and writes them only into the in-memory childAddresses map — it never calls syncStore.insertChildAddresses. Two upstream facts make that fatal:

  1. Core marks requiredFactoryIntervals as cached (syncStore.insertIntervals, runtime/historical.ts) after every interval, regardless of what the sync implementation did.
  2. On startup, core loads child addresses only from the store (runtime/index.tsgetChildAddresses). There is no re-derivation from stored logs.

Stock ponder holds the invariant by calling insertChildAddresses inside syncBlockRangeData (sync-historical/index.ts), in the same transaction as insertIntervals.

So a single uninterrupted run worked (the in-memory map is shared with event building and realtime), but on any restart or crash-resume of a factory app:

  • factory intervals are already marked cached → discovery never re-runs (requiredFactoryIntervals is empty, so the discovery floor never covers the pre-crash range),
  • the store returns zero children → the map is empty,
  • → every factory-child event is silently skipped: on the resumed backfill, in historical event extraction, and in realtime.

Neither the diff harness nor the 15-chain run could catch this: both are single uninterrupted runs, and the diff compares sync-store rows, not post-restart extraction.

The fix

  • Newly discovered children (not already known from the store) are queued during the wide discovery scan.
  • The queue is flushed via syncStore.insertChildAddresses inside syncBlockRangeData — the same DB transaction core commits together with insertIntervals, so the invariant is atomic: factory interval marked cached ⇒ its children are in the store.
  • A failed flush restores the queue and fails the interval loudly (core rolls back the transaction), so children are never silently dropped.
  • Children pre-loaded from the store are never re-queued → no duplicate rows.

Tests

Three regression tests (all fail against the previous portal.ts, verified):

  • persistence: discovered children are handed to insertChildAddresses in the same syncBlockRangeData call, with the factory object, creation block, and chainId — and are not re-inserted on later calls;
  • loud failure: a failing insertChildAddresses rejects the interval and the same children re-flush on the retried call;
  • restart: a new sync instance seeded only with what was persisted (cached factory intervals → no re-discovery) still fetches child logs — with the pre-fix (empty) store this returns nothing, which is exactly the silent post-restart loss.

Full portal suite: 56/56 passing on ponder@0.16.6 (sync-upstream.sh-equivalent tree, vite.portal.config.ts).

🤖 Generated with Claude Code

@mo4islona mo4islona force-pushed the fix/persist-factory-child-addresses branch from 7a5753b to 2b32c50 Compare July 2, 2026 20:20
@mo4islona mo4islona closed this Jul 2, 2026
@mo4islona mo4islona force-pushed the fix/persist-factory-child-addresses branch from 2b32c50 to 68c0a78 Compare July 2, 2026 20:36
@mo4islona mo4islona reopened this Jul 2, 2026
@mo4islona mo4islona marked this pull request as draft July 3, 2026 07:44
The portal historical sync discovered factory children with its wide scan but
wrote them only to the in-memory childAddresses map — never to the sync store.
Core marks requiredFactoryIntervals cached after every interval and, on startup,
loads children ONLY from the store (no re-derivation from logs). So a single
uninterrupted run worked, but any restart/crash-resume of a factory app loaded
an empty child set against already-cached factory intervals — discovery never
re-ran and every factory-child event was silently skipped.

Fix: queue newly discovered children and flush them via
syncStore.insertChildAddresses inside syncBlockRangeData — the same transaction
core commits with insertIntervals, so "factory interval cached ⇒ its children
are in the store" is atomic. The flush is scoped to children whose creation
block falls within the current interval, leaving the rest queued for their
owning interval. Grabbing the whole cross-interval queue would commit one
interval's children in another interval's transaction and, under core's
concurrent per-interval transactions, could lose a child if that transaction
rolled back after a lower interval had committed as cached (restart
re-discovery is floored at the lowest uncached factory interval and would never
rescan it). This mirrors stock sync-historical's per-interval child set. A
failed flush restores the queue and fails the interval loud (core rolls back).

Tests: persistence in the same syncBlockRangeData call; interval-scoped flush
(fails against a whole-queue flush, verified); loud failure + re-flush;
post-flush insertLogs failure rejects loud; restart seeded only from the store;
live re-discovery de-dup.

Also: delete the orphaned packages/portal-sync/src/historical.ts prototype
(unimported duplicate of the live engine); fix the identical
noAssignInExpressions in portal-realtime.ts; add repo code-style rules
(CLAUDE.md + biome useSingleVarDeclarator).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mo4islona mo4islona force-pushed the fix/persist-factory-child-addresses branch from cedc055 to 557967d Compare July 3, 2026 09:07
@mo4islona mo4islona marked this pull request as ready for review July 3, 2026 09:33
@mo4islona mo4islona merged commit a600fc9 into main Jul 3, 2026
8 checks passed
@mo4islona mo4islona deleted the fix/persist-factory-child-addresses branch July 3, 2026 09:34
dzhelezov added a commit that referenced this pull request Jul 3, 2026
…ntier extend into the architecture

Rebase follow-ups onto current main:

- INV-15 (PR #2 a600fc9, ported): portal-discovery queues newly-discovered children
  inside the min-merge guard (store-preloaded children re-discover at prev ≤ bn and
  are never re-queued) and exposes takePendingInRange/restorePending; the portal.ts
  shell flushes the interval-scoped set via syncStore.insertChildAddresses inside the
  SAME syncBlockRangeData whose transaction marks the factory interval cached. A
  failed flush restores the queue and fails the interval loud. All six of PR #2's
  regression tests pass against this architecture unchanged.
- INV-13 frontier extend (open PR #5's semantics, adopted — supersedes the
  evict+refetch approach): each cache entry records coveredTo; a hit past it streams
  ONLY the newly-finalized tail via the shared runStreams (append-only merge, rows
  accounted through the same per-fetch token), with discovery re-driven through the
  tail; a failed extend evicts the whole entry (never leaves the optimistic
  high-water — the G2 lesson). Fully-finalized/bounded chunks keep coveredTo ==
  desiredTo → clean hit, zero behavior change.
- portal-realtime.ts: INV-10 O(1) parentHash/number link tripwire on every append.
- portal-assemble.ts: vacuous INV-2 per-row assert removed (enforced by construction
  behind the single inRange predicate); dead chunkRowCount deleted.
- portal-client.ts: dead NoProgressError guard removed (progress by construction);
  finalizedHeadRetry (injectable sleep); countRows includes headers.
- realtime.ts: fix the pre-existing noAssignInExpressions lint ERROR (summarize's
  byKey bucket init) — the only semantic-file edit outside the layer rewrite.
- CLAUDE.md style compliance (single var declarators, no assignment-in-expression,
  braced control bodies) + repo-Biome (2.5.2, single-quote) formatting throughout.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dzhelezov added a commit that referenced this pull request Jul 3, 2026
…TAL_CHECKS docs

- INVARIANTS.md: every row's enforced-by/checked-at/tested-by now cites real,
  passing tests; INV-2 marked enforced-by-construction (vacuous assert removed);
  INV-3 cites the interleaving + randomized properties; INV-9 cites the runtime
  assert + delegation matrix; INV-10 cites the real link tripwire; INV-13 rewritten
  for progress-by-construction + the frontier EXTEND; INV-1 noted as a
  future-regression tripwire; NEW INV-15 row (child persistence, ported from PR #2).
  Fixed-gaps section lists the found-and-fixed head-boundary bug + mutation evidence.
- scripts/sync-upstream.sh: repo-side Biome gate (pinned 2.5.2, repo biome.json,
  --diagnostic-level=error) runs before the copy under --test — an unformatted or
  lint-error portal file fails the build.
- INTEGRATION.md: PORTAL_CHECKS row in the env-knob table.
- PUBLISHING.md/HOW-IT-WORKS.md: stale prose fixed; portal/config.ts restored as the
  repo-side compat-harness utility (= origin/main's copy).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dzhelezov added a commit that referenced this pull request Jul 3, 2026
…ntier extend into the architecture

Rebase follow-ups onto current main:

- INV-15 (PR #2 a600fc9, ported): portal-discovery queues newly-discovered children
  inside the min-merge guard (store-preloaded children re-discover at prev ≤ bn and
  are never re-queued) and exposes takePendingInRange/restorePending; the portal.ts
  shell flushes the interval-scoped set via syncStore.insertChildAddresses inside the
  SAME syncBlockRangeData whose transaction marks the factory interval cached. A
  failed flush restores the queue and fails the interval loud. All six of PR #2's
  regression tests pass against this architecture unchanged.
- INV-13 frontier extend (open PR #5's semantics, adopted — supersedes the
  evict+refetch approach): each cache entry records coveredTo; a hit past it streams
  ONLY the newly-finalized tail via the shared runStreams (append-only merge, rows
  accounted through the same per-fetch token), with discovery re-driven through the
  tail; a failed extend evicts the whole entry (never leaves the optimistic
  high-water — the G2 lesson). Fully-finalized/bounded chunks keep coveredTo ==
  desiredTo → clean hit, zero behavior change.
- portal-realtime.ts: INV-10 O(1) parentHash/number link tripwire on every append.
- portal-assemble.ts: vacuous INV-2 per-row assert removed (enforced by construction
  behind the single inRange predicate); dead chunkRowCount deleted.
- portal-client.ts: dead NoProgressError guard removed (progress by construction);
  finalizedHeadRetry (injectable sleep); countRows includes headers.
- realtime.ts: fix the pre-existing noAssignInExpressions lint ERROR (summarize's
  byKey bucket init) — the only semantic-file edit outside the layer rewrite.
- CLAUDE.md style compliance (single var declarators, no assignment-in-expression,
  braced control bodies) + repo-Biome (2.5.2, single-quote) formatting throughout.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dzhelezov added a commit that referenced this pull request Jul 3, 2026
…TAL_CHECKS docs

- INVARIANTS.md: every row's enforced-by/checked-at/tested-by now cites real,
  passing tests; INV-2 marked enforced-by-construction (vacuous assert removed);
  INV-3 cites the interleaving + randomized properties; INV-9 cites the runtime
  assert + delegation matrix; INV-10 cites the real link tripwire; INV-13 rewritten
  for progress-by-construction + the frontier EXTEND; INV-1 noted as a
  future-regression tripwire; NEW INV-15 row (child persistence, ported from PR #2).
  Fixed-gaps section lists the found-and-fixed head-boundary bug + mutation evidence.
- scripts/sync-upstream.sh: repo-side Biome gate (pinned 2.5.2, repo biome.json,
  --diagnostic-level=error) runs before the copy under --test — an unformatted or
  lint-error portal file fails the build.
- INTEGRATION.md: PORTAL_CHECKS row in the env-knob table.
- PUBLISHING.md/HOW-IT-WORKS.md: stale prose fixed; portal/config.ts restored as the
  repo-side compat-harness utility (= origin/main's copy).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dzhelezov added a commit that referenced this pull request Jul 3, 2026
* refactor(portal): scaffolding — config, errors, invariant, gate, chunks, metrics

Foundational pure modules for the invariant-first re-architecture:
- portal-config.ts: parse+validate ALL PORTAL_* env once into a frozen PortalConfig
  (INV-14 — garbage values fail fast instead of NaN-poisoning the chunk grid).
- portal-errors.ts: the typed error taxonomy (PortalHttpError, PortalThrottleError,
  PortalSchemaFieldError, PortalDatasetStartError, PortalQueryTooLargeError,
  NoProgressError, InvariantViolation) + isTransientError/isNetworkError.
- portal-invariant.ts: invariant()/invariantStrict() + off|on|strict modes.
- portal-gate.ts: the shared AIMD + row-budget controller as a PURE reducer
  (gateReduce, INV-7/INV-8) behind a lazy process-shared shell; no module-scope
  side effects.
- portal-chunks.ts: pure chunk-grid math (idxOf, chunkRange, density scaling,
  read-ahead / eviction plans); traceSafeChunkBlocks moves here.
- portal-metrics.ts: stats shape + byte-identical writeMetrics + the gate-log ticker
  (moved out of module scope).
- sync-upstream.sh: copy the Portal layer by glob (source+tests) so new modules are
  never silently missed; provision fast-check (dev-only) into node_modules.
- delete dead portal/config.ts (abandoned withPortal injection design).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(portal): functional core — client, filters, discovery, assemble

The pure/shell domain modules the orchestration shell composes:
- portal-client.ts: the HTTP shell — finalizedHead(), stream() with typed error
  mapping (429/5xx/409 + four 400 variants), retry+backoff, per-stream field
  degradation, body-size guard, incremental row registration (onRows, G3), and the
  shared ndjsonLines() splitter. fetchImpl/sleepImpl injectable.
- portal-filters.ts: compileFetchSpec() — the frozen per-chain fetch-spec (INV-1),
  the SINGLE source of log/tx/trace requests + field projections shared with the
  realtime wire. mergeLogRequests preserves the log match-set (INV-11); the trace
  request carries no filter (INV-5).
- portal-discovery.ts: the factory-child discovery state machine — advances the
  watermark ONLY on scan success and rolls back on failure (INV-3 / fixes G2);
  min-merges children to earliest creation (INV-4). Pure planDiscovery/splitWindows.
- portal-assemble.ts: the pure range assembler — interval exactness (INV-2),
  full-tree trace ranking then match (INV-5), seenTx dedupe, closest by loop (C9).
- portal-transform.ts: named raw-NDJSON types; re-export traceSafeChunkBlocks.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(portal): rewrite portal.ts as the orchestration shell; tidy realtime

- portal.ts: createPortalHistoricalSync is now a ~300-line imperative shell that
  wires config→client→gate→filters→chunks→discovery→assemble→metrics and owns only
  the chunk cache, stash, and delegation. Seam methods carry invariant checks at the
  boundaries. Fixes G1 (a rejected chunk promise is evicted immediately + its rows
  freed, INV-13) and G3 (rows registered per arriving batch via the client hook).
  Behavior + the public seam are frozen.
- portal-realtime.ts: use the shared ndjsonLines(); delete the stray bottom
  re-exports (_hx, type re-exports).
- portal-realtime-wire.ts: import log-request construction + field projections from
  portal-filters (delete the duplicated logRequestsFor/mergeLogRequests/
  PORTAL_MAX_ADDRESSES/BLOCK_FIELDS); keep the entry symbols.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(portal): invariant catalog + module-map updates

- portal/INVARIANTS.md: the centerpiece — INV-1…INV-14 with statement · rationale ·
  enforced-by · checked-at · tested-by, plus the G1/G2/G3 fix mapping. Legacy C#
  tags mapped.
- HOW-IT-WORKS.md / INTEGRATION.md: refreshed the "Where the code lives" map for the
  new module layout; link INVARIANTS.md.
- README.md: add an "Invariants" pointer under Learn more.
- versions.json: note the next publish from main is 0.16.6-sqd.2 (DO NOT publish).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(build): provision fast-check from a temp dir outside the clone

npm walks up to the nearest package.json, so installing fast-check inside the cloned
ponder workspace tripped on the post-rename workspace. Install it in a mktemp dir and
copy fast-check + pure-rand into the core's node_modules.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(portal): review round — G2 propagation, typed-token row accounting, INV-9 assert

- portal-discovery: a predecessor extension's failure now PROPAGATES (await earlier
  without swallowing) so a successor can never confirm coverage over an unscanned
  gap (B1/G2 — silent factory-child loss reproduced in review).
- portal.ts: per-fetch RowToken accounting (S1 — a stale evicted fetch can't
  register/free rows against its replacement); coveredTo recorded per cache entry;
  INV-9 runtime assert at request build; INV-12 stash assert now strict-mode-only
  (production keeps pre-refactor overwrite semantics + debug log, S2);
  refreshPortalHead retry via client (injectable sleep, S5k).
- portal-client: finalizedHeadRetry; countRows includes headers (S5g); dead
  NoProgressError guard removed — progress is by construction (B4c).
- portal-errors: NoProgressError deleted; InvariantViolation message points at
  INVARIANTS.md + PORTAL_CHECKS escape hatch (S5b).
- portal/config.ts restored (repo-side compat-harness utility — B2); PUBLISHING.md
  stale sentence fixed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(portal): port main's PR #2 (INV-15 child persistence) + PR #5 frontier extend into the architecture

Rebase follow-ups onto current main:

- INV-15 (PR #2 a600fc9, ported): portal-discovery queues newly-discovered children
  inside the min-merge guard (store-preloaded children re-discover at prev ≤ bn and
  are never re-queued) and exposes takePendingInRange/restorePending; the portal.ts
  shell flushes the interval-scoped set via syncStore.insertChildAddresses inside the
  SAME syncBlockRangeData whose transaction marks the factory interval cached. A
  failed flush restores the queue and fails the interval loud. All six of PR #2's
  regression tests pass against this architecture unchanged.
- INV-13 frontier extend (open PR #5's semantics, adopted — supersedes the
  evict+refetch approach): each cache entry records coveredTo; a hit past it streams
  ONLY the newly-finalized tail via the shared runStreams (append-only merge, rows
  accounted through the same per-fetch token), with discovery re-driven through the
  tail; a failed extend evicts the whole entry (never leaves the optimistic
  high-water — the G2 lesson). Fully-finalized/bounded chunks keep coveredTo ==
  desiredTo → clean hit, zero behavior change.
- portal-realtime.ts: INV-10 O(1) parentHash/number link tripwire on every append.
- portal-assemble.ts: vacuous INV-2 per-row assert removed (enforced by construction
  behind the single inRange predicate); dead chunkRowCount deleted.
- portal-client.ts: dead NoProgressError guard removed (progress by construction);
  finalizedHeadRetry (injectable sleep); countRows includes headers.
- realtime.ts: fix the pre-existing noAssignInExpressions lint ERROR (summarize's
  byKey bucket init) — the only semantic-file edit outside the layer rewrite.
- CLAUDE.md style compliance (single var declarators, no assignment-in-expression,
  braced control bodies) + repo-Biome (2.5.2, single-quote) formatting throughout.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(portal): review-round suite — G1/G3 regressions, delegation matrix, interleaving property, mutation killers

- portal-shell.test.ts (new): seam-level tests — G1 (rejected chunk evicted, rows
  freed, refetch not cached-rejection), G3 (rows visible to the gate MID-chunk),
  the INV-13 frontier-extend regression (adapted from PR #5), the INV-9 delegation
  matrix (past-head → whole-interval RPC via a stubbed rpc + delegated-key
  consume-once; unknown head → delegation; stream-realtime → warn + empty, NOT
  delegated), INV-12 stash consume-once + on/strict overwrite semantics, and the
  S1 row-accounting model (rows return to baseline through evict/reject/in-flight
  interleavings).
- portal-discovery.test.ts: the G2 INTERLEAVING test (a successor extension chained
  on a failing scan rejects — never confirms over the gap) + a randomized property
  (random ensures + injected window failures → coverage below through() is always
  complete; retry converges).
- portal-client.test.ts: retry budget exhaustion (11 tries), retry-after cap
  (ra*1000 capped 30s), per-batch onRows (G3), finalizedHeadRetry backoff.
- portal-filters.test.ts: INV-11 model matcher extended to topic2/topic3 (+ the
  key-drops-topic2/3 mutation killer).
- portal-chunks.test.ts: saturated read-ahead is EXACTLY depth-1 (mutation killer).
- portal-gate.test.ts: an AIMD ramp pumps parked waiters.
- portal-assemble.test.ts: trace assembly upper-bound (hi+1 excluded).
- portal-metrics.test.ts (new): golden field-for-field metrics-file shape (freezes
  the bench-harness contract).
- fast-check seeds pinned (deterministic CI); pre-existing test files reformatted
  to the repo's own biome.json (single-quote) — proven byte-identical to
  biome(origin/main's versions), i.e. formatting-only.

Mutation-verified: reverting G1 eviction, no-op'ing the G3 hook, re-introducing the
G2 swallow, dropping topic2/3 from the merge key, and breaking saturated depth-1
each fail exactly the intended test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs+build(portal): INVARIANTS honesty pass (INV-15), biome gate, PORTAL_CHECKS docs

- INVARIANTS.md: every row's enforced-by/checked-at/tested-by now cites real,
  passing tests; INV-2 marked enforced-by-construction (vacuous assert removed);
  INV-3 cites the interleaving + randomized properties; INV-9 cites the runtime
  assert + delegation matrix; INV-10 cites the real link tripwire; INV-13 rewritten
  for progress-by-construction + the frontier EXTEND; INV-1 noted as a
  future-regression tripwire; NEW INV-15 row (child persistence, ported from PR #2).
  Fixed-gaps section lists the found-and-fixed head-boundary bug + mutation evidence.
- scripts/sync-upstream.sh: repo-side Biome gate (pinned 2.5.2, repo biome.json,
  --diagnostic-level=error) runs before the copy under --test — an unformatted or
  lint-error portal file fails the build.
- INTEGRATION.md: PORTAL_CHECKS row in the env-knob table.
- PUBLISHING.md/HOW-IT-WORKS.md: stale prose fixed; portal/config.ts restored as the
  repo-side compat-harness utility (= origin/main's copy).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* build: pin fast-check exactly (3.23.2) and copy the resolved tree wholesale

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(portal): reconcile with merged PR #5 (802ceed) — extends counter + test dedupe

Main merged PR #5 while this branch was in flight; the architecture already
implements its frontier-extend semantics, so the rebase resolved portal.ts to the
shell version. Deltas ported to MATCH MAIN (now the contract):
- `stats.extends` counter: PortalStats/createStats, writeMetrics `fetch` block
  (after dataChunks — bench-harness shape), the syncBlockRangeData debug log, and
  the increment on the extend path; golden metrics test updated.
- portal.test.ts carries main's #5 regression test VERBATIM (it passes unchanged
  against this architecture — behavior-parity proof); my earlier adaptation in
  portal-shell.test.ts is kept but renamed to its distinct angle ("frontier extend
  streams ONLY the newly-finalized tail") — it additionally asserts the extend
  request starts at coveredTo+1, which main's test doesn't cover.
- One deliberate divergence, documented in INVARIANTS.md: on a FAILED extend main
  leaves a cached rejection (its pre-existing G1 gap); here the entry is evicted
  and its rows freed — the G1-consistent recovery this PR introduces.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: final-review nits — INV-15 range refs, corrected INV-13 citation, honest test counts

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant