refactor(portal): invariant-first re-architecture (fixes G1/G2/G3)#6
Merged
Conversation
0abdb9e to
85670c2
Compare
4 tasks
…ks, metrics Foundational pure modules for the invariant-first re-architecture: - portal-config.ts: parse+validate ALL PORTAL_* env once into a frozen PortalConfig (INV-14 — garbage values fail fast instead of NaN-poisoning the chunk grid). - portal-errors.ts: the typed error taxonomy (PortalHttpError, PortalThrottleError, PortalSchemaFieldError, PortalDatasetStartError, PortalQueryTooLargeError, NoProgressError, InvariantViolation) + isTransientError/isNetworkError. - portal-invariant.ts: invariant()/invariantStrict() + off|on|strict modes. - portal-gate.ts: the shared AIMD + row-budget controller as a PURE reducer (gateReduce, INV-7/INV-8) behind a lazy process-shared shell; no module-scope side effects. - portal-chunks.ts: pure chunk-grid math (idxOf, chunkRange, density scaling, read-ahead / eviction plans); traceSafeChunkBlocks moves here. - portal-metrics.ts: stats shape + byte-identical writeMetrics + the gate-log ticker (moved out of module scope). - sync-upstream.sh: copy the Portal layer by glob (source+tests) so new modules are never silently missed; provision fast-check (dev-only) into node_modules. - delete dead portal/config.ts (abandoned withPortal injection design). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The pure/shell domain modules the orchestration shell composes: - portal-client.ts: the HTTP shell — finalizedHead(), stream() with typed error mapping (429/5xx/409 + four 400 variants), retry+backoff, per-stream field degradation, body-size guard, incremental row registration (onRows, G3), and the shared ndjsonLines() splitter. fetchImpl/sleepImpl injectable. - portal-filters.ts: compileFetchSpec() — the frozen per-chain fetch-spec (INV-1), the SINGLE source of log/tx/trace requests + field projections shared with the realtime wire. mergeLogRequests preserves the log match-set (INV-11); the trace request carries no filter (INV-5). - portal-discovery.ts: the factory-child discovery state machine — advances the watermark ONLY on scan success and rolls back on failure (INV-3 / fixes G2); min-merges children to earliest creation (INV-4). Pure planDiscovery/splitWindows. - portal-assemble.ts: the pure range assembler — interval exactness (INV-2), full-tree trace ranking then match (INV-5), seenTx dedupe, closest by loop (C9). - portal-transform.ts: named raw-NDJSON types; re-export traceSafeChunkBlocks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…realtime - portal.ts: createPortalHistoricalSync is now a ~300-line imperative shell that wires config→client→gate→filters→chunks→discovery→assemble→metrics and owns only the chunk cache, stash, and delegation. Seam methods carry invariant checks at the boundaries. Fixes G1 (a rejected chunk promise is evicted immediately + its rows freed, INV-13) and G3 (rows registered per arriving batch via the client hook). Behavior + the public seam are frozen. - portal-realtime.ts: use the shared ndjsonLines(); delete the stray bottom re-exports (_hx, type re-exports). - portal-realtime-wire.ts: import log-request construction + field projections from portal-filters (delete the duplicated logRequestsFor/mergeLogRequests/ PORTAL_MAX_ADDRESSES/BLOCK_FIELDS); keep the entry symbols. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- portal/INVARIANTS.md: the centerpiece — INV-1…INV-14 with statement · rationale · enforced-by · checked-at · tested-by, plus the G1/G2/G3 fix mapping. Legacy C# tags mapped. - HOW-IT-WORKS.md / INTEGRATION.md: refreshed the "Where the code lives" map for the new module layout; link INVARIANTS.md. - README.md: add an "Invariants" pointer under Learn more. - versions.json: note the next publish from main is 0.16.6-sqd.2 (DO NOT publish). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
npm walks up to the nearest package.json, so installing fast-check inside the cloned ponder workspace tripped on the post-rename workspace. Install it in a mktemp dir and copy fast-check + pure-rand into the core's node_modules. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…g, INV-9 assert - portal-discovery: a predecessor extension's failure now PROPAGATES (await earlier without swallowing) so a successor can never confirm coverage over an unscanned gap (B1/G2 — silent factory-child loss reproduced in review). - portal.ts: per-fetch RowToken accounting (S1 — a stale evicted fetch can't register/free rows against its replacement); coveredTo recorded per cache entry; INV-9 runtime assert at request build; INV-12 stash assert now strict-mode-only (production keeps pre-refactor overwrite semantics + debug log, S2); refreshPortalHead retry via client (injectable sleep, S5k). - portal-client: finalizedHeadRetry; countRows includes headers (S5g); dead NoProgressError guard removed — progress is by construction (B4c). - portal-errors: NoProgressError deleted; InvariantViolation message points at INVARIANTS.md + PORTAL_CHECKS escape hatch (S5b). - portal/config.ts restored (repo-side compat-harness utility — B2); PUBLISHING.md stale sentence fixed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ntier extend into the architecture Rebase follow-ups onto current main: - INV-15 (PR #2 a600fc9, ported): portal-discovery queues newly-discovered children inside the min-merge guard (store-preloaded children re-discover at prev ≤ bn and are never re-queued) and exposes takePendingInRange/restorePending; the portal.ts shell flushes the interval-scoped set via syncStore.insertChildAddresses inside the SAME syncBlockRangeData whose transaction marks the factory interval cached. A failed flush restores the queue and fails the interval loud. All six of PR #2's regression tests pass against this architecture unchanged. - INV-13 frontier extend (open PR #5's semantics, adopted — supersedes the evict+refetch approach): each cache entry records coveredTo; a hit past it streams ONLY the newly-finalized tail via the shared runStreams (append-only merge, rows accounted through the same per-fetch token), with discovery re-driven through the tail; a failed extend evicts the whole entry (never leaves the optimistic high-water — the G2 lesson). Fully-finalized/bounded chunks keep coveredTo == desiredTo → clean hit, zero behavior change. - portal-realtime.ts: INV-10 O(1) parentHash/number link tripwire on every append. - portal-assemble.ts: vacuous INV-2 per-row assert removed (enforced by construction behind the single inRange predicate); dead chunkRowCount deleted. - portal-client.ts: dead NoProgressError guard removed (progress by construction); finalizedHeadRetry (injectable sleep); countRows includes headers. - realtime.ts: fix the pre-existing noAssignInExpressions lint ERROR (summarize's byKey bucket init) — the only semantic-file edit outside the layer rewrite. - CLAUDE.md style compliance (single var declarators, no assignment-in-expression, braced control bodies) + repo-Biome (2.5.2, single-quote) formatting throughout. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ix, interleaving property, mutation killers - portal-shell.test.ts (new): seam-level tests — G1 (rejected chunk evicted, rows freed, refetch not cached-rejection), G3 (rows visible to the gate MID-chunk), the INV-13 frontier-extend regression (adapted from PR #5), the INV-9 delegation matrix (past-head → whole-interval RPC via a stubbed rpc + delegated-key consume-once; unknown head → delegation; stream-realtime → warn + empty, NOT delegated), INV-12 stash consume-once + on/strict overwrite semantics, and the S1 row-accounting model (rows return to baseline through evict/reject/in-flight interleavings). - portal-discovery.test.ts: the G2 INTERLEAVING test (a successor extension chained on a failing scan rejects — never confirms over the gap) + a randomized property (random ensures + injected window failures → coverage below through() is always complete; retry converges). - portal-client.test.ts: retry budget exhaustion (11 tries), retry-after cap (ra*1000 capped 30s), per-batch onRows (G3), finalizedHeadRetry backoff. - portal-filters.test.ts: INV-11 model matcher extended to topic2/topic3 (+ the key-drops-topic2/3 mutation killer). - portal-chunks.test.ts: saturated read-ahead is EXACTLY depth-1 (mutation killer). - portal-gate.test.ts: an AIMD ramp pumps parked waiters. - portal-assemble.test.ts: trace assembly upper-bound (hi+1 excluded). - portal-metrics.test.ts (new): golden field-for-field metrics-file shape (freezes the bench-harness contract). - fast-check seeds pinned (deterministic CI); pre-existing test files reformatted to the repo's own biome.json (single-quote) — proven byte-identical to biome(origin/main's versions), i.e. formatting-only. Mutation-verified: reverting G1 eviction, no-op'ing the G3 hook, re-introducing the G2 swallow, dropping topic2/3 from the merge key, and breaking saturated depth-1 each fail exactly the intended test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…TAL_CHECKS docs - INVARIANTS.md: every row's enforced-by/checked-at/tested-by now cites real, passing tests; INV-2 marked enforced-by-construction (vacuous assert removed); INV-3 cites the interleaving + randomized properties; INV-9 cites the runtime assert + delegation matrix; INV-10 cites the real link tripwire; INV-13 rewritten for progress-by-construction + the frontier EXTEND; INV-1 noted as a future-regression tripwire; NEW INV-15 row (child persistence, ported from PR #2). Fixed-gaps section lists the found-and-fixed head-boundary bug + mutation evidence. - scripts/sync-upstream.sh: repo-side Biome gate (pinned 2.5.2, repo biome.json, --diagnostic-level=error) runs before the copy under --test — an unformatted or lint-error portal file fails the build. - INTEGRATION.md: PORTAL_CHECKS row in the env-knob table. - PUBLISHING.md/HOW-IT-WORKS.md: stale prose fixed; portal/config.ts restored as the repo-side compat-harness utility (= origin/main's copy). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lesale Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…+ test dedupe Main merged PR #5 while this branch was in flight; the architecture already implements its frontier-extend semantics, so the rebase resolved portal.ts to the shell version. Deltas ported to MATCH MAIN (now the contract): - `stats.extends` counter: PortalStats/createStats, writeMetrics `fetch` block (after dataChunks — bench-harness shape), the syncBlockRangeData debug log, and the increment on the extend path; golden metrics test updated. - portal.test.ts carries main's #5 regression test VERBATIM (it passes unchanged against this architecture — behavior-parity proof); my earlier adaptation in portal-shell.test.ts is kept but renamed to its distinct angle ("frontier extend streams ONLY the newly-finalized tail") — it additionally asserts the extend request starts at coveredTo+1, which main's test doesn't cover. - One deliberate divergence, documented in INVARIANTS.md: on a FAILED extend main leaves a cached rejection (its pre-existing G1 gap); here the entry is evicted and its rows freed — the G1-consistent recovery this PR introduces. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
f2b8dd1 to
75b7139
Compare
…n, honest test counts Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mo4islona
added a commit
that referenced
this pull request
Jul 3, 2026
…), rebased onto the invariant re-architecture Re-applies three silent-gap fixes from the pre-rearchitecture branch onto the new module layout (portal.ts monolith → functional core + shell). Each ships a regression test; the full Portal suite stays green (138 tests). - backfill floor (finding 3, portal-filters.ts): an undefined fromBlock is genesis (`filter.fromBlock ?? 0`), so if ANY source omits it the floor MUST be 0. compileFetchSpec's Math.min over the DEFINED-only fromBlocks let a bounded source clamp every chunk fetch past an unbounded source's history, marking that prefix synced over a silent gap. - reconnect on child discovery (finding 4, portal-realtime*.ts): stream mode filters /stream server-side, so a child discovered after a connection opens can't reach it. streamHotBlocks now snapshots a logsRevision at open and re-opens with the widened filter the moment it advances (bumped in the wire on each child); ndjsonLines cancels its reader on early break so the old connection closes. - refuse unsupported stream mode (finding 5, portal-realtime-wire.ts): stream mode emits only log-bearing block events, so a non-log source (trace/transfer/ transaction/block) or a receipt-requiring log source would be silently skipped while its intervals finalized as cached. assertStreamModeSupported() fails loud at startup instead. Findings 6 (finalized-head probe) and 7 (unknown-parent gap) are intentionally dropped: the re-architecture (#6) made the opposite, conservative choice on both (passthrough / clear-window) as documented decisions — raise separately rather than override here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mo4islona
added a commit
that referenced
this pull request
Jul 3, 2026
…meout, add skills symlink Portal test-infra fixes exposed by the invariant-first refactor (#6); the S1 timeout itself is fixed separately by #19, so this no longer touches portal-shell.test.ts. - scripts/sync-upstream.sh: the --coverage branch never provisioned fast-check (only --test did), so the 5 property-test files failed with "Failed to load url fast-check" — the coverage path is broken on main today. Extract a shared provision_fastcheck() called by both branches; in --coverage AFTER `pnpm add` (which re-resolves node_modules and would prune the manually-copied tree). - vite.portal.config.ts: coverage `include` was a stale hand-list of 5 files; the refactor's 10 new portal-*.ts modules were silently dropped from the denominator. Switch to a glob matching the copy step (portal*.ts + realtime*.ts) with exclude ['**/*.test.ts']. - vite.portal.config.ts: bump testTimeout 15000 -> 30000 as general headroom for the real-backoff head-probe tests + coverage instrumentation on a loaded CI box. - .claude/skills -> .agents/skills: a single directory-level symlink so any skill added under the tracked .agents/skills/ is picked up automatically (force-added past the .claude/ gitignore). Verified: --test 140/140; --coverage 140/140 (fast-check provisioned, 94.86% stmts / 85.11% branches / 95.89% funcs on the full glob denominator). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mo4islona
added a commit
that referenced
this pull request
Jul 3, 2026
…symlink Portal test-infra fixes exposed by the invariant-first refactor (#6); the S1 timeout itself is fixed separately by #19, so this no longer touches portal-shell.test.ts. - scripts/sync-upstream.sh: the --coverage branch never provisioned fast-check (only --test did), so the 5 property-test files failed with "Failed to load url fast-check" — the coverage path is broken on main today. Extract a shared provision_fastcheck() called by both branches; in --coverage AFTER `pnpm add` (which re-resolves node_modules and would prune the manually-copied tree). - vite.portal.config.ts: coverage `include` was a stale hand-list of 5 files; the refactor's 10 new portal-*.ts modules were silently dropped from the denominator. Switch to a glob matching the copy step (portal*.ts + realtime*.ts) with exclude ['**/*.test.ts']. - .claude/skills -> .agents/skills: a single directory-level symlink so any skill added under the tracked .agents/skills/ is picked up automatically (force-added past the .claude/ gitignore). Verified: --test 140/140; --coverage 140/140 (fast-check provisioned, 94.86% stmts / 85.11% branches / 95.89% funcs on the full glob denominator). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mo4islona
added a commit
that referenced
this pull request
Jul 3, 2026
…symlink (#18) Portal test-infra fixes exposed by the invariant-first refactor (#6); the S1 timeout itself is fixed separately by #19, so this no longer touches portal-shell.test.ts. - scripts/sync-upstream.sh: the --coverage branch never provisioned fast-check (only --test did), so the 5 property-test files failed with "Failed to load url fast-check" — the coverage path is broken on main today. Extract a shared provision_fastcheck() called by both branches; in --coverage AFTER `pnpm add` (which re-resolves node_modules and would prune the manually-copied tree). - vite.portal.config.ts: coverage `include` was a stale hand-list of 5 files; the refactor's 10 new portal-*.ts modules were silently dropped from the denominator. Switch to a glob matching the copy step (portal*.ts + realtime*.ts) with exclude ['**/*.test.ts']. - .claude/skills -> .agents/skills: a single directory-level symlink so any skill added under the tracked .agents/skills/ is picked up automatically (force-added past the .claude/ gitignore). Verified: --test 140/140; --coverage 140/140 (fast-check provisioned, 94.86% stmts / 85.11% branches / 95.89% funcs on the full glob denominator). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Invariant-first re-architecture of the
portal/layerA correctness-preserving re-architecture of the Portal historical-sync layer around explicit, provable invariants: a functional core (pure, property-tested) behind an imperative shell, runtime invariant checks, and typed errors. Behaviour, the public seam, env knobs, and the wiring patch are frozen — plus a small set of named bug fixes (below), each under invariant cover with a mutation-verified regression test.
Rebased onto current main (through 802ceed): PR #2 (persist discovered factory children) is fully ported into the new architecture as INV-15 — all six of its regression tests pass unchanged; PR #5 (frontier-chunk extend, merged as 802ceed) is re-implemented in the new architecture with behavior parity — main's own #5 regression test passes unchanged against this branch, and the
extendsmetrics counter matches 802ceed exactly. One deliberate divergence (documented in INVARIANTS.md): on a FAILED extend, main leaves a cached rejection (its pre-existing G1 gap); this branch evicts the entry and frees its rows — the G1-consistent recovery this PR introduces.Test counts
132 tests, 15 files (on both ponder 0.16.6 and 0.15.17):
biome.json(single-quote), proven formatting-only: every file at HEAD is byte-identical tobiome(origin/main@802ceed's version).coveredTo+1.)The invariant catalog (
portal/INVARIANTS.md)FetchSpecid; runtime check is a future-regression tripwire)inRangepredicate), property vs brute-force modelonkeeps pre-refactor overwrite semantics for upstream retries (debug-logged),strictis loudcoveredToPortalConfig, garbage fails fastBugs fixed (relative to the original layer)
coveredTo+ tail-only EXTEND + theextendscounter), proven by main's own fix(portal): extend head-truncated frontier chunks (no silent gap when the Portal head advances) #5 test passing unchanged.realtime.ts(noAssignInExpressions) fixed — the only pre-existing source edit besides the layer itself.All five review mutations (G1 revert · G3 no-op · G2 swallow · merge-key drops topic2/3 · saturated read-ahead ≠ depth-1) were applied individually and each fails exactly its intended test.
Module inventory (biome-formatted lines)
portal.ts605 (orchestration shell; was an 811-line monolith with the same formatting conventions applied it would be far larger — the pre-format compact source was ~350) ·portal-filters.ts433 ·portal-client.ts374 ·portal-realtime-wire.ts327 ·portal-assemble.ts322 ·portal-realtime.ts276 ·portal-discovery.ts273 ·portal-transform.ts263 ·portal-gate.ts183 ·portal-metrics.ts146 ·portal-errors.ts142 ·portal-config.ts130 ·portal-chunks.ts89 ·portal-invariant.ts65. (portal/config.ts= origin/main's copy — repo-side compat-harness utility, not shipped.)Formatting / lint
The repo's own Biome (
biome.json, pinned 2.5.2, single-quote) now passes over ALL ofportal/at--diagnostic-level=error, enforced by a gate inscripts/sync-upstream.sh --test(repo-side, pre-copy — an unformatted file fails the build). Note: main's portal files predate PR #4's single-quote config switch, so this PR carries the (formatting-only, proven byte-equivalent) single-quote sweep for the five pre-existing test files +config.ts+vite.portal.config.ts+realtime.ts.Verification gates — all green
sync-upstream.sh 0.16.6 --test— biome gate green, wiring patch applies UNCHANGED (6/6 hunks), upstreamtscgreen, 132/132 tests. ✅sync-upstream.sh 0.15.17 --test— same. ✅Repo CI job (
node --experimental-strip-types --test harness/compat/*.test.ts) — 7/7 (config.ts restored). ✅Live bounded parity (
examples/euler-subgraph,PONDER_START=20529207 PONDER_END=20600000, public Portal + publicnode RPC), current origin/main build (0.16.6-sqd.9local pack) vs branch build (0.16.6-sqd.2local pack) — IDENTICAL:{logs, blocks, txs, receipts, traces}34, 24, 24, 0, 034, 24, 24, 0, 0{vaults, deposits, withdraws, borrows, repays, liquidates, vaultStatus}4,7,1,1,0,0,254,7,1,1,0,0,25(The vault-metadata
readContractenrichment was stubbed identically in both runs — the public RPC free tier can't serve archiveeth_call; it is orthogonal to the Portal historical path being compared.)Diff scope:
portal/,scripts/sync-upstream.sh,HOW-IT-WORKS.md,README.md,PUBLISHING.md,versions.json. Secret scan clean. Every INVARIANTS.md tested-by cites a real, passing test.Review guide (read order)
portal/INVARIANTS.md— the contract (start here).portal.ts— the shell:dataChunk(extend path), the INV-15 flush, the delegation branch.portal-config→portal-gate→portal-chunks→portal-filters→portal-discovery(G2 fix + pending queue) →portal-assemble.portal-client(error mapping,onRows),portal-metrics.portal-shell.test.ts(seam-level),portal-discovery.test.ts(interleaving + property).Spec deviations (with reasons)
node_modulesfrom a temp dir via npm (pinned^3), notpackage.jsondevDependencies — the post-rename pnpm workspace can't re-resolve (renaming the core orphansbenchmark'sponder@workspace:*).coveredTo, consistent with G1.__resetSharedGateis used inportal-shell.test.ts(the only file that asserts on gate state; vitest isolates modules per file) rather than editingportal.test.ts— keeping that file exactlybiome(main).🤖 Generated with Claude Code