refactor(portal): invariant-first re-architecture (#6)

* refactor(portal): scaffolding — config, errors, invariant, gate, chunks, metrics

Foundational pure modules for the invariant-first re-architecture:
- portal-config.ts: parse+validate ALL PORTAL_* env once into a frozen PortalConfig
  (INV-14 — garbage values fail fast instead of NaN-poisoning the chunk grid).
- portal-errors.ts: the typed error taxonomy (PortalHttpError, PortalThrottleError,
  PortalSchemaFieldError, PortalDatasetStartError, PortalQueryTooLargeError,
  NoProgressError, InvariantViolation) + isTransientError/isNetworkError.
- portal-invariant.ts: invariant()/invariantStrict() + off|on|strict modes.
- portal-gate.ts: the shared AIMD + row-budget controller as a PURE reducer
  (gateReduce, INV-7/INV-8) behind a lazy process-shared shell; no module-scope
  side effects.
- portal-chunks.ts: pure chunk-grid math (idxOf, chunkRange, density scaling,
  read-ahead / eviction plans); traceSafeChunkBlocks moves here.
- portal-metrics.ts: stats shape + byte-identical writeMetrics + the gate-log ticker
  (moved out of module scope).
- sync-upstream.sh: copy the Portal layer by glob (source+tests) so new modules are
  never silently missed; provision fast-check (dev-only) into node_modules.
- delete dead portal/config.ts (abandoned withPortal injection design).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(portal): functional core — client, filters, discovery, assemble

The pure/shell domain modules the orchestration shell composes:
- portal-client.ts: the HTTP shell — finalizedHead(), stream() with typed error
  mapping (429/5xx/409 + four 400 variants), retry+backoff, per-stream field
  degradation, body-size guard, incremental row registration (onRows, G3), and the
  shared ndjsonLines() splitter. fetchImpl/sleepImpl injectable.
- portal-filters.ts: compileFetchSpec() — the frozen per-chain fetch-spec (INV-1),
  the SINGLE source of log/tx/trace requests + field projections shared with the
  realtime wire. mergeLogRequests preserves the log match-set (INV-11); the trace
  request carries no filter (INV-5).
- portal-discovery.ts: the factory-child discovery state machine — advances the
  watermark ONLY on scan success and rolls back on failure (INV-3 / fixes G2);
  min-merges children to earliest creation (INV-4). Pure planDiscovery/splitWindows.
- portal-assemble.ts: the pure range assembler — interval exactness (INV-2),
  full-tree trace ranking then match (INV-5), seenTx dedupe, closest by loop (C9).
- portal-transform.ts: named raw-NDJSON types; re-export traceSafeChunkBlocks.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(portal): rewrite portal.ts as the orchestration shell; tidy realtime

- portal.ts: createPortalHistoricalSync is now a ~300-line imperative shell that
  wires config→client→gate→filters→chunks→discovery→assemble→metrics and owns only
  the chunk cache, stash, and delegation. Seam methods carry invariant checks at the
  boundaries. Fixes G1 (a rejected chunk promise is evicted immediately + its rows
  freed, INV-13) and G3 (rows registered per arriving batch via the client hook).
  Behavior + the public seam are frozen.
- portal-realtime.ts: use the shared ndjsonLines(); delete the stray bottom
  re-exports (_hx, type re-exports).
- portal-realtime-wire.ts: import log-request construction + field projections from
  portal-filters (delete the duplicated logRequestsFor/mergeLogRequests/
  PORTAL_MAX_ADDRESSES/BLOCK_FIELDS); keep the entry symbols.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(portal): invariant catalog + module-map updates

- portal/INVARIANTS.md: the centerpiece — INV-1…INV-14 with statement · rationale ·
  enforced-by · checked-at · tested-by, plus the G1/G2/G3 fix mapping. Legacy C#
  tags mapped.
- HOW-IT-WORKS.md / INTEGRATION.md: refreshed the "Where the code lives" map for the
  new module layout; link INVARIANTS.md.
- README.md: add an "Invariants" pointer under Learn more.
- versions.json: note the next publish from main is 0.16.6-sqd.2 (DO NOT publish).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(build): provision fast-check from a temp dir outside the clone

npm walks up to the nearest package.json, so installing fast-check inside the cloned
ponder workspace tripped on the post-rename workspace. Install it in a mktemp dir and
copy fast-check + pure-rand into the core's node_modules.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(portal): review round — G2 propagation, typed-token row accounting, INV-9 assert

- portal-discovery: a predecessor extension's failure now PROPAGATES (await earlier
  without swallowing) so a successor can never confirm coverage over an unscanned
  gap (B1/G2 — silent factory-child loss reproduced in review).
- portal.ts: per-fetch RowToken accounting (S1 — a stale evicted fetch can't
  register/free rows against its replacement); coveredTo recorded per cache entry;
  INV-9 runtime assert at request build; INV-12 stash assert now strict-mode-only
  (production keeps pre-refactor overwrite semantics + debug log, S2);
  refreshPortalHead retry via client (injectable sleep, S5k).
- portal-client: finalizedHeadRetry; countRows includes headers (S5g); dead
  NoProgressError guard removed — progress is by construction (B4c).
- portal-errors: NoProgressError deleted; InvariantViolation message points at
  INVARIANTS.md + PORTAL_CHECKS escape hatch (S5b).
- portal/config.ts restored (repo-side compat-harness utility — B2); PUBLISHING.md
  stale sentence fixed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(portal): port main's PR #2 (INV-15 child persistence) + PR #5 frontier extend into the architecture

Rebase follow-ups onto current main:

- INV-15 (PR #2 a600fc9, ported): portal-discovery queues newly-discovered children
  inside the min-merge guard (store-preloaded children re-discover at prev ≤ bn and
  are never re-queued) and exposes takePendingInRange/restorePending; the portal.ts
  shell flushes the interval-scoped set via syncStore.insertChildAddresses inside the
  SAME syncBlockRangeData whose transaction marks the factory interval cached. A
  failed flush restores the queue and fails the interval loud. All six of PR #2's
  regression tests pass against this architecture unchanged.
- INV-13 frontier extend (open PR #5's semantics, adopted — supersedes the
  evict+refetch approach): each cache entry records coveredTo; a hit past it streams
  ONLY the newly-finalized tail via the shared runStreams (append-only merge, rows
  accounted through the same per-fetch token), with discovery re-driven through the
  tail; a failed extend evicts the whole entry (never leaves the optimistic
  high-water — the G2 lesson). Fully-finalized/bounded chunks keep coveredTo ==
  desiredTo → clean hit, zero behavior change.
- portal-realtime.ts: INV-10 O(1) parentHash/number link tripwire on every append.
- portal-assemble.ts: vacuous INV-2 per-row assert removed (enforced by construction
  behind the single inRange predicate); dead chunkRowCount deleted.
- portal-client.ts: dead NoProgressError guard removed (progress by construction);
  finalizedHeadRetry (injectable sleep); countRows includes headers.
- realtime.ts: fix the pre-existing noAssignInExpressions lint ERROR (summarize's
  byKey bucket init) — the only semantic-file edit outside the layer rewrite.
- CLAUDE.md style compliance (single var declarators, no assignment-in-expression,
  braced control bodies) + repo-Biome (2.5.2, single-quote) formatting throughout.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(portal): review-round suite — G1/G3 regressions, delegation matrix, interleaving property, mutation killers

- portal-shell.test.ts (new): seam-level tests — G1 (rejected chunk evicted, rows
  freed, refetch not cached-rejection), G3 (rows visible to the gate MID-chunk),
  the INV-13 frontier-extend regression (adapted from PR #5), the INV-9 delegation
  matrix (past-head → whole-interval RPC via a stubbed rpc + delegated-key
  consume-once; unknown head → delegation; stream-realtime → warn + empty, NOT
  delegated), INV-12 stash consume-once + on/strict overwrite semantics, and the
  S1 row-accounting model (rows return to baseline through evict/reject/in-flight
  interleavings).
- portal-discovery.test.ts: the G2 INTERLEAVING test (a successor extension chained
  on a failing scan rejects — never confirms over the gap) + a randomized property
  (random ensures + injected window failures → coverage below through() is always
  complete; retry converges).
- portal-client.test.ts: retry budget exhaustion (11 tries), retry-after cap
  (ra*1000 capped 30s), per-batch onRows (G3), finalizedHeadRetry backoff.
- portal-filters.test.ts: INV-11 model matcher extended to topic2/topic3 (+ the
  key-drops-topic2/3 mutation killer).
- portal-chunks.test.ts: saturated read-ahead is EXACTLY depth-1 (mutation killer).
- portal-gate.test.ts: an AIMD ramp pumps parked waiters.
- portal-assemble.test.ts: trace assembly upper-bound (hi+1 excluded).
- portal-metrics.test.ts (new): golden field-for-field metrics-file shape (freezes
  the bench-harness contract).
- fast-check seeds pinned (deterministic CI); pre-existing test files reformatted
  to the repo's own biome.json (single-quote) — proven byte-identical to
  biome(origin/main's versions), i.e. formatting-only.

Mutation-verified: reverting G1 eviction, no-op'ing the G3 hook, re-introducing the
G2 swallow, dropping topic2/3 from the merge key, and breaking saturated depth-1
each fail exactly the intended test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs+build(portal): INVARIANTS honesty pass (INV-15), biome gate, PORTAL_CHECKS docs

- INVARIANTS.md: every row's enforced-by/checked-at/tested-by now cites real,
  passing tests; INV-2 marked enforced-by-construction (vacuous assert removed);
  INV-3 cites the interleaving + randomized properties; INV-9 cites the runtime
  assert + delegation matrix; INV-10 cites the real link tripwire; INV-13 rewritten
  for progress-by-construction + the frontier EXTEND; INV-1 noted as a
  future-regression tripwire; NEW INV-15 row (child persistence, ported from PR #2).
  Fixed-gaps section lists the found-and-fixed head-boundary bug + mutation evidence.
- scripts/sync-upstream.sh: repo-side Biome gate (pinned 2.5.2, repo biome.json,
  --diagnostic-level=error) runs before the copy under --test — an unformatted or
  lint-error portal file fails the build.
- INTEGRATION.md: PORTAL_CHECKS row in the env-knob table.
- PUBLISHING.md/HOW-IT-WORKS.md: stale prose fixed; portal/config.ts restored as the
  repo-side compat-harness utility (= origin/main's copy).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* build: pin fast-check exactly (3.23.2) and copy the resolved tree wholesale

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(portal): reconcile with merged PR #5 (802ceed) — extends counter + test dedupe

Main merged PR #5 while this branch was in flight; the architecture already
implements its frontier-extend semantics, so the rebase resolved portal.ts to the
shell version. Deltas ported to MATCH MAIN (now the contract):
- `stats.extends` counter: PortalStats/createStats, writeMetrics `fetch` block
  (after dataChunks — bench-harness shape), the syncBlockRangeData debug log, and
  the increment on the extend path; golden metrics test updated.
- portal.test.ts carries main's #5 regression test VERBATIM (it passes unchanged
  against this architecture — behavior-parity proof); my earlier adaptation in
  portal-shell.test.ts is kept but renamed to its distinct angle ("frontier extend
  streams ONLY the newly-finalized tail") — it additionally asserts the extend
  request starts at coveredTo+1, which main's test doesn't cover.
- One deliberate divergence, documented in INVARIANTS.md: on a FAILED extend main
  leaves a cached rejection (its pre-existing G1 gap); here the entry is evicted
  and its rows freed — the G1-consistent recovery this PR introduces.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: final-review nits — INV-15 range refs, corrected INV-13 citation, honest test counts

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: dzhelezov

HOW-IT-WORKS.md

@@ -70,7 +70,7 @@ Read-ahead raises an obvious question on a multichain app: how deep, and across
 
 **The naive answer breaks.** Give each of 15 chains its own private read-ahead and you get one of two failures. Either the buffers are unbounded and the process **runs out of memory**, or 15 chains each fan out concurrently against the one shared endpoint and the app **rate-limits itself** into 429s. A per-chain design fights itself.
 
-The fix follows from a fact the naive design ignores: every chain streams from the **same** Portal endpoint, so request concurrency and buffered memory are *one shared budget*, not fifteen private ones. The fork routes all chains through a single module-scope controller (`portalGate`) with two self-tuning, zero-config controls.
+The fix follows from a fact the naive design ignores: every chain streams from the **same** Portal endpoint, so request concurrency and buffered memory are *one shared budget*, not fifteen private ones. The fork routes all chains through a single shared, lazily-created controller (`portal-gate.ts` — a pure AIMD/row-budget reducer behind a process-shared shell) with two self-tuning, zero-config controls.
 
 **Adaptive concurrency (AIMD).** The client cannot know the endpoint's live capacity — it varies by Portal, by load, and over time — so the controller *discovers* it. It starts at 16 in-flight requests, adds 2 after every 8 clean responses up to a ceiling of 48, and **halves** — down to a floor of 8 — on any back-pressure signal: an HTTP 429, any 5xx, a 409 on the finalized stream, a `retry-after`, or a dropped or timed-out connection. This is the same additive-increase / multiplicative-decrease discipline as Ponder's native RPC limiter, but **global**, because the endpoint is shared. Transient failures retry with exponential back-off; only a genuinely unrecoverable response fails the run.
 
@@ -121,11 +121,22 @@ This is proven, not asserted. The differential harness ([`harness/diff`](harness
 
 ## Where the code lives
 
+The layer is organised around explicit, provable **invariants** — a functional core (pure, property-tested) behind an imperative shell. See [`portal/INVARIANTS.md`](portal/INVARIANTS.md) for the catalog (INV-1…INV-15) that ties doc ⟷ code ⟷ test together.
+
 | File | What it holds |
 |---|---|
-| [`portal/portal.ts`](portal/portal.ts) | `createPortalHistoricalSync` — the seam implementation: chunk cache, read-ahead, the shared `portalGate` controller, factory discovery, and the finality-gap fallback. |
+| [`portal/portal.ts`](portal/portal.ts) | `createPortalHistoricalSync` — the orchestration shell: wires the modules below, owns the chunk cache, stash, delegation, and the seam methods with invariant checks. |
+| [`portal/portal-config.ts`](portal/portal-config.ts) | Parse + validate all `PORTAL_*` env once into a frozen `PortalConfig` (INV-14). |
+| [`portal/portal-errors.ts`](portal/portal-errors.ts), [`portal/portal-invariant.ts`](portal/portal-invariant.ts) | The typed error taxonomy and the runtime `invariant()` checks. |
+| [`portal/portal-client.ts`](portal/portal-client.ts) | The Portal HTTP shell: `finalizedHead()`, `stream()`, error mapping, retry, field degradation, the shared `ndjsonLines()`. |
+| [`portal/portal-gate.ts`](portal/portal-gate.ts) | The shared AIMD concurrency + row-budget controller as a pure reducer + async shell. |
+| [`portal/portal-filters.ts`](portal/portal-filters.ts) | The frozen per-chain fetch-spec — the single source of log/tx/trace requests + field projections, shared with realtime. |
+| [`portal/portal-chunks.ts`](portal/portal-chunks.ts) | Pure chunk-grid math (tiling, density scaling, read-ahead / eviction plans). |
+| [`portal/portal-discovery.ts`](portal/portal-discovery.ts) | The factory-child discovery state machine (advance-on-success). |
+| [`portal/portal-assemble.ts`](portal/portal-assemble.ts) | The pure range assembler (interval exactness, full-tree trace ranking, `closest`). |
 | [`portal/portal-transform.ts`](portal/portal-transform.ts) | Pure Portal-NDJSON → Ponder `Sync*` transforms, unit-tested against captured fixtures. |
+| [`portal/portal-metrics.ts`](portal/portal-metrics.ts) | The per-chain stats shape, metrics-file writer, and gate-log ticker. |
 | [`portal/portal-realtime.ts`](portal/portal-realtime.ts), [`portal/portal-realtime-wire.ts`](portal/portal-realtime-wire.ts) | The opt-in Portal `/stream` realtime path and its adapter into Ponder's realtime runtime. |
 | [`portal/wiring/`](portal/wiring/) | The short per-version upstream patch — the `portal` config field and the one-line selector. |
 
-For the tuning knobs, metrics fields, and defaults, see [`portal/INTEGRATION.md`](portal/INTEGRATION.md). For the full measured run, see [`harness/euler-multichain/REPORT.md`](harness/euler-multichain/REPORT.md).
+For the invariant catalog, see [`portal/INVARIANTS.md`](portal/INVARIANTS.md). For the tuning knobs, metrics fields, and defaults, see [`portal/INTEGRATION.md`](portal/INTEGRATION.md). For the full measured run, see [`harness/euler-multichain/REPORT.md`](harness/euler-multichain/REPORT.md).

PUBLISHING.md

@@ -11,9 +11,11 @@ field per chain that routes the historical backfill through SQD Portal (realtime
   number, so a plain mirror (`0.16.6`) can't be re-published after a bad build. Each release is
   published with `--tag latest`, so `npm i @subsquid/ponder` resolves it (npm doesn't auto-pick a
   prerelease otherwise); an exact build can be pinned as `@subsquid/ponder@0.16.6-sqd.1`.
-- **We don't hand-maintain a fork.** This repo holds only the **Portal layer** (`portal/`): two modules
-  (`portal.ts`, `portal-transform.ts`) + a per-version `wiring/<ver>.patch` (the 4 one-line touch-points)
-  + `config.ts` (`withPortal`). That's the entire diff against upstream.
+- **We don't hand-maintain a fork.** This repo holds only the **Portal layer** (`portal/`): the
+  `portal-*.ts` modules (an invariant-first functional core behind the `portal.ts` shell — see
+  `portal/INVARIANTS.md`) + a per-version `wiring/<ver>.patch` (the 4 one-line touch-points). That's
+  the entire diff against upstream. (`portal/config.ts` is a repo-side compat-harness utility — not
+  part of the published build.)
 - **The fork is generated.** `scripts/sync-upstream.sh <ver>` clones `ponder@<ver>`, applies the layer,
   renames the package, and builds — producing the publishable package. Tracking a new ponder version is
   "author one small patch + run the script", not "merge a fork".

README.md

@@ -99,6 +99,7 @@ The free public Portal is ideal for trying the fork and for development, but sha
 ## Learn more
 
 - [**How it works**](HOW-IT-WORKS.md) — the design story: why a streamed range beats per-topic RPC lookups, the historical-sync seam, the shared read-ahead controller, factory discovery over ranges, and where the single-thread ceiling honestly is. Operational reference: [`portal/INTEGRATION.md`](portal/INTEGRATION.md).
+- [**Invariants**](portal/INVARIANTS.md) — the catalog (INV-1…INV-15) the `portal/` layer is built around: each invariant's statement, how it's enforced, where it's checked at runtime, and the property test that proves it.
 - [**Observability**](portal/INTEGRATION.md) — `PORTAL_METRICS_FILE` writes a per-chain JSON metrics file (throughput, bytes, errors, RPC-fallback); `PORTAL_GATE_LOG=1` logs the adaptive controller.
 - **Portal-native realtime** (experimental) — realtime runs on your RPC by default; set `PORTAL_REALTIME=stream` to serve the tip from the Portal's fork-aware `/stream` instead of RPC.
 - [**Versioning & releases**](PUBLISHING.md) — `@subsquid/ponder@<ponder-version>-sqd.<rev>`, generated from upstream Ponder + a per-version patch.

portal/INTEGRATION.md

@@ -44,7 +44,7 @@ Historical sync owns `[start, finalized]`; realtime owns `(finalized, tip]`. The
 
 ## The shared controller
 
-Every chain streams from the **same** Portal endpoint, so request concurrency and buffered memory are one shared budget, not a per-chain one. A 15-chain app that gave each chain its own private read-ahead is exactly what exhausts memory and trips rate limits. The fork therefore routes all chains through a single module-scope controller (`portalGate`) that governs two things, both self-tuning and zero-config:
+Every chain streams from the **same** Portal endpoint, so request concurrency and buffered memory are one shared budget, not a per-chain one. A 15-chain app that gave each chain its own private read-ahead is exactly what exhausts memory and trips rate limits. The fork therefore routes all chains through a single shared controller (`portal-gate.ts` — a pure AIMD/row-budget reducer behind a lazily-created process-shared shell) that governs two things, both self-tuning and zero-config:
 
 ### Adaptive concurrency (AIMD)
 
@@ -128,6 +128,7 @@ The defaults run well without configuration. These environment variables overrid
 | `PORTAL_FINALIZED_HEAD` | unset | Pin the Portal finalized head (testing/ops); overrides the `/finalized-head` probe. |
 | `PORTAL_METRICS_FILE` | unset | Write per-chain JSON metrics to `<path>.<chainId>`. |
 | `PORTAL_GATE_LOG` | unset | Set to `1` to log the shared controller every 20 s. |
+| `PORTAL_CHECKS` | `on` | Runtime invariant checks ([INVARIANTS.md](INVARIANTS.md)): `on` runs the O(1) checks (a violated invariant crashes loud instead of corrupting silently); `strict` adds O(n) whole-structure checks (CI/tests); `off` disables all checks — only as a last-resort perf escape hatch or to bypass a false-positive crash while a fix ships. |
 
 Raising concurrency does not help when indexing is the bottleneck, which for a full-history resync it usually is — the fetch is already ahead. The knobs that matter most in practice are the memory budget (`PORTAL_MAX_ROWS_IN_MEM`) on a constrained box and, for a keyed deployment, `PORTAL_API_KEY`.
 
@@ -148,3 +149,16 @@ The A/B in that run is the useful lesson. A modest, well-tuned configuration bea
 | Avg throughput | 7,024 ev/s | **10,513 ev/s** |
 
 Both runs indexed the identical 28.4M events; only the configuration differed. The lever for going faster is not a bigger heap — it is **sharding chains across processes** to lift the single-thread indexing ceiling (for example, running the event-heaviest chain on its own). The full write-up is in [`harness/euler-multichain/REPORT.md`](../harness/euler-multichain/REPORT.md).
+
+## Where the code lives
+
+The `portal/` layer is a functional core behind an imperative shell, organised around explicit invariants (see [`INVARIANTS.md`](INVARIANTS.md), INV-1…INV-15):
+
+- `portal.ts` — orchestration shell (`createPortalHistoricalSync`): chunk cache, stash, delegation, seam methods.
+- `portal-config.ts` / `portal-errors.ts` / `portal-invariant.ts` — frozen config (INV-14), typed errors, runtime checks.
+- `portal-client.ts` — HTTP: `finalizedHead()`, `stream()`, error mapping, retry, field degradation, `ndjsonLines()`.
+- `portal-gate.ts` — the shared AIMD + row-budget controller (pure reducer + shell).
+- `portal-filters.ts` — the frozen per-chain fetch-spec (log/tx/trace requests + field projections), shared with realtime.
+- `portal-chunks.ts` — pure chunk-grid math; `portal-discovery.ts` — factory-child discovery; `portal-assemble.ts` — the pure range assembler.
+- `portal-transform.ts` — Portal-NDJSON → Ponder `Sync*` transforms; `portal-metrics.ts` — stats + metrics file.
+- `portal-realtime.ts` / `portal-realtime-wire.ts` — the opt-in Portal `/stream` realtime path.