|
3 | 3 | The `portal/` layer is organised around **explicit, provable invariants**. Every module has a functional |
4 | 4 | core (pure, unit-tested) behind an imperative shell; the core asserts these invariants at runtime |
5 | 5 | (`invariant("INV-…", …)` from `portal-invariant.ts`) and the test suite proves them with property-based |
6 | | -tests (`fast-check`, seed-pinned for deterministic CI). IDs are stable (INV-1…INV-21) and grep-able |
| 6 | +tests (`fast-check`, seed-pinned for deterministic CI). IDs are stable (INV-1…INV-24) and grep-able |
7 | 7 | across **doc ⟷ code ⟷ test**. (INV-19 is the low-edge chunk-coverage guard from #50; INV-20 is the |
8 | | -ancestor-error trace cascade; INV-21 is the root-frame gas parity — the numbering is chronological, not |
9 | | -table order.) |
| 8 | +ancestor-error trace cascade; INV-21 is the root-frame gas parity; INV-22 is the realtime heartbeat |
| 9 | +finality cadence; INV-23 is the idle-bounded read + tick-transparent line-wait keeping that cadence |
| 10 | +alive on a silent-open connection; INV-24 is the delivery-progress watchdog — a live chain whose /stream |
| 11 | +is starving indexing fails loud, a quiet chain never does — the numbering is chronological, not table order.) |
10 | 12 |
|
11 | 13 | **Runtime modes** (`PORTAL_CHECKS`, default `on`): `off` disables all checks; `on` runs the O(1) checks |
12 | 14 | and throws `InvariantViolation` (a loud crash beats silent corruption); `strict` additionally runs the |
@@ -39,6 +41,9 @@ Legacy `C#` tags from the pre-refactor comments are mapped where they existed. |
39 | 41 | | **INV-18** | **Bounded-cutover skip.** An end-capped chain (configured end ≤ finalized) is never live-probed at the historical→realtime cutover, and a fully-bounded app does zero cutover refetches — with omnichain refinements (pending-aware break, two-phase probe, clamp bypass) that never drop parked cross-chain events. **[INV-18 in detail ↓](#inv-18-in-detail)** | — | — | — | — | |
40 | 42 | | **INV-20** | **Ancestor-error trace cascade.** Portal historical trace assembly replicates stock-ponder's ancestor-error cascade: within one transaction's trace tree a reverted ancestor's `error`/`revertReason` smears onto every descendant that lacks its OWN error, transitively (an inheriting frame becomes a source for its own children); a frame with its own error keeps it (own-error-wins). So `traces.error`/`traces.revert_reason` are byte-identical to the RPC store path. The inherited `revertReason` is assigned UNCONDITIONALLY (even when the ancestor's is `undefined`) — byte-exact with upstream `rpc/actions.ts`. | Portal delivers geth-faithful per-frame errors and never invokes upstream's `debug_traceBlock*` DFS, so a succeeded child of a reverted parent stored `error=null` where the RPC path inherits the parent's error — a `traces.error`/`revert_reason` divergence between the two store paths. Realtime/fallback already flow through the untouched RPC cascade, so the fork DB must be uniformly ponder-shaped. | `portal-assemble.ts`: `cascadeTraceErrors(sorted)` derives each frame's effective error over the DFS-sorted (`cmpTraceAddr`) frames of a tx, keyed by traceAddress prefix (parent of `[a,b,c]` is `[a,b]`); `rankTraces` applies it to each produced frame. Applied ONLY on the historical path — the realtime path rides the untouched upstream cascade and cannot double-apply. Non-prefix-closed input inherits nothing (conservative; never fabricates a smear; cannot occur for geth-faithful trees). | — (byte-identity property; the divergence it prevents has no safe runtime signal on the app path, so it is proven by test against the RPC-store oracle). | `portal-assemble.test.ts` — "INV-20: real reverted tx — the whole reverted subtree inherits the ancestor error (oracle: RPC store)" (real eth block 20351061 tx 26 fixture), "INV-20 (a): a child with its OWN distinct error SURVIVES (own-error branch wins)", "INV-20 (b): ≥3-deep nesting inherits the ancestor error transitively", "INV-20 (c): a non-reverted tx keeps every frame error/revertReason null", "INV-20 (d): an inheriting child of an ancestor with an error but NO revertReason gets error set AND revertReason === undefined (unconditional assignment)". Mutation-verified: neutering the cascade (empty effective-error map / skipping the applied `cascaded`) fails INV-20 / (a) / (b) / (d) while (c) still passes. Upstream divergence documented in VALIDATION.md §5. | |
41 | 43 | | **INV-21** | **Root-frame gas parity.** Portal historical trace assembly stores the transaction's FULL gasLimit on the ROOT trace frame (`traceAddress []`, the top-level call), byte-identical to the stock-ponder RPC store. Portal serves Parity-style traces whose root `action.gas` is gasLimit MINUS the EIP-2028 intrinsic (base 21000 + 16/nonzero + 4/zero calldata byte + access-list + creation costs), while geth callTracer — which the RPC path stores — seeds the top frame's `gas` with the full tx.gasLimit before any intrinsic deduction. Non-root frames' gas already matches both paths and is left untouched. (Grounded on 2196 root frames — legacy + EIP-1559, intrinsic-independent; contract-creation and access-list roots were absent from the corpus, so those tx classes rest on the geth callTracer mechanism, not a direct measurement.) | Portal delivers Parity `action.gas` (intrinsic-reduced) on the root frame where geth's callTracer top frame carries the full gasLimit, so the Portal store wrote `traces.gas` at the root that differed from the RPC store by exactly the tx's intrinsic (gas_used byte-identical) — a `traces.gas` divergence between the two store paths. Realtime/fallback ride the untouched RPC path, so a mixed backfill+realtime fork DB would carry a path-dependent root-frame gas with an invisible seam at the cutover — the fork DB must be uniformly ponder-shaped. | `portal-assemble.ts`: `buildTraces` overrides the root frame's gas with the parent tx's gasLimit (`frame.gas = hx(rawTx.gas)`) when `traceAddress.length === 0` and the parent tx is present. Keyed on `traceAddress`, NOT the DFS rank (a chunk can hold only deep frames). No intrinsic math — geth's top-frame gas equals the gasLimit exactly, so the override is the raw gasLimit. Applied at the store-write site only; the shared trace matcher (`buildRawTraceMatcher`/`parityToCallFrame`) is untouched (trace matching never reads gas). | — (byte-identity property; the divergence it prevents has no safe runtime signal on the app path, so it is proven by test against the RPC-store oracle). | `portal-assemble.test.ts` — "INV-21: real diff-root tx — the root frame carries the tx gasLimit (oracle: RPC store), non-root untouched" (real eth block 19068568 tx 7 Uniswap V2 swap: pre-fix Portal `action.gas` 0x74700 = gasLimit − intrinsic vs RPC/gasLimit 0x7a120; non-root child 0x3616b already byte-identical), "INV-21 (a): an already-matching root … is unchanged — idempotent", "INV-21 (b): a rootless trace chunk (only a deep frame) is NOT rewritten". Mutation-verified: reverting the root override (guard never fires ⇒ Parity `action.gas` at root) fails INV-21 (`expected '0x74700' to be '0x7a120'`); (a) and (b) still pass. Grounded on a real captured Portal-vs-RPC store diff (eth mainnet): 133/133 divergences gas-only, at the root, gas_used byte-identical, RPC.gas == tx gasLimit on every root frame. Upstream divergence documented in VALIDATION.md §5. | |
| 44 | +| **INV-22** | **Realtime heartbeat finality cadence.** The finalize poll and B1 defer watchdog run at ≤2× `finalizePollMs` regardless of delivery, with latency slack equal to one `/stream` request attempt. | A no-delivery outage (204s, bodyless 200s, reconnect loops, transient fetch errors, or 429/5xx load) must not freeze finality or the B1 watchdog until another block arrives. | `portalRealtimeEvents` derives `half = max(1, floor(finalizePollMs / 2))`, threads bounded `tickSleepMs`/`errorSleepMs` into `streamHotBlocks`, and runs `runFinalizeCadence()` on every `{kind:'tick'}`. `streamHotBlocks` emits zero-payload ticks before no-data sleeps; the 204 E2 tick site is the representative heartbeat source and E1/E3/E4/E5 use the same contract. | — (cadence is a wall-clock construction property; runtime checks would be timing-flaky). | `portal-realtime.test.ts` — "portalRealtimeEvents: heartbeat ticks keep finalize polling on a 204-only wall-clock cadence (RT-1 SC2 T2)" pins a 204-only stream with zero deliveries and asserts repeated `finalizedHead` calls while no events are emitted; T1/T3 prove the same tick clock finalizes a delivered block and trips the B1 fatal during a post-delivery 204 stall. | |
| 45 | +| **INV-23** | **Idle-bounded read + tick-transparent line-wait.** An OPEN-but-silent `/stream` connection (headers OK, body never delivers, no FIN/RST) keeps the INV-22 finalize/B1 cadence alive: while the NDJSON read is suspended waiting for the next line, `streamHotBlocks` yields `{kind:'tick'}` every `tickSleepMs` of silence WITHOUT cutting the connection, and after `idleMs` of CUMULATIVE silence recycles the connection (reconnect from `cursor`, NOT fatal). A slow-but-alive stream is never cut — the idle guard re-arms on every received chunk; a heartbeat tick neither consumes/reorders a line nor advances progress toward `idleMs`. Completes INV-22: the silent-open read is the one no-delivery state the loop-turn ticks cannot reach (R1). | INV-22's ticks fire only on loop turns (204/error/reconnect); a wedged connection that delivers headers then stalls the body has no loop turn, so pre-SC1 the `for await` suspended FOREVER — no blocks, no finalize polls, no watchdogs (the same silent-freeze class PR #16 hardened the backfill client against). Cutting a merely-slow stream is equally wrong (needless reconnect churn), so the idle bound must count only cumulative silence and re-arm per chunk. | `streamHotBlocks` drives the reader MANUALLY: `ndjsonLines(res.body, undefined, idleMs)` arms the per-chunk idle guard; a single pending `it.next()` is held across `raceHeartbeat(readerP, tickSleepMs, signal)` races and NEVER re-requested until it settles (order preserved, no line dropped), yielding a tick when the heartbeat wins. `raceHeartbeat` clears its timer whoever wins (no timer accrues). On idle expiry the read rejects (`'idle timeout'`) → caught → reconnect from `cursor` + `onIdleReconnect` debug seam. `idleMs` default 120_000, env `PORTAL_STREAM_IDLE_MS` (positive-integer, else loud) via `resolveStreamIdleMs`, mirroring `resolveRedeliveryTimeoutMs`. | — (a liveness/cadence construction property; runtime checks would be timing-flaky). | `portal-realtime.test.ts` — "tick-transparent line-wait keeps finalize polling on a SILENT-OPEN connection (RT-1 SC1)" delivers exactly one block then holds the connection open+silent and asserts finalize fires on the tick clock after 1 block; "idle bound reconnects from cursor on a silent-open stream (RT-1 SC1)" proves the idle bound reconnects at the advanced cursor; "idle bound fires onIdleReconnect and a slow-but-alive stream is NOT cut (RT-1 SC1)" pins the never-cut invariant. `portal-realtime-wire.test.ts` — "resolveStreamIdleMs: precedence … (RT-1 SC1)" pins the env validation. Mutation-verified: removing the tick-transparent wait fails the silent-open test (`SILENT-OPEN-STARVED`); removing the idle bound fails the reconnect test (`IDLE-RECONNECT-STARVED`); each fails independently and on the pre-SC1 (SC2) code. | |
| 46 | +| **INV-24** | **Delivery-progress watchdog.** When the probed finalized head has advanced ≥ `deliveryProgressThreshold` blocks past the HIGHEST DELIVERED block number WHILE ZERO blocks were delivered for ≥ `deliveryProgressMaxMs`, `portalRealtimeEvents` fails LOUD (throws with `diagDump`, never an RPC fallback). The baseline is the delivered HEIGHT — `max(first-observed finalized head, highest delivered block number)` — NOT the finalized head observed at the last delivery (RT-G10 fix). The `/stream` delivers UNFINALIZED tip blocks that run FAR ahead of the finalized head, so finality CATCHING UP over already-delivered blocks (finalized ≤ delivered — e.g. an L2 tip freeze while finality drains its backlog) can NEVER trip it; only finality climbing ≥ threshold BEYOND the highest delivered block — provably new blocks the stream is not delivering — does. PROGRESS-CONDITIONED: a quiet/halted chain (finality static, or ≤ threshold past the delivered tip) NEVER trips it (parity with RPC-realtime idling); a single-block finality lag is below the threshold by construction. The silent E1 fetch-error retry loop counts as non-delivery (it yields a tick, stamps no delivery) and gains a rate-limited warn at the wire. Consumer-only; the tick is the clock, wall time is the data; the block arm never probes, the tick arm never touches the window. | INV-22/INV-23 keep finality POLLING alive during a no-delivery outage, but a chain that has genuinely FINALIZED blocks ABOVE everything the `/stream` delivered — endless 204s/reconnect churn or a transient-fetch-error retry storm — would index nothing indefinitely with no signal: the finalize poll keeps succeeding (head known) so B1 never arms, yet zero blocks reach the store. Bounding it on the finalized-vs-DELIVERED gap turns that silent stall into a loud, restartable fatal; baselining on the delivered HEIGHT (not the finalized head at delivery) is what keeps a legitimate tip-freeze-while-finality-catches-up from false-fataling — the old baseline used FINALITY progress as a proxy for NEW-BLOCK production and the two diverge on a tip-freeze. | `portalRealtimeEvents` initializes `lastDeliveryMs` to loop entry (an outage from process start arms from start, not epoch 0) and `deliveryBaseline` to undefined; the FIRST successful finalize probe folds its head into the baseline via `deliveryBaseline = max(deliveryBaseline ?? fhNumber, fhNumber)` WITHOUT resetting the clock — the first probe, not the resume anchor, so a post-downtime resume may backfill a pre-existing finality deficit for the whole bound without false-fataling. This first-probe fold is gated on a dedicated `firstProbeAdopted` flag, NOT on `deliveryBaseline === undefined`, precisely so it still fires when a delivery seeded the baseline to the delivered height BEFORE the first poll ran (the deliver-before-first-poll resume: on a resume the `/stream` hands us the anchor before the finalize cadence ticks once) — keying the fold on `undefined` alone silently dropped the first-probed head on that ordering and false-fataled a static pre-existing deficit (RT-G10, glm Finding 1). The block arm stamps `lastDeliveryMs = Date.now()` and RAISES `deliveryBaseline = max(deliveryBaseline ?? light.number, light.number)` on each delivered block (max, never lower — a resume that adopts a high head then delivers lower catch-up blocks must not regress the baseline); `runFinalizeCadence` evaluates the trip on the head it ALREADY probes and throws when the finalized head is ≥ threshold past `deliveryBaseline` and the time bound is crossed. `streamHotBlocks` calls `onFetchError` at the E1 catch (yields a tick, no delivery stamp). `deliveryProgressMaxMs` default 600_000 (aligned with B1), `deliveryProgressThreshold` default 16 blocks, both env-tunable via `resolveDeliveryProgressMaxMs`/`resolveDeliveryProgressThreshold` (`PORTAL_STREAM_DELIVERY_MAX_MS`/`PORTAL_STREAM_DELIVERY_THRESHOLD`, positive-integer, else loud), mirroring `resolveStreamIdleMs`. | — (a wall-clock liveness construction property; runtime checks would be timing-flaky). | `portal-realtime.test.ts` — "the finalized head advancing while ZERO blocks are delivered is FATAL … (RT-1 SC3 T1)" delivers one block then 204s forever under a climbing number-only head and asserts the loud fatal on a deterministic clock; "a QUIET chain (head STATIC) … NEVER trips the watchdog … (RT-1 SC3 T2)" and "a SINGLE-block finality lag … does NOT trip … (RT-1 SC3 T3)" pin the progress-conditioning (no throw though the time bound is crossed many times over); "FINALITY catching up over ALREADY-DELIVERED blocks (finalized head ≤ highest delivered) must NOT trip the watchdog … (RT-1 SC3 T4 / RT-G10)" pins the false-fatal fix — a delivered tip 500 with finality merely draining 100→120 over already-delivered blocks DRAINS with no throw; "a GENUINE post-delivery stall — finalized head climbs ≥ threshold BEYOND the highest delivered block … STILL fatals (RT-1 SC3 T5 / RT-G10)" pins the must-still-fire half — finality climbing past the delivered tip 500 throws the loud fatal; "a RESUME that delivers its anchor BEFORE the first finalize poll, then sees a STATIC finalized head far above it, must NOT false-fatal … (RT-1 SC3 T6 / RT-G10 / glm Finding 1)" pins the deliver-before-first-poll fold — anchor 100 delivered before any poll, then a STATIC head 500 far above it DRAINS with no throw (baseline folds to `max(500,100)=500`, so `500−500=0 < threshold`); "a transient /stream fetch throw invokes onFetchError … and yields a tick, not a block (RT-1 SC3)" pins the E1 non-delivery seam. Mutation-verified: disabling the trip by neutering the comparison's finalized-head term to the baseline itself (`fhNumber - deliveryBaseline` → `deliveryBaseline - deliveryBaseline`, type-valid ⇒ `0 ≥ threshold` never true) fails BOTH genuine-stall tests T1 and T5 with `PROGRESS-WATCHDOG-STARVED` while the false-fatal repro T4 stays GREEN (T2/T3 pass); the symmetric sign-flip (`deliveryBaseline - fhNumber`) instead RE-INTRODUCES the false-fatal (T4 red) — proving the delivered-height baseline is load-bearing in BOTH directions; dropping the first-probe fold (`max(deliveryBaseline ?? fhNumber, fhNumber)` → `deliveryBaseline ?? fhNumber`, type-valid) RE-INTRODUCES the deliver-before-first-poll false-fatal (T6 red with the `400 blocks PAST the highest delivered block 100` fatal) while T1–T5 stay GREEN — proving the first-probe `max` fold is load-bearing for the resume-grace case. | |
42 | 47 |
|
43 | 48 | ## Invariants in detail |
44 | 49 |
|
|
0 commit comments