Skip to content

Commit 0518da8

Browse files
dzhelezovclaude
dzhelezov
and
claude
committed
docs: rewrite portal/INTEGRATION.md from scratch against shipped code
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2 parents 1673727 + 78bffac commit 0518da8

1 file changed

Lines changed: 133 additions & 42 deletions

File tree

‎portal/INTEGRATION.md‎

Lines changed: 133 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -1,59 +1,150 @@
1-
# Integrating `createPortalHistoricalSync` into `@ponder/core`
1+
# How the Portal backfill works
22

3-
Three localized edits turn `packages/portal-sync` into a drop-in core fork. Verified against `ponder-sh/ponder` HEAD (cloned). Realtime is untouched.
3+
`@subsquid/ponder` changes one thing about Ponder: the source of the **historical** backfill. Instead of replaying history over JSON-RPC — one `eth_getLogs` per topic, one `eth_getBlockByNumber` per matched block — a chain with a `portal:` source streams its history from the [SQD Portal](https://sqd.dev/portal/). The Portal is an HTTP access layer to the SQD Network: it answers an arbitrary `[from, to]` block range as a single streamed response, so hundreds of thousands of small round-trips collapse into a handful of large streamed reads.
44

5-
## 1. Config surface — `packages/core/src/config/index.ts`
5+
Everything else is unchanged. Handlers, schema, config, the `ponder` CLI, realtime, and Ponder's reorg handling all stay the same. This document describes the layer that lives in `portal/`: where it plugs into Ponder, how it keeps every chain fed without exhausting memory, and how to observe it.
66

7-
Add an optional `portal` field to `ChainConfig` (~line 92):
7+
## The historical-sync seam
88

9-
```diff
10-
type ChainConfig<chain> = {
11-
id: chain extends { id: infer id extends number } ? id | number : number;
12-
rpc: string | string[] | Transport | undefined;
13-
+ /** SQD Portal dataset URL for historical backfill. Realtime still uses `rpc`. */
14-
+ portal?: string;
15-
ws?: string;
16-
...
17-
};
9+
Ponder's engine drives the backfill by handing its historical sync one block interval at a time. That contract is two methods on `HistoricalSync`:
10+
11+
- `syncBlockRangeData({ interval, ... })` — fetch and persist the data an interval needs, return its matched logs.
12+
- `syncBlockData({ interval, ... })` — finalize the interval and return the highest block that carried data.
13+
14+
The fork implements exactly this interface in `portal/portal.ts` (`createPortalHistoricalSync`) and selects it per chain in Ponder's `runtime/historical.ts`:
15+
16+
```ts
17+
const historicalSync = params.chain.portal
18+
? createPortalHistoricalSync(params) // stream the interval from the Portal
19+
: createHistoricalSync(params); // stock RPC path, unchanged
1820
```
1921

20-
Thread it through `Chain` in `internal/types.ts` (~line 299) and the config build in `build/config.ts`.
21-
22-
## 2. Selection — `packages/core/src/runtime/historical.ts:1224`
23-
24-
```diff
25-
- const historicalSync = createHistoricalSync(params);
26-
+ const historicalSync = params.chain.portal
27-
+ ? createPortalHistoricalSync({
28-
+ chainId: params.chain.id,
29-
+ dataset: params.chain.portal,
30-
+ metrics: params.common.portalMetrics, // see §3
31-
+ sources: toPortalSources(params.eventCallbacks.map((c) => c.filter), params.chain),
32-
+ childAddresses: params.childAddresses, // reuse Ponder's live factory map
33-
+ })
34-
+ : createHistoricalSync(params);
22+
The Portal sync receives the **same** `params` as the stock sync — `chain`, `rpc`, `childAddresses`, and the chain's full `eventCallbacks` (its complete filter set). No handler or config shape is translated; the fork reads Ponder's own filters and factory map directly. Because it satisfies Ponder's existing interface rather than a public API, the integration is a small fork generated from upstream Ponder plus a short wiring patch (`portal/wiring/`), and it tracks Ponder closely — the seam is identical across the tested range (0.15.17–0.16.6, see [`versions.json`](../versions.json)).
23+
24+
Within an interval the two methods split the work:
25+
26+
- `syncBlockRangeData` resolves the interval to the Portal chunks that cover it, transforms the rows falling inside `[interval[0], interval[1]]` into Ponder's `Sync*` shapes, inserts the logs immediately, and stashes the interval's blocks, transactions, receipts, and traces keyed by the interval.
27+
- `syncBlockData` flushes that stash — blocks, then transactions, receipts, and traces — and returns the highest block with data so Ponder can advance its checkpoint.
28+
29+
### The read-ahead chunk buffer
30+
31+
Ponder feeds intervals that are small relative to a full history. A Portal request, on the other hand, is latency-bound per call but has very large parallel bandwidth, so serving one small interval per request would waste it. The fork instead fetches **large aligned chunks** and serves every interval from an in-memory cache keyed by chunk index. A chunk covers `PORTAL_CHUNK_BLOCKS` blocks (default 500,000), and the fork scales that up by the chain's block density — roughly `head ÷ 25,000,000`, capped at 25M blocks — so a high-block-rate chain like Arbitrum takes proportionally fewer round-trips rather than 19× as many 500k chunks. The Portal charges by data touched, not by block width, so wider chunks cut round-trips at no extra cost. Set `PORTAL_CHUNK_FIXED` to disable density scaling. Trace and block-interval sources return much denser data and are capped to `PORTAL_TRACE_CHUNK_BLOCKS` (default 2,000) so a single chunk can't buffer a busy contract's entire trace set.
32+
33+
Chunks are also **prefetched**: as an interval finishes, the fork issues the next chunks concurrently (up to `PORTAL_READAHEAD` deep, default 6) so the Portal's per-request latency overlaps with indexing instead of serializing in front of it. How far read-ahead actually runs is bounded by the shared memory budget below, not by a fixed count. Chunks behind the current interval are evicted as it advances.
34+
35+
## The backfill → realtime handoff
36+
37+
Historical sync owns `[start, finalized]`; realtime owns `(finalized, tip]`. The Portal serves only **finalized** data, which keeps the split clean.
38+
39+
**Realtime stays on your RPC by default.** When the backfill reaches the finalized head, Ponder hands off to its own realtime sync over `rpc`, with its own reorg handling — unchanged from stock Ponder. The Portal is never in the realtime path unless you opt in.
40+
41+
**Finality-gap fallback.** The Portal's finalized head can occasionally lag Ponder's target finalized block. Any interval that reaches past the Portal head — or that runs while the head probe is failing — is delegated *whole* to the stock RPC `createHistoricalSync`, so the tip is never silently under-served. This is why `rpc` must stay configured. `PORTAL_FINALIZED_HEAD` pins the head for testing and ops.
42+
43+
**Portal-native realtime** (opt-in, experimental). Set `PORTAL_REALTIME=stream` to serve the tip from the Portal's fork-aware `/stream` instead of RPC. In this mode `clampFinalizedToPortalHead` lowers Ponder's finalized block to the Portal head, so historical stops exactly there and realtime streams `[portal-head+1 → tip]`, reconciling reorgs from the stream's parent-hash chain. The RPC finality-gap fallback is then neither needed nor used. Your RPC is still used for chain setup and `readContract`. With the flag unset or `rpc`, this path is inert and the RPC realtime path is byte-for-byte unchanged.
44+
45+
## The shared controller
46+
47+
Every chain streams from the **same** Portal endpoint, so request concurrency and buffered memory are one shared budget, not a per-chain one. A 15-chain app that gave each chain its own private read-ahead is exactly what exhausts memory and trips rate limits. The fork therefore routes all chains through a single module-scope controller (`portalGate`) that governs two things, both self-tuning and zero-config:
48+
49+
### Adaptive concurrency (AIMD)
50+
51+
The controller caps how many Portal requests are in flight across all chains and adapts that cap to the endpoint's live capacity, which the client cannot know ahead of time and which drifts over time. It follows additive-increase / multiplicative-decrease:
52+
53+
- **Start** at `PORTAL_START_CONCURRENCY` (16).
54+
- **Increase** by 2 after every 8 clean responses, up to `PORTAL_MAX_CONCURRENCY` (48).
55+
- **Halve** — down to `PORTAL_MIN_CONCURRENCY` (8) — on any back-pressure signal: HTTP 429, any 5xx, a 409 on the finalized stream, a `retry-after`, or a dropped/timed-out connection.
56+
57+
This mirrors Ponder's native RPC AIMD, but it is global because the endpoint is shared. A request that back-off marks as transient is retried with exponential back-off (honoring `retry-after`); only a genuinely unrecoverable response fails the run.
58+
59+
### Memory backpressure
60+
61+
Concurrency is bounded above; memory is bounded by a **row budget**. `PORTAL_MAX_ROWS_IN_MEM` (default 250,000) caps the log/transaction/trace/block records held live across every chain's read-ahead — roughly 1.5–2.5 GB once Ponder's derived copies are counted. Read-ahead prefetches until the shared buffer reaches the budget, then pauses; as indexing consumes chunks they are evicted and the buffer refills. So the buffer stays full enough that indexing rarely waits on a fetch, while total memory stays capped no matter how many chains run at once. Read-ahead always keeps at least the immediate next chunk ready per chain, and goes deeper only while the shared buffer is below budget. Raise the budget together with a larger Node `--max-old-space-size`; lower it when the indexer is memory-constrained.
62+
63+
## Factory discovery
64+
65+
A factory source (an EVault factory that emits thousands of child vaults, say) has to be discovered before its children's events can be fetched. The fork decouples the **discovery** timeline from the **data** timeline:
66+
67+
- Discovery is a wide scan of the factory's creation event over `[deploy, head]`, pinned to the factory's real deploy block rather than block 0. The Portal serializes a single stream in block order, so the scan is split into `PORTAL_DISCOVERY_WINDOWS` (default 8) disjoint windows issued concurrently; each window additionally fans out across `PORTAL_BUFFER_SIZE` (default 100) chunk workers on the Portal side, at no extra cost.
68+
- A data chunk only fetches once discovery has completed *through its own block range*. Data chunks may be fetched out of order, but no child event is ever missed, because the child set is known to be complete up to each chunk's blocks before that chunk streams.
69+
70+
Discovered child addresses are pushed into the Portal's server-side log filter (`address` + `topic0..3`), so a factory with thousands of children still resolves to a small number of streamed reads rather than per-address lookups.
71+
72+
## Correctness: byte-identity with `eth_getLogs`
73+
74+
Every row filter is pushed to the Portal's native server-side filters — logs by address and topics, traces by call target / caller / sighash, account transactions by from / to. The only client-side row filter is the block-interval (offset/modulo) test, which the Portal has no native equivalent for. Field projection requests exactly the columns Ponder's sync store persists, and header fields are normalized so a stored block is byte-identical to the RPC path (which always carries `nonce`, `mixHash`, `sha3Uncles`, `totalDifficulty`).
75+
76+
The result is that Portal-derived rows are **byte-identical** to what the stock RPC backfill would store, across logs, transactions, receipts, and traces. The differential test in [`harness/diff`](../harness/diff) proves it: it indexes the same bounded range twice on this fork — once with `portal:` set (the Portal path) and once without it (the stock RPC path; the only difference is the backfill source) — into two separate `ponder_sync` stores, then diffs every row of `logs`, `transactions`, `transaction_receipts`, and `traces`. Exit `0` means identical.
77+
78+
## Observability
79+
80+
Two opt-in channels expose what the backfill and the shared controller are doing. Neither is on by default.
81+
82+
### Per-chain metrics — `PORTAL_METRICS_FILE`
83+
84+
Set `PORTAL_METRICS_FILE` to a path and the fork writes one JSON file per chain, at `<path>.<chainId>`, rewritten as each interval finalizes and holding cumulative counters for the run:
85+
86+
| Field | Meaning |
87+
|---|---|
88+
| `chain`, `chainId` | Chain name and id. |
89+
| `wallMs` | Wall-clock since this chain's first interval. |
90+
| `chunkBlocks` | Effective chunk width after density scaling. |
91+
| `portalFinalizedHead` | The Portal's finalized head (or `null` if unknown). |
92+
| `fetch` | `dataChunks`, `discChunks`, `http`, `bytes`, `errors`, `retries`, `cacheHits`, `maxInflight`. |
93+
| `timing` | `gateWaitMs`, `fetchMs`, `transformMs` (cumulative). |
94+
| `portalGate` | Controller snapshot: `limit`, `active`, `rows`. |
95+
| `inserted` | Rows written: `logs`, `blocks`, `txs`, `receipts`, `traces`. |
96+
| `rpcFallbackIntervals` | Intervals delegated to RPC across the finality gap. |
97+
98+
The `timing` split isolates where each request's time goes: `gateWaitMs` is time blocked on the shared concurrency budget, `fetchMs` is Portal I/O (POST plus stream drain), and `transformMs` is NDJSON-to-`Sync*` decode. Database-write time is Ponder's and is reported by Ponder, not here.
99+
100+
### Controller log — `PORTAL_GATE_LOG`
101+
102+
Set `PORTAL_GATE_LOG=1` to log the shared controller every 20 seconds:
103+
35104
```
105+
[portalGate] concurrency_limit=48 active=12 buffered_rows=180000
106+
```
107+
108+
`concurrency_limit` is the current adaptive ceiling, `active` the requests in flight, and `buffered_rows` the read-ahead depth against the memory budget. When `buffered_rows` sits near `PORTAL_MAX_ROWS_IN_MEM`, the fetch is ahead and indexing is the bottleneck — the Portal is not.
36109

37-
`toPortalSources(filters, chain)` maps Ponder's `Filter`/`Factory` (`internal/types.ts:54-183`) to this fork's `PortalSources`:
38-
- `LogFilter` → `logFilters[]` (address + topic0..3)
39-
- `Factory` → `factories[]` (factory address + discovery `eventSelector` + `childAddressLocation` → `child` rule + the child contract's event topic0s)
40-
- `TraceFilter`/`TransferFilter` → `traceFilters[]`
41-
- `includeTransactionReceipts` → `includeReceipts`
110+
## Tuning
42111

43-
The runtime's adaptive `estimate()` (`runtime/historical.ts:1340`, range `[25, 100_000]`) feeds the Portal sync ever-larger intervals automatically, so throughput ramps as the backfill proceeds.
112+
The defaults run well without configuration. These environment variables override them.
44113

45-
## 3. Observability — register Portal metrics
114+
| Variable | Default | Effect |
115+
|---|---|---|
116+
| `PORTAL_START_CONCURRENCY` | `16` | Initial in-flight request limit before AIMD adapts. |
117+
| `PORTAL_MAX_CONCURRENCY` | `48` | Upper bound on concurrent Portal requests. |
118+
| `PORTAL_MIN_CONCURRENCY` | `8` | Lower bound the back-off will not cross. |
119+
| `PORTAL_MAX_ROWS_IN_MEM` | `250000` | Shared buffered-row budget for read-ahead (~1.5–2.5 GB). Raise with a larger `--max-old-space-size`. |
120+
| `PORTAL_READAHEAD` | `6` | Max chunks prefetched ahead of the indexer, per chain. |
121+
| `PORTAL_CHUNK_BLOCKS` | `500000` | Base block range per request; scaled by block density unless fixed. |
122+
| `PORTAL_CHUNK_FIXED` | unset | Set to any value to disable density-based chunk scaling. |
123+
| `PORTAL_TRACE_CHUNK_BLOCKS` | `2000` | Chunk width when a chain has trace or block-interval sources (denser data). |
124+
| `PORTAL_BUFFER_SIZE` | `100` | Chunk workers the Portal fans a single request across. |
125+
| `PORTAL_DISCOVERY_WINDOWS` | `8` | Disjoint windows a factory scan is split into and fetched concurrently. |
126+
| `PORTAL_REALTIME` | unset | Set to `stream` to serve realtime from the Portal `/stream` instead of RPC. |
127+
| `PORTAL_API_KEY` | unset | Sent as `x-api-key` on every Portal request (for a keyed or dedicated Portal). |
128+
| `PORTAL_FINALIZED_HEAD` | unset | Pin the Portal finalized head (testing/ops); overrides the `/finalized-head` probe. |
129+
| `PORTAL_METRICS_FILE` | unset | Write per-chain JSON metrics to `<path>.<chainId>`. |
130+
| `PORTAL_GATE_LOG` | unset | Set to `1` to log the shared controller every 20 s. |
46131

47-
In `internal/metrics.ts`, construct a `PortalMetrics` and expose it on `common`. Append `portalMetrics.prometheus()` to the `/metrics` handler in `server/` so Portal stream/CU/worker-pressure counters sit beside Ponder's RPC metrics. (Native Ponder metrics are RPC-bucket-centric and cannot see these.)
132+
Raising concurrency does not help when indexing is the bottleneck, which for a full-history resync it usually is — the fetch is already ahead. The knobs that matter most in practice are the memory budget (`PORTAL_MAX_ROWS_IN_MEM`) on a constrained box and, for a keyed deployment, `PORTAL_API_KEY`.
48133

49-
## Block/tx/receipt split
134+
## Performance
50135

51-
This fork's `syncBlockRangeData` already fetches logs **and** blocks/txs/receipts/traces in the same range stream (Portal returns them inline), then `syncBlockData` is a light finalizer. To match Ponder's exact two-method contract, move the block/tx/receipt/trace writes into `syncBlockData({ interval, logs })` and have `syncBlockRangeData` request only `fields.log` — both are one Portal stream per interval either way.
136+
Once the Portal serves the backfill, a resync is bound by **indexing**, not fetching. Ponder indexes on a single thread, so the practical ceiling for a large multichain app is that one thread.
52137

53-
## Backfill → frontfill handoff
138+
The reference run — all 15 Portal-supported Euler V2 chains, full history, **28,405,932 events**, byte-verified complete against the Portal — finished in **44m 55s** on an indexer capped at 16 GB and 2 cores (peak 9.2 GB, about one core of real work). Postgres was a separate, throughput-tuned database. Over the same run the Portal's fetch queue drained to idle with the buffer full while a single event loop worked through the tail: the Portal outruns the indexer, and more RAM or cores sit idle.
54139

55-
Historical owns `[start, finalized]`, realtime owns `(finalized, tip]`. Portal's `/finalized-stream` never 409s, so backfill needs no reorg logic. If Portal's finalized head lags Ponder's target finalized block, the thin gap `(portalFinalizedHead, finalized]` falls back to the stock RPC `createHistoricalSync` (keep `rpc` configured for this + realtime + state reads). In practice the gap is small.
140+
The A/B in that run is the useful lesson. A modest, well-tuned configuration beat an over-provisioned one:
56141

57-
## Distribution
142+
| | Over-provisioned | Modest — recommended |
143+
|---|---|---|
144+
| Wall time | 67m 10s | **44m 55s** |
145+
| Indexer peak memory | 19.0 GB | **9.2 GB** |
146+
| Indexer cap | 32 GB heap, density chunks | **16 GB / 2 cores**, fixed 300k chunks |
147+
| Postgres | default | **tuned** |
148+
| Avg throughput | 7,024 ev/s | **10,513 ev/s** |
58149

59-
Publish as `@your-org/ponder-core`. Consumers swap one dependency and add `portal:` per chain in `ponder.config.ts`; **handlers and schema are unchanged.**
150+
Both runs indexed the identical 28.4M events; only the configuration differed. The lever for going faster is not a bigger heap — it is **sharding chains across processes** to lift the single-thread indexing ceiling (for example, running the event-heaviest chain on its own). The full write-up is in [`harness/euler-multichain/REPORT.md`](../harness/euler-multichain/REPORT.md).

0 commit comments

Comments
 (0)