|
| 1 | +# Euler V2 multichain — full-history e2e on `@subsquid/ponder` |
| 2 | + |
| 3 | +**One Ponder app indexing every Portal-supported Euler V2 chain (15), full history `[deploy → finalized head]`, the complete 24-event EVault superset — streamed from the SQD Portal into Postgres.** This is the production shape of an Euler indexer and a stress test of the Portal-backed fork. |
| 4 | + |
| 5 | +Captured on a single OVH box (Ubuntu 24.04). The run is deliberately bounded to a fixed head so it's a clean, reproducible benchmark; live/realtime is a separate mode (see *Roadmap*). |
| 6 | + |
| 7 | +--- |
| 8 | + |
| 9 | +## TL;DR |
| 10 | + |
| 11 | +- **28,405,932 events across 15 chains, 2,484 vaults**, full history, in **44m 55s**. |
| 12 | +- **Byte-complete and correct** — verified independently against the Portal ground truth: **60/60 sampled windows across all 15 chains match exactly**, and two independent runs produced the **identical** 28,405,932 total. |
| 13 | +- **Modest footprint** — the indexer ran capped at **16 GB / 2 cores** (peak 9.2 GB, ~1 core of real work); Postgres is a separate, throughput-tuned production DB. No 96-core/125 GB machine required — the box was just what we had; its cores and RAM sat idle. |
| 14 | +- **The Portal is not the bottleneck.** It feeds data faster than a single Ponder event-loop can consume it; wall-time is set by Ponder's single-threaded indexing, not by fetch, RAM, or cores. |
| 15 | + |
| 16 | +--- |
| 17 | + |
| 18 | +## What it indexes |
| 19 | + |
| 20 | +- **15 chains:** ethereum, base, arbitrum, avalanche, sonic, plasma, monad, binance, linea, berachain, unichain, tac, bob, polygon, hyperliquid. |
| 21 | +- **The `eVaultFactory` (`GenericFactory` → `ProxyCreated` → child EVaults)** per chain. Per-chain **start blocks come from Euler's own `euler-subgraph` config**, and every `eVaultFactory` address is **cross-verified against `euler-interfaces` `CoreAddresses.json`** (all 15 ✓). |
| 22 | +- **The full 24-event `IEVault` interface** (`Deposit/Withdraw/Borrow/Repay/Liquidate/Transfer/Approval/DebtSocialized/PullDebt/InterestAccrued/VaultStatus/ConvertFees/BalanceForwarderStatus/EVaultCreated` + all `GovSet*`) — a superset of the public euler-subgraph's 3-event set, since Euler runs both Ponder and subgraphs and prod reads more than any single consumer. |
| 23 | + |
| 24 | +--- |
| 25 | + |
| 26 | +## Results — per chain |
| 27 | + |
| 28 | +| chain | events | vaults (indexed) | vs Euler subgraph | windows | |
| 29 | +|---|--:|--:|--:|:--| |
| 30 | +| berachain | 20,185,992 | 59 | 59 / 59 ✓ | 4/4 ✓ | |
| 31 | +| unichain | 2,134,319 | 50 | 50 / 51 | 4/4 ✓ | |
| 32 | +| ethereum | 1,351,179 | 850 | 850 / 897 | 4/4 ✓ | |
| 33 | +| base | 1,282,875 | 325 | 325 / 346 | 4/4 ✓ | |
| 34 | +| arbitrum | 729,050 | 137 | 137 / 151 | 4/4 ✓ | |
| 35 | +| sonic | 663,464 | 174 | 174 / 176 | 4/4 ✓ | |
| 36 | +| avalanche | 555,877 | 251 | 251 / 265 | 4/4 ✓ | |
| 37 | +| plasma | 481,982 | 156 | 156 / 175 | 4/4 ✓ | |
| 38 | +| monad | 285,415 | 122 | 122 / 131 | 4/4 ✓ | |
| 39 | +| binance | 260,605 | 126 | 126 / 129 | 4/4 ✓ | |
| 40 | +| linea | 257,099 | 88 | 88 / 100 | 4/4 ✓ | |
| 41 | +| bob | 100,634 | 27 | 27 / 27 ✓ | 4/4 ✓ | |
| 42 | +| hyperliquid | 67,314 | 58 | **58 / 0** | 4/4 ✓ | |
| 43 | +| tac | 49,324 | 36 | 36 / 36 ✓ | 4/4 ✓ | |
| 44 | +| polygon | 803 | 25 | 25 / 25 ✓ | 4/4 ✓ | |
| 45 | +| **total** | **28,405,932** | **2,484** | | **60/60 ✓** | |
| 46 | + |
| 47 | +`hyperliquid 58 / 0`: Euler's public subgraph reports **zero** vaults there — the fork indexes 58. High-block / high-TPS chains are exactly where subgraphs are slow-to-unviable, and where the Portal backfill wins. |
| 48 | + |
| 49 | +--- |
| 50 | + |
| 51 | +## Correctness (the part that matters) |
| 52 | + |
| 53 | +Data correctness was treated as the gate: **the report does not ship unless the data is provably complete and correct.** Three independent lines of evidence, all against the **SQD Portal ground truth** (Portal-derived logs are byte-identical to JSON-RPC `eth_getLogs` — see the differential test in the repo): |
| 54 | + |
| 55 | +1. **Windowed completeness — 60/60 exact.** For each chain we sampled four 100k-block windows (early / 40% / 75% / head) and compared the indexed event count to the Portal's, both filtered to the discovered children + the 24 event topics. **Every window matched exactly** — e.g. ethereum `[24.21M–24.31M]` 28,825 = 28,825; berachain `[17.48M–17.58M]` **282,320 = 282,320**. |
| 56 | +2. **Cross-run reproducibility — identical.** Two independent runs with *different* chunking, heap, and Postgres configs produced the **identical** 28,405,932-event total. A data-loss bug would diverge; it doesn't. |
| 57 | +3. **Vault discovery cross-checked vs Euler's own live subgraph.** Discovered vault counts match Euler exactly where the chain is fully active (polygon 25/25, tac 36/36, bob 27/27, berachain 59/59…). The gaps on larger chains are **not missing data**: for ethereum, the Portal shows **872** vaults created by our head vs Euler's 897 — **25 were created after our fixed head** (Euler's subgraph runs ahead), and the remaining **22 are discovered-but-eventless** (created, never used), confirmed because our indexed total equals the Portal total for the events that do exist. |
| 58 | + |
| 59 | +**On the 24-event superset:** it is complete for the `IEVault` interface. The only vault log *outside* it is `Genesis()` — a no-parameter genesis marker emitted once per vault — which is already captured as vault creation via `ProxyCreated` discovery, so nothing is lost. |
| 60 | + |
| 61 | +--- |
| 62 | + |
| 63 | +## Performance, footprint, and the A/B |
| 64 | + |
| 65 | +We ran two configurations to find the right operating point. **Both indexed the identical 28.4M events; only the config differed.** |
| 66 | + |
| 67 | +| | (a) over-provisioned | (b) modest — recommended | |
| 68 | +|---|---|---| |
| 69 | +| wall time | 67m 10s | **44m 55s** | |
| 70 | +| indexer peak mem | 19.0 GB | **9.2 GB** | |
| 71 | +| indexer cap | 32 GB heap, density chunks | **16 GB / 2 cores**, fixed 300k chunks | |
| 72 | +| Postgres | default | **tuned** (`synchronous_commit=off`, 16 GB shared_buffers) | |
| 73 | +| Portal concurrency | thrashy (48↔8 oscillation) | **steady** (mostly 48) | |
| 74 | +| avg throughput | 7,024 ev/s | **10,513 ev/s** | |
| 75 | + |
| 76 | +The modest config is **faster, uses half the memory, and runs steadier**. The lesson is counter-intuitive but important: **don't over-provision the indexer — right-size it and tune the database.** (Honest caveat: (b) improved both the indexer footprint *and* the PG tuning, so its win reflects the combined informed config, not the smaller heap alone.) |
| 77 | + |
| 78 | +**Footprint, stated plainly:** the *indexer* is ~1 CPU core (Ponder is single-threaded) and ~9 GB RAM. Postgres is a separate, uncapped, tuned production database — as any real indexer needs. It is **not** 16 GB total, and it does **not** need a big machine. |
| 79 | + |
| 80 | +--- |
| 81 | + |
| 82 | +## Where the ceiling actually is (measured) |
| 83 | + |
| 84 | +Once the Portal makes the backfill fast, the bottleneck moves entirely to the indexer: |
| 85 | + |
| 86 | +- **Ponder is single-threaded.** Under load the node process pinned **one core at ~92%** while ~4 libuv/GC helper threads idled; system load was ~3 of 96 cores. **93 cores sat unused.** |
| 87 | +- **Postgres was faster than the indexer** — its backends sat idle in `ClientRead`, waiting for Ponder to send data. |
| 88 | +- **The Portal outruns both** — during the tail its fetch queue drained to idle with the buffer full; every remaining second was single-threaded decode+index of berachain's 20M events. |
| 89 | + |
| 90 | +So more RAM/cores don't buy wall-time here — they're already idle. The lever for going faster is **sharding chains across processes** (e.g. berachain on its own), not bigger hardware. |
| 91 | + |
| 92 | +--- |
| 93 | + |
| 94 | +## What's managed vs what you tune |
| 95 | + |
| 96 | +- **Portal capacity is managed by Subsquid Labs** as part of the enterprise subscription — a flat plan, scalable transparently. Clients never touch compute-units or infra; throughput is a dial we turn, not a client concern. |
| 97 | +- **The indexer** (this fork of Ponder) is what you run — modest footprint, your handlers and schema unchanged, one `portal:` line per chain. |
| 98 | + |
| 99 | +The numbers here are a **floor** at the current provisioning and a single-process indexer — not a ceiling. |
| 100 | + |
| 101 | +--- |
| 102 | + |
| 103 | +## Roadmap |
| 104 | + |
| 105 | +- **Realtime / unbounded:** this benchmark is bounded to a fixed head for clean measurement. The fork already supports unbounded backfill → finalized head → **RPC realtime** for the tip (with automatic RPC fallback for the finality gap and Portal errors). Making the Portal's fork-aware hot-blocks `/stream` the *primary* realtime source (RPC as fallback-only) is the next step. |
| 106 | +- **Multi-process sharding** to lift the single-threaded indexing ceiling for event-heavy chains. |
| 107 | + |
| 108 | +--- |
| 109 | + |
| 110 | +## Reproduce |
| 111 | + |
| 112 | +```bash |
| 113 | +docker compose up -d postgres # or a tuned standalone Postgres |
| 114 | +cp .env.example .env # PORTAL_API_KEY (+ SQD_RPC_KEY for the 10 SQD-served chains) |
| 115 | +./run.sh |
| 116 | +``` |
| 117 | + |
| 118 | +See [`README.md`](./README.md) for the config and metrics. Correctness was verified with per-window and aggregate cross-checks against the Portal; vault discovery against Euler's `euler-subgraph` (Goldsky) endpoints. |
0 commit comments