Skip to content

Commit 15421d9

Browse files
dzhelezovclaude
andcommitted
docs: VALIDATION.md — scope chaos evidence to what it proves (atomicity/idempotence, not attributable partial resume; #50)
The §4 chaos acceptance claimed "41/41 clean resumes — resuming from its persisted ponder_sync state", implying the campaign demonstrated attributable resume-from-partial-persisted-state. Under the granularity property in #50, the fork's first durable commit only lands after full-range discovery + the entire first data chunk (default 500k), so a 50k range inside one chunk goes 0%→100% in one transaction: at almost every kill the durable store was empty-or-complete and restarts re-paid from zero. The campaign therefore proves SIGKILL-atomicity, restart idempotence, and byte-identical completion — but not attributable partial-state resume. - §4: rewrite the "clean resumes" bullet and add a "what this proves — and what it does not" paragraph, citing #50 and the exact numbers. - §5.4 (new): findings-log row for the first-durable-commit granularity property (discovered by the chaos campaign — a 60-kill run ended byte-empty; correctness unaffected, availability/livelock regression vs upstream; filed as #50), plus a note that a re-parameterized campaign (small chunks, per-kill coverage snapshots, witnessed-partial-resume acceptance) is queued to close the gap. - §6: scope the crash/resume summary bullet to match. Docs-only; no code touched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 458dc8c commit 15421d9

1 file changed

Lines changed: 41 additions & 5 deletions

File tree

VALIDATION.md

Lines changed: 41 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -190,14 +190,35 @@ SQD_PONDER_TARBALL=<tarball> RPC_URL_OVERRIDE=<free-eth-rpc> \
190190
The chaos kill-loop was accepted against the campaign's acceptance criteria. Aggregate result:
191191

192192
- **203 `SIGKILL`s** delivered across **80 attempted / 41 completed** backfill runs.
193-
- **41 / 41 clean resumes** — every completed backfill finished after being killed and resuming from
194-
its persisted `ponder_sync` state.
193+
- **41 / 41 completed backfills survived kill-and-resume** — every completed backfill was `SIGKILL`ed
194+
at least twice mid-flight and still reached a correct, complete final store on resume.
195195
- Final stores **byte-identical** to an unkilled baseline across **logs, transactions, receipts,
196196
traces, blocks**.
197197
- **Every sync interval fragment tiled the range exactly** (including the factory-discovery
198198
fragments) — no gap, no overlap.
199199
- Zero `InvariantViolation` under `PORTAL_CHECKS=strict`. (Both `kill-loop.sh` and `verify-resume.sh` export `PORTAL_CHECKS=strict`; an `InvariantViolation` is fatal to the run, so 41/41 clean completions entail zero violations.)
200200

201+
**What this proves — and what it does not.** The evidence above establishes three things precisely:
202+
**(1) `SIGKILL`-atomicity** — a kill never leaves torn state; rows and their sync-interval fragments
203+
commit together (all-or-nothing per commit), so the store never overstates coverage. **(2) Restart
204+
idempotence** — a killed process, restarted, converges to the same correct store regardless of when
205+
it was killed. **(3) Byte-identical completion** — the resumed store equals an uninterrupted baseline
206+
across all five row families with intervals tiling exactly.
207+
208+
It does **not**, as parameterized, prove **attributable resume-from-partial-persisted-state**. Under
209+
[#50](../../issues/50), the fork's historical path makes its **first durable commit only after
210+
full-range discovery plus the entire first data chunk** (default `PORTAL_CHUNK_BLOCKS` 500k); for a
211+
range inside one chunk the durable store goes **0% → 100% in a single transaction, seconds before
212+
completion**. The chaos range `[20529207, 20579207]` (50k blocks) sits inside one chunk, so at almost
213+
every kill instant the durable store was **either empty or already complete** — a restart re-paid the
214+
discovery scan and re-streamed the chunk from zero rather than continuing from a persisted partial
215+
watermark. With ~one atomic commit per backfill, the great majority of the 203 kills exercised
216+
**restart-from-zero**; any kills that happened to land in the sub-second commit window and produced a
217+
genuine mid-range resume are **statistically expected but were not recorded or attributed** by this
218+
run (`kill-loop.sh` counts kills regardless of the durable coverage present at kill time — see
219+
[#50](../../issues/50)). So "resume from a persisted partial `ponder_sync` state" is **plausible but
220+
unproven here**; what is proven is atomicity, idempotence, and byte-identical completion.
221+
201222
Conditions: Poisson kill schedule (mean 30 s), `MIN_KILLS=2` enforced per completed backfill (a run
202223
that finished without being killed proves nothing about resume and is rejected), chain 1 (ethereum)
203224
range `[20529207, 20579207]`, build `0.16.6-sqd.2`, PGlite throwaway stores, **public** Portal +
@@ -277,6 +298,19 @@ spot audit** (Layer F): in the confirmed cases (#36, #27) leg B's rows matched a
277298
byte-for-byte, establishing leg A as the lossy side. The status JSON carries a bounded sample of
278299
tolerated block numbers per table specifically to keep that audit reproducible.
279300

301+
### 5.4 Chaos-discovered findings (Layer C)
302+
303+
| Issue | State | Finding | Attribution / layer |
304+
|-------|-------|---------|---------------------|
305+
| [#50](../../issues/50) | OPEN | **First-durable-commit granularity.** The fork's historical path makes its first durable sync-store commit only after **full-range factory discovery + the entire first data chunk** stream (default `PORTAL_CHUNK_BLOCKS` 500k); for a range inside one chunk the durable store goes **0% → 100% in one transaction, seconds before completion**. A restart loop shorter than that window makes **zero forward progress** and re-pays discovery + chunk re-stream each cycle (upstream, which commits proportionally to a 25-block first interval, creeps forward instead). An availability/progress regression vs upstream, with a zero-progress-livelock shape under sub-window crash loops. **Correctness is unaffected**: coverage never overstates and rows+intervals still commit atomically (that invariant held under all 203 chaos kills). | Discovered by the chaos campaign (Layer C): a 60-kill Poisson run (mean 5 s) ended with a **provably byte-empty store** — every restart began from zero — root-caused to the first-commit granularity and filed as [#50](../../issues/50). |
306+
307+
This finding also bounds what §4 can claim: because the 50k chaos range fits inside one 500k chunk,
308+
the campaign's kills overwhelmingly hit an empty-or-complete durable store, so *attributable*
309+
resume-from-partial state was not witnessed (see §4). A **re-parameterized chaos campaign** — small
310+
fixed chunks (`PORTAL_CHUNK_BLOCKS` on the order of 2k) that force staircase durable commits,
311+
per-kill coverage snapshots, and an acceptance criterion requiring kills observed with
312+
`0 < coverage < 100%` at restart — is queued to close that evidence gap directly.
313+
280314
---
281315

282316
## 6. Current status — what a reader can rely on today
@@ -286,9 +320,11 @@ tolerated block numbers per table specifically to keep that audit reproducible.
286320
- The Portal layer's invariants (INV-1 … INV-16) hold under property-based tests **on both supported
287321
upstream Ponder versions** (`0.16.6`, `0.15.17`), and every fix is backed by a mutation-verified
288322
regression test.
289-
- **Crash/resume is byte-safe** at the accepted chaos scale: 203 kills, 41/41 clean resumes,
290-
byte-identical to an unkilled baseline across all five row families, intervals tiling exactly, zero
291-
invariant violations (§4).
323+
- **Crash/resume is byte-safe** at the accepted chaos scale: 203 kills across 41/41 completed
324+
backfills, `SIGKILL`-atomic and restart-idempotent, byte-identical to an unkilled baseline across
325+
all five row families, intervals tiling exactly, zero invariant violations (§4). *Attributable*
326+
resume-from-partial-persisted-state is not yet proven at this parameterization — see the
327+
[#50](../../issues/50) granularity property and §4.
292328
- The fork-vs-stock **byte-diff plumbing** is proven end-to-end by the SMOKE cell (byte-identical
293329
across all five row families on public endpoints, §3.1).
294330
- The A/B dual-implementation soak is **actively cross-validating** the Portal path against the RPC

0 commit comments

Comments
 (0)