You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: VALIDATION.md — scope chaos evidence to what it proves (atomicity/idempotence, not attributable partial resume; #50)
The §4 chaos acceptance claimed "41/41 clean resumes — resuming from its
persisted ponder_sync state", implying the campaign demonstrated attributable
resume-from-partial-persisted-state. Under the granularity property in #50, the
fork's first durable commit only lands after full-range discovery + the entire
first data chunk (default 500k), so a 50k range inside one chunk goes 0%→100% in
one transaction: at almost every kill the durable store was empty-or-complete and
restarts re-paid from zero. The campaign therefore proves SIGKILL-atomicity,
restart idempotence, and byte-identical completion — but not attributable
partial-state resume.
- §4: rewrite the "clean resumes" bullet and add a "what this proves — and what
it does not" paragraph, citing #50 and the exact numbers.
- §5.4 (new): findings-log row for the first-durable-commit granularity property
(discovered by the chaos campaign — a 60-kill run ended byte-empty; correctness
unaffected, availability/livelock regression vs upstream; filed as #50), plus a
note that a re-parameterized campaign (small chunks, per-kill coverage
snapshots, witnessed-partial-resume acceptance) is queued to close the gap.
- §6: scope the crash/resume summary bullet to match.
Docs-only; no code touched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
-**41 / 41 clean resumes** — every completed backfill finished after being killed and resuming from
194
-
its persisted `ponder_sync` state.
193
+
-**41 / 41 completed backfills survived kill-and-resume** — every completed backfill was `SIGKILL`ed
194
+
at least twice mid-flight and still reached a correct, complete final store on resume.
195
195
- Final stores **byte-identical** to an unkilled baseline across **logs, transactions, receipts,
196
196
traces, blocks**.
197
197
-**Every sync interval fragment tiled the range exactly** (including the factory-discovery
198
198
fragments) — no gap, no overlap.
199
199
- Zero `InvariantViolation` under `PORTAL_CHECKS=strict`. (Both `kill-loop.sh` and `verify-resume.sh` export `PORTAL_CHECKS=strict`; an `InvariantViolation` is fatal to the run, so 41/41 clean completions entail zero violations.)
200
200
201
+
**What this proves — and what it does not.** The evidence above establishes three things precisely:
202
+
**(1) `SIGKILL`-atomicity** — a kill never leaves torn state; rows and their sync-interval fragments
203
+
commit together (all-or-nothing per commit), so the store never overstates coverage. **(2) Restart
204
+
idempotence** — a killed process, restarted, converges to the same correct store regardless of when
205
+
it was killed. **(3) Byte-identical completion** — the resumed store equals an uninterrupted baseline
206
+
across all five row families with intervals tiling exactly.
207
+
208
+
It does **not**, as parameterized, prove **attributable resume-from-partial-persisted-state**. Under
209
+
[#50](../../issues/50), the fork's historical path makes its **first durable commit only after
210
+
full-range discovery plus the entire first data chunk** (default `PORTAL_CHUNK_BLOCKS` 500k); for a
211
+
range inside one chunk the durable store goes **0% → 100% in a single transaction, seconds before
212
+
completion**. The chaos range `[20529207, 20579207]` (50k blocks) sits inside one chunk, so at almost
213
+
every kill instant the durable store was **either empty or already complete** — a restart re-paid the
214
+
discovery scan and re-streamed the chunk from zero rather than continuing from a persisted partial
215
+
watermark. With ~one atomic commit per backfill, the great majority of the 203 kills exercised
216
+
**restart-from-zero**; any kills that happened to land in the sub-second commit window and produced a
217
+
genuine mid-range resume are **statistically expected but were not recorded or attributed** by this
218
+
run (`kill-loop.sh` counts kills regardless of the durable coverage present at kill time — see
219
+
[#50](../../issues/50)). So "resume from a persisted partial `ponder_sync` state" is **plausible but
220
+
unproven here**; what is proven is atomicity, idempotence, and byte-identical completion.
221
+
201
222
Conditions: Poisson kill schedule (mean 30 s), `MIN_KILLS=2` enforced per completed backfill (a run
202
223
that finished without being killed proves nothing about resume and is rejected), chain 1 (ethereum)
@@ -277,6 +298,19 @@ spot audit** (Layer F): in the confirmed cases (#36, #27) leg B's rows matched a
277
298
byte-for-byte, establishing leg A as the lossy side. The status JSON carries a bounded sample of
278
299
tolerated block numbers per table specifically to keep that audit reproducible.
279
300
301
+
### 5.4 Chaos-discovered findings (Layer C)
302
+
303
+
| Issue | State | Finding | Attribution / layer |
304
+
|-------|-------|---------|---------------------|
305
+
| [#50](../../issues/50) | OPEN | **First-durable-commit granularity.** The fork's historical path makes its first durable sync-store commit only after **full-range factory discovery + the entire first data chunk** stream (default `PORTAL_CHUNK_BLOCKS` 500k); for a range inside one chunk the durable store goes **0% → 100% in one transaction, seconds before completion**. A restart loop shorter than that window makes **zero forward progress** and re-pays discovery + chunk re-stream each cycle (upstream, which commits proportionally to a 25-block first interval, creeps forward instead). An availability/progress regression vs upstream, with a zero-progress-livelock shape under sub-window crash loops. **Correctness is unaffected**: coverage never overstates and rows+intervals still commit atomically (that invariant held under all 203 chaos kills). | Discovered by the chaos campaign (Layer C): a 60-kill Poisson run (mean 5 s) ended with a **provably byte-empty store** — every restart began from zero — root-caused to the first-commit granularity and filed as [#50](../../issues/50). |
306
+
307
+
This finding also bounds what §4 can claim: because the 50k chaos range fits inside one 500k chunk,
308
+
the campaign's kills overwhelmingly hit an empty-or-complete durable store, so *attributable*
309
+
resume-from-partial state was not witnessed (see §4). A **re-parameterized chaos campaign** — small
310
+
fixed chunks (`PORTAL_CHUNK_BLOCKS` on the order of 2k) that force staircase durable commits,
311
+
per-kill coverage snapshots, and an acceptance criterion requiring kills observed with
312
+
`0 < coverage < 100%` at restart — is queued to close that evidence gap directly.
313
+
280
314
---
281
315
282
316
## 6. Current status — what a reader can rely on today
@@ -286,9 +320,11 @@ tolerated block numbers per table specifically to keep that audit reproducible.
286
320
- The Portal layer's invariants (INV-1 … INV-16) hold under property-based tests **on both supported
287
321
upstream Ponder versions** (`0.16.6`, `0.15.17`), and every fix is backed by a mutation-verified
288
322
regression test.
289
-
-**Crash/resume is byte-safe** at the accepted chaos scale: 203 kills, 41/41 clean resumes,
290
-
byte-identical to an unkilled baseline across all five row families, intervals tiling exactly, zero
291
-
invariant violations (§4).
323
+
-**Crash/resume is byte-safe** at the accepted chaos scale: 203 kills across 41/41 completed
324
+
backfills, `SIGKILL`-atomic and restart-idempotent, byte-identical to an unkilled baseline across
325
+
all five row families, intervals tiling exactly, zero invariant violations (§4). *Attributable*
326
+
resume-from-partial-persisted-state is not yet proven at this parameterization — see the
327
+
[#50](../../issues/50) granularity property and §4.
292
328
- The fork-vs-stock **byte-diff plumbing** is proven end-to-end by the SMOKE cell (byte-identical
293
329
across all five row families on public endpoints, §3.1).
294
330
- The A/B dual-implementation soak is **actively cross-validating** the Portal path against the RPC
0 commit comments