harness(rg3): realtime chaos kill/resume harness (Phase A)#158
Merged
Conversation
…inalized rows scanned)
Adversarial review (codex) BLOCKed on coverage overstatement in the realtime
chaos harness. Three findings, all now honest and mutation/subset-proven:
K6 (HIGH): the "cutover" kill landed on the FIRST /finalized-head probe, before
any live /stream block — the scenario-root `killAt` was consumed by
finalizedHead() (gatePhaseForBlock(scenario, head.number)), not by a served
block. Surviving artifacts showed stream:0, finalizedHead:1. Move `killAt` into
the `blocks` step at block 107 (matching K1/K4/K5), so the gate fires in
handleBlocks only AFTER the live /stream has served block 106. Subset proof (4
iters): kill.mock-stats.json now shows stream:1 (was 0), finalizedHeadGates:0
(was 1), request log `… -> stream@106 -> finalizedHead`; every resume
storeMatch+appMatch, 0 dup finalized rows.
K2 (LOW): all runs killed at one fixed block (107) yet RESULT claimed "arbitrary
N/M". Add `MOCK_KILLAT_BLOCK` (retargetKillAtBlock retargets the phase-carrying
killAt) + orchestrate `kill_block_for` to vary K2's kill block across the
interior 103..110 per iteration. Subset proof (4 iters): distinct kill blocks
103,104,105,106, each stream:2, all clean + digest-matched. The block is
recovered empirically from kill.phase.log (nextBlock) into the run record, not
trusted from env; summary/RESULT now report the distinct kill blocks and the
"kills after >=1 live /stream block" count.
K5 (taste call → relabel candidly): the wrongfork variant's injected wrong
finalized head is never consumed by the client (finalize poll ~4s > the
post-injection window ~1.8s), so wrongForkFinalizes is mock-side only. Forcing
consumption would require mutating shared portal/ (a finalizePollMs env knob —
forbidden overlay-wise and gate-triggering) and would flip the variant to
expected-fatal, reworking the clean-resume/digest acceptance accounting. The
client-side wrong-fork-finalize FATAL guard is already proven deterministically
by portal-realtime.test.ts (finalizePollMs:0, hash-mismatch → throw /losing
fork/). So the candor string now labels wrongForkFinalizes as a mock-side
handler-reachability count and stops presenting it as clean-resume wrong-fork
evidence, pointing to the unit test for the real guard.
Also: remove the dead `cursorTypes` Set in cursorMatchesStep (routing uses
isCursorStep()/wantFrom/wantParent); add a mutation-verified unit test for
retargetKillAtBlock (empty-string override is a no-op — Number("")===0 guarded);
add a CHAOS_CLASSES subset filter to orchestrate for targeted proofs (default
full K1..K6). No portal/ or shared euler-app changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The k1-append.{store,app}.digest baselines were tracked despite the intent to
ignore them (the .gitignore glob `k[1-6].*.digest` never matched `k1-append.*`
— the `-append` infix sits between `k1` and the first dot). They are inert
(run_baseline regenerates them every run), so untrack them and fix the glob to
`k[1-6]*.digest` so future baselines stay ignored.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…le-quote) Post-re-cert hardening of the RG3 realtime chaos harness. All false-FAIL direction (cannot fabricate a PASS); the 238-kill re-cert was already accepted with these un-folded, so its evidence stands. - invariant() (orchestrate.sh): drop the dead `wrong-fork|wrong fork` alts — 0 matches across ALL 238 re-cert app logs; they match no real client fatal — and add `losing fork`, the actual thrown string for the wrong-fork finalize guard (portal-realtime.ts: "on a losing fork at/below finality. Cannot finalize safely; restart..."). Net: removes a latent benign-substring false-FAIL AND closes a tripwire gap (the real losing-fork fatal was previously unmatched by any alternative). - completed() (kill-controller.sh): align to orchestrate.sh's fuller 'Completed indexing across all chains' (was the broader 'Completed indexing across'). - verify-resume.sh: double-quote the $table SQL identifier in both finalized dup-scan queries (defensive against non-lowercase ponder table names). Gates: bash -n clean (all 3 scripts); mock-portal.test.mjs 9/9. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ity + complete the fatal tripwire 3-seat committee (opus + glm + codex, blind) returned UNANIMOUS MERGE-WITH-NITS on the re-certified harness. This folds their convergent, false-FAIL-direction asks (no re-cert needed — verified below): - acceptance gate (codex MEDIUM): the K6 cutover-non-vacuity and K2 kill-spread evidence was COMPUTED and printed but never gated in `accepted`, so a regression to the original vacuous-K6 bug (killStats.stream===0) or a fixed-block K2 could silently re-produce `accepted:true` on a byte-identical resume. Added `k6CutoverNonVacuousOk` (every K6 kill after >=1 live /stream block) and `k2KillSpreadOk` (>=3 distinct interior kill blocks) to the acceptance object AND the `accepted` conjunction — mirroring the existing `dupCheckLoadBearingOk` self-cert precedent. - invariant() tripwire (opus + codex + glm): the fast-path fatal grep matched only ~2 of the client's 7 realtime fatal throws and carried a dead `gap fatal` alt. Replaced with `restart to re-sync|deterministic, not retried|Cannot finalize safely|Cannot reconcile safely` (+ InvariantViolation) — verified against portal-realtime.ts to catch ALL 7 fatal strings. Not a false-PASS gap even before (an unmatched fatal still fails the run via process-death), but now the fast-path is complete. Synced the same pattern into kill-controller.sh (was InvariantViolation-only). - removed dead `completed_live()` (kill-controller.sh, 0 call sites; opus nit). - K5 candor wording (glm): `wrongForkFinalizes` "counts mock-side splices" -> "counts mock-side handler entries (reachability)" — the counter increments pre-gate, so a killed run may not splice. - generated RESULT title "RG3 Phase B" -> "Phase A" (codex nit; this branch is Phase A). Verification (independent, over the SHIPPED 238-record run-records.ndjson): - re-ran the edited write_summary -> accepted:true with both new gates present (evidence stands). - MUTATION-VERIFIED both new gates are load-bearing: one K6 killStats.stream->0 => k6CutoverNonVacuousOk false => accepted false; K2 collapsed to a single block => k2KillSpreadOk false => accepted false. - invariant() new pattern caught all 7 real portal-realtime.ts fatal strings (old caught 2). - Gates: bash -n clean x3; mock-portal.test.mjs 9/9. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RG3 — realtime chaos kill/resume harness (Phase A)
Adds a deterministic chaos-testing harness for the Portal realtime (stream-mode) indexer under
harness/chaos/realtime/. It repeatedly SIGKILLs a running ponder-on-Portal indexer at controlledpoints during realtime ingestion and proves the resume is byte-identical with zero duplicate
FINALIZED rows — the core durability guarantee for a crash/restart of a live indexer.
Kill classes
A mock Portal (
mock-portal.mjs) drives scripted scenarios; each class kills mid-phase and verifies resume:/streamblock served before the kill), so resume re-drives the cutoverVerification per kill
verify-resume.sh+app-digest.mjscompute a store digest and an app-row digest and require bothto match the pre-kill baseline, plus a scan of the finalized region for duplicate rows. Non-vacuity
guards (empty-digest tripwire, non-zero row counts, finalized-checkpoint availability) gate the
okstatus so an empty/degenerate run cannot pass.
Self-certifying acceptance
orchestrate.shaggregates a run intoorchestrate.summary.jsonwith anacceptanceobject. Beyondthe count/clean-resume/digest/dup gates, the summary self-certifies the two non-vacuity properties
the harness exists to prove, so a regression can't silently pass:
k6CutoverNonVacuousOk— every K6 kill landed after ≥1 live/streamblock (killStats.stream > 0);k2KillSpreadOk— K2 kills spanned ≥3 distinct interior blocks;dupCheckLoadBearingOk— the dup scan actually scanned finalized rows in the load-bearing classes.An
invariant()fast-path tripwire greps app logs for the client's realtime fatal strings(
restart to re-sync,Cannot finalize safely,Cannot reconcile safely,deterministic, not retried,InvariantViolation) — covering all of the realtime fatal throws; an unmatched fatalwould additionally fail the run via process death.
Certification run
A full run at 34 kills/class (238 kills across K1–K4, K5/409, K5/wrongfork, K6) produced:
0 failed statuses · 238/238 clean resumes · 238/238 digest matches · 0 duplicate FINALIZED rows
(204 finalized rows scanned), with
accepted: trueincluding all non-vacuity gates. The K6 cutoverclass showed a live
/streamblock before the kill on every iteration, and K2 kills spanned 8distinct interior blocks.
Notes / candor
(the injected wrong finalized head is not consumed within the scenario window). The client-side
wrong-fork-finalize FATAL guard (hash mismatch at finality) is proven separately by
portal/portal-realtime.test.ts.Portal / halted-chain simulation (future phase).
/finalized-headprobe-retry loop (the mock succeeds first try); thatfidelity rests on
portal/portal-realtime-wire.test.ts.Phase A. Follow-up: a
VALIDATION.mdsection + gate row will document this evidence.