test(chaos): fault-proxy campaign — byte-identical backfill under injected Portal faults#149
Merged
Conversation
… AIMD labels, biome gate Addresses the blind 3-model REQUEST-CHANGES review of the P1 fault-proxy chaos campaign. The core correctness property (digest-identity + interval-tiling, fails closed) is preserved and unweakened; the fixes stop the head-* and AIMD *labels* from overclaiming. 1. biome gate: `biome format --write` the 11 scenario JSONs (semantically identical) and rename the unused `init` param at chaos-aimd-trace.cjs:66 → `_init`, so `biome@2.5.2 check chaos-aimd-trace.cjs scenarios/faultproxy/` exits 0 (error-level). 2. AIMD sparse candor (finding #3): a single gate sample can no longer assert `recovered`. `sparse` (samples < 3) is reported honestly ("sparse — insufficient samples"), never as recovery. AIMD stays SECONDARY — sparse does not fail a scenario whose digest+intervals pass; only a genuine bounds violation (concurrency outside [min,max]) still fails the AIMD gate. 3. head recovery elif false-pass (finding #4): recovery now runs ONLY for head-freeze and ONLY when phase 1 did not complete (done != 1). A failed head_recovery_check propagates instead of retaining a stale pass. `byte-identical-recovery` is reserved for a genuine stall+resume; an unexpected completion labels `byte-identical-completion`. 4. head-freeze label candor (finding #5): honest note — "stalled without corruption; clean restart on the same DB (fault removed) completed byte-identical" — no partial-progress-resume implication. 5. head-regression-100k & head-flap vacuity (findings #1/#2, unanimous): these manipulate the head ABOVE the pinned endBlock so it never intersects the fixed range [20529207,20579207]; a fixed-range historical backfill never rolls back an already-finalized+indexed block, so they prove nothing the clean baseline did. Marked NOT-APPLICABLE (distinct verdict, not a head-tolerance PASS, not a fail) with a note pointing head-reorg/rollback coverage to the stream-mode soak. head-freeze (frozen head BELOW endBlock) remains the one meaningful in-range head scenario. Chosen the ORCHESTRATOR-approved N/A fallback over an in-range redesign because a genuine in-range regression/flap is not achievable non-contrived: regression is anchored to the drifting live head, and an in-range flap only duplicates head-freeze without exercising rollback (a stream-mode property). Verified: biome check EXIT 0 on changed files; bash -n clean; head-freeze re-run on the throwaway PG (54329) PASS byte-identical-recovery, digest == baseline, intervals tile exactly, honest note emitted; N/A routing confirmed by code-trace; proxy.test.mjs 6/6. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…p hardcoded absolute paths) Env-overridable defaults under $ART (the gitignored artifacts dir) so the harness is portable and carries no machine-specific absolute paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a fault-injecting chaos campaign under
harness/chaos/that stands up an HTTP proxy in front of the SQD Portal and runs a fixed-range historical backfill (euler-app, range[20529207,20579207]) under one injected fault class per scenario, then verifies the resulting store is byte-identical (logical digest) to a clean baseline and that the indexed block intervals tile the window exactly.Scenarios
8 body-fault scenarios — each fires ≥1 real fault and completes byte-identical, no missed faults:
429-burst,5xx-storm,retry-after,tcp-reset-mid-ndjson,90s-stall,truncated-gzip,malformed-ndjson,spurious-204.Head scenarios:
head-freeze(frozen head below endBlock) — the Portal stalls (finalized head behind the requested range); the app makes little/no progress, then a clean restart on the same DB with the fault removed completes byte-identical. The one meaningful in-range head scenario.head-regression-100k,head-flap— marked not-applicable: for a fixed-range historical backfill the manipulated head stays ≥ endBlock and never intersects the pinned range, and such a backfill never rolls back an already-finalized+indexed block. Head-reorg/rollback tolerance is a stream-mode property (covered by the stream soak), not this harness. These are reported as N/A — never counted as a head-tolerance PASS.Verification is fail-closed
A scenario passes only if the verifier independently recomputes both digests (
pg-digest) and confirms full interval tiling over the window; it fails closed on any empty digest or any gap. A byte-identical store still fails a run if a secondary gate (fault-fired count, or an AIMD concurrency bounds violation) fails — so the gate is not vacuous.AIMD tracing
The fork's AIMD portal-gate concurrency is traced. Runs with fewer than 3 samples are labelled honestly as
sparse — insufficient samplesand never claimrecovered.Test/harness-only change (no product-code surface); evidence artifacts are gitignored.