Skip to content

Commit 13654d2

Browse files
dzhelezovclaude
andcommitted
harness(rg3): fold committee nits — K7 vacuous-pass report + K6 null-row cert gap
Review committee (glm + codex + reviewer) on #166 converged on two harness self-cert weaknesses. Both empirically fixed and re-verified: - k7RollbackNonVacuousOk was `!hasClass('K7') || (...)` → evaluated to `true` when K7 is absent, so the `!== undefined` guards always fired and RESULT.md printed a vacuous "K7 rollback non-vacuous: true" for a class that never ran. Now `undefined` when K7 is absent → the guards skip it entirely, so an unrun class no longer self-reports a pass. Verified: a K6-only run emits 0 K7 lines and omits the k7 field from the summary JSON. - k6CutoverOrganicOk checked only the distinct non-null killBlocks set, so a kill row whose block was unreadable/null could slip past `length===1 && [0]===106` when another row recorded 106. Now also requires killBlocksRecordedCount === killCount (every kill recorded a block). Re-verified the cert still bites: reverting k6-cutover.json to the pre-PR fixed-block form flips accepted:false / k6CutoverOrganicOk:false, while the organic form stays accepted:true with recorded==kill count==2. node --test harness/chaos/realtime/mock-portal.test.mjs: 9/9. biome check .: exit 0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent f70ace6 commit 13654d2

1 file changed

Lines changed: 11 additions & 3 deletions

File tree

harness/chaos/realtime/orchestrate.sh

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -546,8 +546,11 @@ for (const klass of classes) {
546546
wrongForkFinalizeRejected: 0,
547547
});
548548
// Distinct empirical kill blocks (from each run's kill.phase.log) — proves K2's varied mid-stream
549-
// coverage rather than one repeated fixed block.
550-
const killBlocks = [...new Set(rows.map((r) => r.killAtBlock).filter((b) => b != null))].sort((a, b) => a - b);
549+
// coverage rather than one repeated fixed block. `killBlocksRecorded` keeps every row's non-null
550+
// block so a cert can demand that EVERY kill recorded a block (a dropped/null row must not be able
551+
// to slip past a distinct-set check where another row happened to record the expected block).
552+
const killBlocksRecorded = rows.map((r) => r.killAtBlock).filter((b) => b != null);
553+
const killBlocks = [...new Set(killBlocksRecorded)].sort((a, b) => a - b);
551554
// Live-cutover proof: kills where the mock had seen >=1 live /stream request before the kill. For K6
552555
// this is the load-bearing evidence that the kill lands at the backfill→live cutover request, not on a startup probe.
553556
const killsWithLiveStream = rows.filter((r) => Number(r.killStats?.stream ?? 0) > 0).length;
@@ -558,6 +561,7 @@ for (const klass of classes) {
558561
variantLabels: [...new Set(rows.map((r) => r.variant).filter(Boolean))].join(', '),
559562
killCount: rows.length,
560563
killBlocks,
564+
killBlocksRecordedCount: killBlocksRecorded.length,
561565
killsWithLiveStream,
562566
wrongForkRunCount: wrongForkRows.length,
563567
wrongForkConsumedRuns,
@@ -605,12 +609,16 @@ const acceptance = {
605609
|| (perClass.K6.killCount > 0 && perClass.K6.killsWithLiveStream === perClass.K6.killCount),
606610
k6CutoverOrganicOk: !hasClass('K6')
607611
|| (perClass.K6.killCount > 0
612+
&& perClass.K6.killBlocksRecordedCount === perClass.K6.killCount
608613
&& perClass.K6.killBlocks.length === 1
609614
&& perClass.K6.killBlocks[0] === 106
610615
&& perClass.K6.killsWithLiveStream === perClass.K6.killCount),
611616
k2KillSpreadOk: !hasClass('K2') || perClass.K2.killBlocks.length >= 3,
617+
// Left `undefined` (not `true`) when K7 is absent so the guards below skip it entirely — a class
618+
// that never ran must not self-report a vacuous "non-vacuous: true" pass in RESULT.md.
612619
k7RollbackNonVacuousOk: !hasClass('K7')
613-
|| (perClass.K7.killCount > 0
620+
? undefined
621+
: (perClass.K7.killCount > 0
614622
&& perClass.K7.cleanResumeCount === perClass.K7.killCount
615623
&& perClass.K7.digestMatchCount === perClass.K7.killCount),
616624
};

0 commit comments

Comments
 (0)