harness(rg3): fix K2 baseline block-skip race + empty-string MOCK_TRACE guard#169
Merged
Conversation
…cursor) The K2 baseline (turnDelayMs:450) aborted at K2-midstream: the mock's /stream delivered 100→101→103, skipping 102, so the product correctly fataled on a corrupt sequence (portal-realtime.ts:1101 "unknown parent"). Root cause (mock artifact, NOT a product bug): handleBlocks iterated block numbers off a MODULE-LEVEL `streamCursor`. K2's 450ms inter-block delay lets the client open an overlapping /stream (a legitimate reconnect on the finalized-head advancing 100→103 mid-stream). Two concurrent handleBlocks turns then read+wrote the one shared cursor across their delays and interleaved: conn1 wrote 102 and set cursor=102, conn2 woke, read cursor=102 and wrote 103 — skipping 102 on conn2, the connection the client consumed. Fast classes (K3) never hit it: the window is too small to overlap. Confirmed via MOCK_TRACE request-interleave. Fix: seed a per-connection `localCursor` from the request's fromBlock and do all per-block arithmetic against it; a real Portal serves each /stream connection independently. The shared `streamCursor` is still mirrored best-effort for the fromBlock-less default of a subsequent request. Also adds MOCK_TRACE-gated (no-op when unset) request/cursor tracing for future harness debugging. No product code under portal/ is touched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
orchestrate.sh forwards MOCK_TRACE via `${MOCK_TRACE:-}`, so a caller that
never sets it (the acceptance capstone) reaches node as an empty string, not
undefined. The `=== undefined` guard let that empty value through to
appendFileSync(''), which throws ENOENT — returned as a 500 on the very first
/finalized-head probe and surfacing as a spurious "cannot establish finality
boundary" fatal on every non-tracing run (i.e. the whole capstone). Guard on
`!TRACE_FILE` so unset and empty both mean off; a real trace path is unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two harness-only fixes in the realtime chaos mock (
harness/chaos/realtime/). Noportal/product code changes.1. K2 baseline block-skip race (
mock-portal.mjs,handleBlocks)K2 is the only scenario with
turnDelayMs: 450. That inter-block gap lets the client open an overlapping/streamconnection (e.g. on a finalized-head advance) while a prior turn is still mid-write. BothhandleBlocksturns read+wrote the shared module-levelstreamCursor, so their block-number arithmetic interleaved and emitted a skipped number (100 → 101 → 103). The client correctly fataled on the corrupt sequence (streamed block 103 has an unknown parent … skipped block) — a mock defect, not a product one: a real Portal serves each/streamfrom its ownfromBlockindependently.Fix: seed a per-connection
localCursorfor the block-number arithmetic; the sharedstreamCursoris only mirrored best-effort for thefromBlock-less default of the next request. AddsMOCK_TRACE-gated request-interleave instrumentation used to pin the mechanism.2. Empty-string
MOCK_TRACEguard (mock-portal.mjs,trace)orchestrate.shforwardsMOCK_TRACEvia${MOCK_TRACE:-}, so a caller that never sets it reaches node as an empty string, notundefined. The=== undefinedguard let that empty value through toappendFileSync(''), which throwsENOENT— returned as a 500 on the very first/finalized-headprobe and surfacing as a spuriouscannot establish finality boundaryfatal for every non-tracing run. Guard on!TRACE_FILEso unset and empty both mean "tracing off"; a real trace path is unaffected.Evidence
102-skip in the post-fix trace (clean sequential105..112)./finalized-headprobe —MOCK_TRACE=""→500 {"error":"ENOENT: … open ''"}pre-fix,200post-fix; validMOCK_TRACEpath still writes the trace file; fully-unset stays200.The K2 midstream-kill semantics are preserved (
killAt.blockstill fires mid-stream; kill spread over the stream interior).