fix(scheduler): break non-converging SSP catch-up loop + row-level diagnostic - #35
Open
mono424 wants to merge 2 commits into
Open
fix(scheduler): break non-converging SSP catch-up loop + row-level diagnostic#35mono424 wants to merge 2 commits into
mono424 wants to merge 2 commits into
Conversation
… diagnostic A deterministic scheduler-vs-circuit catch-up hash gap (observed on the job and statistics_job outbox tables) made the SSP re-bootstrap forever: verify_catchup_at_m failed, forced a resync, the SSP refetched the identical diverging state, and failed again. Nothing counted the failures and the supervisor exempts integrity resyncs from the crash cap, so the cluster sat with no ready SSP and every live query failing. Add a consecutive catch-up failure counter per SSP and escalate instead of looping: - after SPKY_CATCHUP_RECLONE_AFTER (default 3) failures, re-clone the replica from upstream once (serialized by a shared lock) and flag all SSPs to re-bootstrap from the fresh snapshot; - after SPKY_CATCHUP_ADMIT_AFTER (default 5), admit the SSP to broadcast anyway with a loud alert, restoring sync. A plain re-bootstrap can never fix a deterministic representation gap, so serving is preferable to an indefinite outage. Add a both-sides diagnostic: a new SSP endpoint GET /debug/catchup-rows/:table dumps the circuit's rows, and verify_catchup_at_m fetches it to log the real row-level diff (missing, extra, and differing rows with canonical JSON) rather than the old one-sided dump of the scheduler's first N rows. Falls back to the one-sided dump when the SSP is an older build, so there is no strict co-deploy order (the breaker is scheduler-only). Also gate the "fully caught up and ready" log on actual verification success. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Break the non-converging SSP catch-up loop (reclone-then-admit breaker) and add a both-sides row-level catch-up diagnostic. See the preceding fix commit for detail. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A deterministic scheduler-vs-circuit catch-up hash gap (seen on the
job/statistics_joboutbox tables) makes an SSP re-bootstrap forever:verify_catchup_at_mfails, forces a resync, the SSP refetches the identical diverging state, and fails again. Nothing counts the failures and thespky devsupervisor exempts integrity resyncs from the crash cap, so the cluster sits with no ready SSP and every live query failing (observed on staging 2026-07-05, 14 cycles in ~2 min).This is NOT the canary.98 null/
_00_rvasymmetry (that strip is already active). A plain re-bootstrap can never fix a deterministic representation gap.Fix
Circuit-breaker (per-SSP consecutive catch-up failure counter):
SPKY_CATCHUP_RECLONE_AFTER(default 3) failures, re-clone the replica from upstream once (serialized by a shared lock), then flag all SSPs to re-bootstrap from the fresh snapshot;SPKY_CATCHUP_ADMIT_AFTER(default 5), admit the SSP to broadcast anyway with a loud alert, restoring sync. Set equal to skip re-clone; set admit huge to disable admit.Both-sides diagnostic: new SSP endpoint
GET /debug/catchup-rows/:tabledumps the circuit's rows;verify_catchup_at_mfetches it and logs the real row-level diff (missing / extra / differing rows with canonical JSON) instead of the old one-sided dump. Falls back to the one-sided dump on older SSP builds, so no strict co-deploy order (breaker is scheduler-only). This will finally reveal the exact diverging field.Also gates the misleading "fully caught up and ready" log on actual verification success.
Tests
cargo test -p scheduler --lib(17, incl. 3 new: pool counter + threshold flooring),ssp-protocol+ssp --lib(20) green. Both binaries build clean.Note:
apps/scheduler/tests/integration.rshad pre-existingMetricsStatefield drift (since canary.58); only theSspManagementStateliteral was updated here.🤖 Generated with Claude Code