Skip to content

Commit 8dd02dd

Browse files
JeffreyDallasclaude
andcommitted
fix: replace waitForRunningPhase with 120s sleep in block node restart recovery
waitForRunningPhase's createdAfter filter can miss the StatefulSet replacement pod, causing it to block for its full 15-minute timeout and exhaust the entire waitForMirrorReadiness window. In CI run 29062610572 the mirror importer had recovered to only 62 s of lag (down from 928 s) but the window expired with no time remaining for the final catch-up. Replace the waitForRunningPhase call with a simple 120 s sleep. This leaves ~720 s of the 900 s readiness window for lag checks after the restart, which is sufficient for the mirror to drain its remaining backlog. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>
1 parent 0175e74 commit 8dd02dd

1 file changed

Lines changed: 10 additions & 13 deletions

File tree

src/commands/rapid-fire.ts

Lines changed: 10 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -549,22 +549,19 @@ export class RapidFireCommand extends BaseCommand {
549549
const k8: ReturnType<typeof this.k8Factory.getK8> = this.k8Factory.getK8(config.context);
550550
const blockNodePods: Pod[] = await k8.pods().list(config.namespace, [constants.SOLO_BLOCK_NODE_NAME_LABEL]);
551551
if (blockNodePods.length > 0) {
552-
const restartedAfter: Date = new Date();
553552
await Promise.all(
554553
blockNodePods.map((pod: Pod): Promise<void> => k8.pods().readByReference(pod.podReference).killPod(0)),
555554
);
556-
this.logger.info(`Killed ${blockNodePods.length} block node pod(s), waiting for replacement pod`);
557-
await k8
558-
.pods()
559-
.waitForRunningPhase(
560-
config.namespace,
561-
[constants.SOLO_BLOCK_NODE_NAME_LABEL],
562-
constants.BLOCK_NODE_PODS_RUNNING_MAX_ATTEMPTS,
563-
constants.BLOCK_NODE_PODS_RUNNING_DELAY,
564-
undefined,
565-
restartedAfter,
566-
);
567-
this.logger.info('Block node pod restart complete; resuming mirror readiness wait');
555+
// Wait 120 s for the StatefulSet to restart the pod, CN to reconnect, and CN to
556+
// resend the missing WRB block as backfill. We do not call waitForRunningPhase
557+
// here because its createdAfter filter can miss the replacement pod, blocking for
558+
// its full 15-minute timeout and exhausting the readiness window before any further
559+
// lag checks can confirm that the mirror importer has resumed.
560+
this.logger.info(
561+
`Killed ${blockNodePods.length} block node pod(s); waiting 120 s for CN reconnect and backfill`,
562+
);
563+
await sleep(Duration.ofSeconds(120));
564+
this.logger.info('Block node restart wait complete; resuming mirror readiness wait');
568565
} else {
569566
this.logger.info('No block node pods found; skipping restart');
570567
}

0 commit comments

Comments
 (0)