Skip to content

Commit 26443f0

Browse files
JeffreyDallasclaude
andcommitted
fix: wait for block node pod termination before checking new pod readiness
waitForReadyStatus returned immediately seeing the still-Ready old pod after deletion, completing deployment while the new pod was still starting. This caused the consensus node to exhaust its gRPC publisher reconnect attempts against a non-existent endpoint and never reconnect in the 5-minute window before PUBLISHER_UNAVAILABILITY_TIMEOUT fired. Add waitForPodsToTerminate between delete() and waitForReadyStatus() so that the new pod is the one polled for Ready status, giving the CN time to reconnect before the block node's publisher unavailability timer expires. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>
1 parent 6c006a4 commit 26443f0

1 file changed

Lines changed: 12 additions & 0 deletions

File tree

src/commands/one-shot/orchestrator/deploy/default-one-shot-deploy-orchestrator.ts

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -804,6 +804,18 @@ export class DefaultOneShotDeployOrchestrator implements OneShotDeployOrchestrat
804804

805805
// Restart the pod so BlockNodeApp.loadApplicationState() picks up the file on startup.
806806
await k8.pods().delete(blockNodePod.podReference);
807+
// Wait for old pod to fully terminate before waiting for the new pod.
808+
// Without this, waitForReadyStatus returns immediately seeing the still-Ready old pod,
809+
// completing deployment before the new pod exists and causing the consensus node to
810+
// exhaust its gRPC publisher reconnect attempts against a non-existent endpoint.
811+
await k8
812+
.pods()
813+
.waitForPodsToTerminate(
814+
namespace,
815+
blockNodeLabels,
816+
constants.BLOCK_NODE_PODS_RUNNING_MAX_ATTEMPTS,
817+
constants.BLOCK_NODE_PODS_RUNNING_DELAY,
818+
);
807819
await k8
808820
.pods()
809821
.waitForReadyStatus(

0 commit comments

Comments
 (0)