Description
In initCluster() (internal/controller/valkey_controller.go), the operator executes CLUSTER MEET for all connected nodes and then immediately iterates over CLUSTER NODES output to forget any peer that appears as disconnected. This creates a race condition: nodes that were just introduced via CLUSTER MEET may not have completed the handshake yet and still appear as disconnected in the CLUSTER NODES output. The operator then issues CLUSTER FORGET on those very same nodes it just met, effectively undoing the meet.
Root Cause
The upstream code in initCluster does the following back-to-back with no delay or safeguard:
- Meet all peers (lines 522–537):
for _, node := range connectedNodes {
for _, peer := range connectedNodes {
if node == peer { continue }
node.client.Do(ctx, node.client.B().ClusterMeet().Ip(peer.ip).Port(int64(peer.port)).Build())
}
}
- Forget disconnected peers (lines 539–573):
for _, shard := range cluster.shards {
for _, node := range shard.nodes {
info, _ := node.client.Do(ctx, node.client.B().ClusterNodes().Build()).ToString()
for _, line := range strings.Split(info, "\n") {
// ...parse peerId and connected status...
if connected == "disconnected" {
node.client.Do(ctx, node.client.B().ClusterForget().NodeId(peerId).Build())
}
}
}
}CLUSTER MEET is asynchronous — the Valkey gossip protocol needs time to complete the handshake. When CLUSTER NODES is queried immediately after, newly-met nodes may still report link-status as disconnected. The forget loop then removes them, causing:
- Nodes to be repeatedly met and forgotten on every reconciliation cycle.
- Cluster formation to stall or fail entirely, especially under network latency or high load.
- Flapping cluster topology that never converges.
Impact
- Cluster bootstrap failures: New clusters may never fully form because nodes keep getting forgotten before the handshake completes.
- Replica loss: Replica nodes (which are not flagged
master) are particularly vulnerable since the forget loop explicitly skips master-flagged nodes but not replicas that are still handshaking.
Description
In
initCluster()(internal/controller/valkey_controller.go), the operator executesCLUSTER MEETfor all connected nodes and then immediately iterates overCLUSTER NODESoutput to forget any peer that appears asdisconnected. This creates a race condition: nodes that were just introduced viaCLUSTER MEETmay not have completed the handshake yet and still appear asdisconnectedin theCLUSTER NODESoutput. The operator then issuesCLUSTER FORGETon those very same nodes it just met, effectively undoing the meet.Root Cause
The upstream code in
initClusterdoes the following back-to-back with no delay or safeguard:CLUSTER MEETis asynchronous — the Valkey gossip protocol needs time to complete the handshake. WhenCLUSTER NODESis queried immediately after, newly-met nodes may still reportlink-statusasdisconnected. The forget loop then removes them, causing:Impact
master) are particularly vulnerable since the forget loop explicitly skipsmaster-flagged nodes but not replicas that are still handshaking.