Pre-flight checks
Bug Description
ValkeyCluster already has careful sequencing when it updates ValkeyNode specs: one node at a time, replicas before primaries, and a failover attempt before rolling a primary.
That care stops at the ValkeyNode object. Each ValkeyNode owns a single-pod StatefulSet (or Deployment) and rewrites the pod template as soon as the computed template differs from what is live. There is no coordination across nodes at that layer.
So when every node needs a new pod template at the same time, every pod restarts together. A common case is changing the Valkey image on the cluster. The same full restart also shows up when an operator upgrade changes how the pod template is built (for example defaults, probes, or sidecars), even if the ValkeyCluster spec did not change.
On a multi-node cluster that is a full outage, not a rolling update.
Steps to Reproduce
- Deploy a multi-node ValkeyCluster that is healthy (several shards and replicas).
- Change something that updates every node's pod template at once. Easiest cases:
- bump the Valkey image on the ValkeyCluster, or
- upgrade the operator to a version that builds a different pod template for the same ValkeyNode spec.
- Watch the StatefulSets and pods for all ValkeyNodes in that cluster.
Expected Behaviour
Nodes come up on the new pod template in a staged way, consistent with the existing cluster roll behaviour: limited concurrency, replicas before primaries, and no simultaneous restart of the whole cluster.
Actual Behaviour
Every ValkeyNode applies its StatefulSet or Deployment template update in the same wave. All pods restart together. The cluster loses capacity across shards at once.
Operator version
main / recent releases (behaviour is in the ValkeyNode workload apply path; not limited to one tag)
Kubernetes version
Reproduced on kind and on multi-node clusters (server 1.28+)
Kubernetes distribution / environment
kind (repro) and production-like multi-node clusters
Additional context
The gap is the split between:
- ValkeyCluster → ValkeyNode spec updates — already staged (one at a time, replica-first, proactive failover before primary).
- ValkeyNode → StatefulSet/Deployment pod template — applied immediately whenever the computed template differs, with no cluster-wide permit or sibling check.
Image changes are the example operators feel first, because the pod template image is what people change on purpose. Operator upgrades that rewrite the built template hit the same path without any ValkeyCluster spec edit.
I have a fix in progress that stages pod template applies behind a cluster-granted permit (same spirit as the existing one-at-a-time Spec rolls) and can open a PR once this issue is filed.
Pre-flight checks
main) and the issue still reproduces.Bug Description
ValkeyCluster already has careful sequencing when it updates ValkeyNode specs: one node at a time, replicas before primaries, and a failover attempt before rolling a primary.
That care stops at the ValkeyNode object. Each ValkeyNode owns a single-pod StatefulSet (or Deployment) and rewrites the pod template as soon as the computed template differs from what is live. There is no coordination across nodes at that layer.
So when every node needs a new pod template at the same time, every pod restarts together. A common case is changing the Valkey image on the cluster. The same full restart also shows up when an operator upgrade changes how the pod template is built (for example defaults, probes, or sidecars), even if the ValkeyCluster spec did not change.
On a multi-node cluster that is a full outage, not a rolling update.
Steps to Reproduce
Expected Behaviour
Nodes come up on the new pod template in a staged way, consistent with the existing cluster roll behaviour: limited concurrency, replicas before primaries, and no simultaneous restart of the whole cluster.
Actual Behaviour
Every ValkeyNode applies its StatefulSet or Deployment template update in the same wave. All pods restart together. The cluster loses capacity across shards at once.
Operator version
main / recent releases (behaviour is in the ValkeyNode workload apply path; not limited to one tag)
Kubernetes version
Reproduced on kind and on multi-node clusters (server 1.28+)
Kubernetes distribution / environment
kind (repro) and production-like multi-node clusters
Additional context
The gap is the split between:
Image changes are the example operators feel first, because the pod template image is what people change on purpose. Operator upgrades that rewrite the built template hit the same path without any ValkeyCluster spec edit.
I have a fix in progress that stages pod template applies behind a cluster-granted permit (same spirit as the existing one-at-a-time Spec rolls) and can open a PR once this issue is filed.