Skip to content

Commit 05010b1

Browse files
authored
feat: Add zone scheduling axis (#340)
<!-- Thanks for contributing to Valkey Operator! Please make sure you are aware of our contributing guidelines [available here](https://github.com/valkey-io/valkey-operator/blob/main/CONTRIBUTING.md) --> This PR closes #314. ### Summary Implements `spec.scheduling.zone` as defined in the epic #299. ```yaml spec: scheduling: node: spread: shards: mode: Disabled primaries: mode: Disabled pods: mode: Disabled ``` ### Features / Behaviour Changes - Adds spec.scheduling.zone.spread.{shard, primaries, pods}, the availability-zone counterpart to node.spread. - Opt-in: every dimension defaults to Disabled, so an operator upgrade adds no constraints and triggers no rolling restart. - The node and zone axes are independent — they key on different topology labels (kubernetes.io/hostname vs topology.kubernetes.io/zone), so both can be enabled at once. - Unlike the node axis, zone.spread.shard renders as a topology spread constraint (balancing), not pod anti-affinity. Anti-affinity would cap a shard's members at one per zone, making Required unsatisfiable whenever a shard has more members than there are zones; a balancing TSC spreads them as evenly as the zone count allows without that cap. ### Implementation All three dimensions render as topology spread constraints on topology.kubernetes.io/zone: - shard — label-selects each shard (valkey.io/shard-index), applied to every pod, so a shard's members balance across zones. - primaries — label-selects primaries only (node-index=0), so each shard's primary-at-creation balances across zones. - pods — label-selects every cluster pod. Required → whenUnsatisfiable: DoNotSchedule, Preferred → ScheduleAnyway, Disabled → nothing rendered. Curated constraints are appended to any user topologySpreadConstraints (passed through verbatim), never mutating the user's list. ### Limitations - Kubernetes allows only one topology spread constraint per {topologyKey, whenUnsatisfiable}. Because zone shard is also a TSC, all three zone dimensions compete for the two slots per zone, so at most one may be Required and one Preferred. CEL validation on ValkeyClusterSpec rejects violations on manifest apply. - A passthrough topologySpreadConstraints entry on topology.kubernetes.io/zone that collides with an enabled zone dimension of the same whenUnsatisfiable is likewise rejected at admission. - zone.pinning (deterministic per-pod zone assignment) is not included in this PR — it is deferred to a later phase. ### Testing Unit tests cover the rendering helpers (effectiveZoneSpread, zoneSpreadTSCs) and the controller wiring in buildClusterValkeyNode. envtest cases cover the CEL validation (the two-slot rules, passthrough-vs-curated collisions, and the omitted-field fallback). Manually tested on a kind cluster with zone-labelled workers: ```yaml kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane - role: worker labels: topology.kubernetes.io/zone: az-a - role: worker labels: topology.kubernetes.io/zone: az-b - role: worker labels: topology.kubernetes.io/zone: az-c ``` **Applied below to spread pods in a shard across zones:** ```yaml apiVersion: valkey.io/v1alpha1 kind: ValkeyCluster metadata: name: cluster-sample-topology-spread spec: shards: 3 replicas: 1 scheduling: zone: spread: shard: mode: Required ``` Templates on the pods: ```yaml topologySpreadConstraints: - labelSelector: matchLabels: valkey.io/cluster: cluster-sample-topology-spread # Differs per shard valkey.io/shard-index: "0" maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule ``` **Applied below to spread primaries across zones:** ```yaml apiVersion: valkey.io/v1alpha1 kind: ValkeyCluster metadata: name: cluster-sample-topology-spread spec: shards: 3 replicas: 1 scheduling: zone: spread: primaries: mode: Required ``` Created these pods: ``` NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES valkey-cluster-sample-topology-spread-0-0-0 2/2 Running 0 21s 10.244.2.4 valkey-operator-zone-test-worker2 <none> <none> valkey-cluster-sample-topology-spread-0-1-0 2/2 Running 0 21s 10.244.1.5 valkey-operator-zone-test-worker3 <none> <none> valkey-cluster-sample-topology-spread-1-0-0 2/2 Running 0 21s 10.244.3.5 valkey-operator-zone-test-worker <none> <none> valkey-cluster-sample-topology-spread-1-1-0 2/2 Running 0 21s 10.244.3.4 valkey-operator-zone-test-worker <none> <none> valkey-cluster-sample-topology-spread-2-0-0 2/2 Running 0 21s 10.244.1.6 valkey-operator-zone-test-worker3 <none> <none> valkey-cluster-sample-topology-spread-2-1-0 2/2 Running 0 21s 10.244.2.5 valkey-operator-zone-test-worker2 <none> <none> ``` Templates on the primary pods: ```yaml topologySpreadConstraints: - labelSelector: matchLabels: valkey.io/cluster: cluster-sample-topology-spread valkey.io/node-index: "0" maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule ``` ### Checklist Before submitting the PR make sure the following are checked: - [x] This Pull Request is related to one issue. - [x] Commit message explains what changed and why - [x] Tests are added or updated. - [x] Documentation files are updated. - [ ] I have run pre-commit locally (`pre-commit run --all-files` or hooks on commit) --------- Signed-off-by: Joseph Heyburn <jdheyburn@gmail.com>
1 parent 7b02994 commit 05010b1

11 files changed

Lines changed: 778 additions & 7 deletions

api/v1alpha1/valkeycluster_types.go

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,12 @@ type SchedulingSpec struct {
128128
// Disabled and no scheduling primitives are emitted.
129129
// +optional
130130
Node *NodeScheduling `json:"node,omitempty"`
131+
132+
// Zone groups scheduling constraints on the zone axis
133+
// (topologyKey topology.kubernetes.io/zone). When unset, every zone spread is
134+
// Disabled and no scheduling primitives are emitted.
135+
// +optional
136+
Zone *ZoneScheduling `json:"zone,omitempty"`
131137
}
132138

133139
// SpreadMode selects the strength of a spread constraint.
@@ -183,16 +189,68 @@ type NodeScheduling struct {
183189
Spread NodeSpread `json:"spread,omitempty"`
184190
}
185191

192+
// ZoneSpread controls how the cluster's pods are distributed across zones
193+
// (topologyKey topology.kubernetes.io/zone). Every dimension renders as a
194+
// topology spread constraint — balancing, not anti-affinity — so zone shard
195+
// members may share a zone when the shard is larger than the zone count.
196+
type ZoneSpread struct {
197+
// Shard balances the pods of each shard across zones, rendered as a topology
198+
// spread constraint. Defaults to Disabled when unset.
199+
// +optional
200+
Shard SpreadConstraint `json:"shard,omitempty"`
201+
202+
// Primaries balances each shard's node-index-0 pod across zones, rendered as
203+
// a topology spread constraint. Defaults to Disabled when unset.
204+
// +optional
205+
Primaries SpreadConstraint `json:"primaries,omitempty"`
206+
207+
// Pods balances all of the cluster's pods across zones, rendered as a
208+
// topology spread constraint. Defaults to Disabled when unset. Enabling this
209+
// alongside an explicit shard or primaries spread of the same strength is
210+
// rejected (only one topology spread constraint per strength is permitted per
211+
// zone).
212+
// +optional
213+
Pods SpreadConstraint `json:"pods,omitempty"`
214+
}
215+
216+
// ZoneScheduling groups scheduling constraints on the zone axis
217+
// (topologyKey topology.kubernetes.io/zone).
218+
type ZoneScheduling struct {
219+
// Spread distributes the cluster's pods across zones.
220+
// +optional
221+
Spread ZoneSpread `json:"spread,omitempty"`
222+
}
223+
186224
// ValkeyClusterSpec defines the desired state of ValkeyCluster.
187225
// +kubebuilder:validation:XValidation:rule="!(has(self.persistence) && self.workloadType == 'Deployment')",message="persistence requires workloadType StatefulSet"
188226
// +kubebuilder:validation:XValidation:rule="!has(oldSelf.persistence) || has(self.persistence)",message="persistence cannot be removed once set"
189227
// +kubebuilder:validation:XValidation:rule="has(oldSelf.persistence) || !has(self.persistence)",message="persistence cannot be added after creation"
190228
// +kubebuilder:validation:XValidation:rule="!has(self.persistence) || !has(oldSelf.persistence) || quantity(self.persistence.size).compareTo(quantity(oldSelf.persistence.size)) >= 0",message="persistence.size may only be expanded"
191229
// +kubebuilder:validation:XValidation:rule="!has(self.persistence) || !has(oldSelf.persistence) || ((!has(self.persistence.storageClassName) && !has(oldSelf.persistence.storageClassName)) || (has(self.persistence.storageClassName) && has(oldSelf.persistence.storageClassName) && self.persistence.storageClassName == oldSelf.persistence.storageClassName))",message="persistence.storageClassName is immutable"
230+
//
231+
// node.spread: reject primaries and pods both Required (they would render duplicate hostname DoNotSchedule constraints).
192232
// +kubebuilder:validation:XValidation:rule="!has(self.scheduling) || !has(self.scheduling.node) || !has(self.scheduling.node.spread) || !( ((has(self.scheduling.node.spread.primaries) && has(self.scheduling.node.spread.primaries.mode)) ? self.scheduling.node.spread.primaries.mode : 'Disabled') == 'Required' && ((has(self.scheduling.node.spread.pods) && has(self.scheduling.node.spread.pods.mode)) ? self.scheduling.node.spread.pods.mode : 'Disabled') == 'Required' )",message="node.spread.primaries and node.spread.pods cannot both be Required: they render duplicate kubernetes.io/hostname DoNotSchedule topology spread constraints"
233+
//
234+
// node.spread: reject primaries and pods both Preferred (they would render duplicate hostname ScheduleAnyway constraints).
193235
// +kubebuilder:validation:XValidation:rule="!has(self.scheduling) || !has(self.scheduling.node) || !has(self.scheduling.node.spread) || !( ((has(self.scheduling.node.spread.primaries) && has(self.scheduling.node.spread.primaries.mode)) ? self.scheduling.node.spread.primaries.mode : 'Disabled') == 'Preferred' && ((has(self.scheduling.node.spread.pods) && has(self.scheduling.node.spread.pods.mode)) ? self.scheduling.node.spread.pods.mode : 'Disabled') == 'Preferred' )",message="node.spread.primaries and node.spread.pods cannot both be Preferred: they render duplicate kubernetes.io/hostname ScheduleAnyway topology spread constraints (set one to Disabled or Required)"
236+
//
237+
// node.spread: reject a user hostname DoNotSchedule TSC that collides with a Required primaries/pods spread.
194238
// +kubebuilder:validation:XValidation:rule="!has(self.scheduling) || !has(self.scheduling.topologySpreadConstraints) || !self.scheduling.topologySpreadConstraints.exists(c, c.topologyKey == 'kubernetes.io/hostname' && c.whenUnsatisfiable == 'DoNotSchedule') || !has(self.scheduling.node) || !has(self.scheduling.node.spread) || !( ((has(self.scheduling.node.spread.primaries) && has(self.scheduling.node.spread.primaries.mode)) ? self.scheduling.node.spread.primaries.mode : 'Disabled') == 'Required' || ((has(self.scheduling.node.spread.pods) && has(self.scheduling.node.spread.pods.mode)) ? self.scheduling.node.spread.pods.mode : 'Disabled') == 'Required' )",message="a topologySpreadConstraints entry on kubernetes.io/hostname with whenUnsatisfiable DoNotSchedule collides with node.spread.primaries or node.spread.pods set to Required, which render the same hostname DoNotSchedule constraint: set that node.spread mode to Disabled, or remove the passthrough constraint"
239+
//
240+
// node.spread: reject a user hostname ScheduleAnyway TSC that collides with a Preferred primaries/pods spread.
195241
// +kubebuilder:validation:XValidation:rule="!has(self.scheduling) || !has(self.scheduling.topologySpreadConstraints) || !self.scheduling.topologySpreadConstraints.exists(c, c.topologyKey == 'kubernetes.io/hostname' && c.whenUnsatisfiable == 'ScheduleAnyway') || !has(self.scheduling.node) || !has(self.scheduling.node.spread) || !( ((has(self.scheduling.node.spread.primaries) && has(self.scheduling.node.spread.primaries.mode)) ? self.scheduling.node.spread.primaries.mode : 'Disabled') == 'Preferred' || ((has(self.scheduling.node.spread.pods) && has(self.scheduling.node.spread.pods.mode)) ? self.scheduling.node.spread.pods.mode : 'Disabled') == 'Preferred' )",message="a topologySpreadConstraints entry on kubernetes.io/hostname with whenUnsatisfiable ScheduleAnyway collides with node.spread.primaries or node.spread.pods set to Preferred, which render the same hostname ScheduleAnyway constraint: set that node.spread mode to Disabled or Required, or remove the passthrough constraint"
242+
//
243+
// zone.spread: reject more than one of shard/primaries/pods Required (they would render duplicate zone DoNotSchedule constraints).
244+
// +kubebuilder:validation:XValidation:rule="!has(self.scheduling) || !has(self.scheduling.zone) || !has(self.scheduling.zone.spread) || !( (((has(self.scheduling.zone.spread.shard) && has(self.scheduling.zone.spread.shard.mode)) ? self.scheduling.zone.spread.shard.mode : 'Disabled') == 'Required' && ((has(self.scheduling.zone.spread.primaries) && has(self.scheduling.zone.spread.primaries.mode)) ? self.scheduling.zone.spread.primaries.mode : 'Disabled') == 'Required') || (((has(self.scheduling.zone.spread.shard) && has(self.scheduling.zone.spread.shard.mode)) ? self.scheduling.zone.spread.shard.mode : 'Disabled') == 'Required' && ((has(self.scheduling.zone.spread.pods) && has(self.scheduling.zone.spread.pods.mode)) ? self.scheduling.zone.spread.pods.mode : 'Disabled') == 'Required') || (((has(self.scheduling.zone.spread.primaries) && has(self.scheduling.zone.spread.primaries.mode)) ? self.scheduling.zone.spread.primaries.mode : 'Disabled') == 'Required' && ((has(self.scheduling.zone.spread.pods) && has(self.scheduling.zone.spread.pods.mode)) ? self.scheduling.zone.spread.pods.mode : 'Disabled') == 'Required') )",message="at most one of zone.spread.shard, zone.spread.primaries, zone.spread.pods may be Required: they render duplicate topology.kubernetes.io/zone DoNotSchedule topology spread constraints"
245+
//
246+
// zone.spread: reject more than one of shard/primaries/pods Preferred (they would render duplicate zone ScheduleAnyway constraints).
247+
// +kubebuilder:validation:XValidation:rule="!has(self.scheduling) || !has(self.scheduling.zone) || !has(self.scheduling.zone.spread) || !( (((has(self.scheduling.zone.spread.shard) && has(self.scheduling.zone.spread.shard.mode)) ? self.scheduling.zone.spread.shard.mode : 'Disabled') == 'Preferred' && ((has(self.scheduling.zone.spread.primaries) && has(self.scheduling.zone.spread.primaries.mode)) ? self.scheduling.zone.spread.primaries.mode : 'Disabled') == 'Preferred') || (((has(self.scheduling.zone.spread.shard) && has(self.scheduling.zone.spread.shard.mode)) ? self.scheduling.zone.spread.shard.mode : 'Disabled') == 'Preferred' && ((has(self.scheduling.zone.spread.pods) && has(self.scheduling.zone.spread.pods.mode)) ? self.scheduling.zone.spread.pods.mode : 'Disabled') == 'Preferred') || (((has(self.scheduling.zone.spread.primaries) && has(self.scheduling.zone.spread.primaries.mode)) ? self.scheduling.zone.spread.primaries.mode : 'Disabled') == 'Preferred' && ((has(self.scheduling.zone.spread.pods) && has(self.scheduling.zone.spread.pods.mode)) ? self.scheduling.zone.spread.pods.mode : 'Disabled') == 'Preferred') )",message="at most one of zone.spread.shard, zone.spread.primaries, zone.spread.pods may be Preferred: they render duplicate topology.kubernetes.io/zone ScheduleAnyway topology spread constraints (set one to Disabled or Required)"
248+
//
249+
// zone.spread: reject a user zone DoNotSchedule TSC that collides with a Required shard/primaries/pods spread.
250+
// +kubebuilder:validation:XValidation:rule="!has(self.scheduling) || !has(self.scheduling.topologySpreadConstraints) || !self.scheduling.topologySpreadConstraints.exists(c, c.topologyKey == 'topology.kubernetes.io/zone' && c.whenUnsatisfiable == 'DoNotSchedule') || !has(self.scheduling.zone) || !has(self.scheduling.zone.spread) || !( ((has(self.scheduling.zone.spread.shard) && has(self.scheduling.zone.spread.shard.mode)) ? self.scheduling.zone.spread.shard.mode : 'Disabled') == 'Required' || ((has(self.scheduling.zone.spread.primaries) && has(self.scheduling.zone.spread.primaries.mode)) ? self.scheduling.zone.spread.primaries.mode : 'Disabled') == 'Required' || ((has(self.scheduling.zone.spread.pods) && has(self.scheduling.zone.spread.pods.mode)) ? self.scheduling.zone.spread.pods.mode : 'Disabled') == 'Required' )",message="a topologySpreadConstraints entry on topology.kubernetes.io/zone with whenUnsatisfiable DoNotSchedule collides with zone.spread.shard, primaries, or pods set to Required, which render the same zone DoNotSchedule constraint: set that zone.spread mode to Disabled, or remove the passthrough constraint"
251+
//
252+
// zone.spread: reject a user zone ScheduleAnyway TSC that collides with a Preferred shard/primaries/pods spread.
253+
// +kubebuilder:validation:XValidation:rule="!has(self.scheduling) || !has(self.scheduling.topologySpreadConstraints) || !self.scheduling.topologySpreadConstraints.exists(c, c.topologyKey == 'topology.kubernetes.io/zone' && c.whenUnsatisfiable == 'ScheduleAnyway') || !has(self.scheduling.zone) || !has(self.scheduling.zone.spread) || !( ((has(self.scheduling.zone.spread.shard) && has(self.scheduling.zone.spread.shard.mode)) ? self.scheduling.zone.spread.shard.mode : 'Disabled') == 'Preferred' || ((has(self.scheduling.zone.spread.primaries) && has(self.scheduling.zone.spread.primaries.mode)) ? self.scheduling.zone.spread.primaries.mode : 'Disabled') == 'Preferred' || ((has(self.scheduling.zone.spread.pods) && has(self.scheduling.zone.spread.pods.mode)) ? self.scheduling.zone.spread.pods.mode : 'Disabled') == 'Preferred' )",message="a topologySpreadConstraints entry on topology.kubernetes.io/zone with whenUnsatisfiable ScheduleAnyway collides with zone.spread.shard, primaries, or pods set to Preferred, which render the same zone ScheduleAnyway constraint: set that zone.spread mode to Disabled or Required, or remove the passthrough constraint"
196254
type ValkeyClusterSpec struct {
197255

198256
// Override the default Valkey image

api/v1alpha1/zz_generated.deepcopy.go

Lines changed: 39 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)