Commit 05010b1
authored
feat: Add zone scheduling axis (#340)
<!--
Thanks for contributing to Valkey Operator!
Please make sure you are aware of our contributing guidelines [available
here](https://github.com/valkey-io/valkey-operator/blob/main/CONTRIBUTING.md)
-->
This PR closes #314.
### Summary
Implements `spec.scheduling.zone` as defined in the epic #299.
```yaml
spec:
scheduling:
node:
spread:
shards:
mode: Disabled
primaries:
mode: Disabled
pods:
mode: Disabled
```
### Features / Behaviour Changes
- Adds spec.scheduling.zone.spread.{shard, primaries, pods}, the
availability-zone counterpart to node.spread.
- Opt-in: every dimension defaults to Disabled, so an operator upgrade
adds no constraints and triggers no rolling restart.
- The node and zone axes are independent — they key on different
topology labels (kubernetes.io/hostname vs topology.kubernetes.io/zone),
so both can be enabled at once.
- Unlike the node axis, zone.spread.shard renders as a topology spread
constraint (balancing), not pod anti-affinity. Anti-affinity would cap a
shard's members at one per zone, making Required unsatisfiable whenever
a shard has more members than there are zones; a balancing TSC spreads
them as evenly as the zone count allows without that cap.
### Implementation
All three dimensions render as topology spread constraints on
topology.kubernetes.io/zone:
- shard — label-selects each shard (valkey.io/shard-index), applied to
every pod, so a shard's members balance across zones.
- primaries — label-selects primaries only (node-index=0), so each
shard's primary-at-creation balances across zones.
- pods — label-selects every cluster pod.
Required → whenUnsatisfiable: DoNotSchedule, Preferred → ScheduleAnyway,
Disabled → nothing rendered. Curated constraints are appended to any
user topologySpreadConstraints (passed through verbatim), never mutating
the user's list.
### Limitations
- Kubernetes allows only one topology spread constraint per
{topologyKey, whenUnsatisfiable}. Because zone shard is also a TSC, all
three zone dimensions compete for the two slots per zone, so at most one
may be Required and one Preferred. CEL validation on ValkeyClusterSpec
rejects violations on manifest apply.
- A passthrough topologySpreadConstraints entry on
topology.kubernetes.io/zone that collides with an enabled zone dimension
of the same whenUnsatisfiable is likewise rejected at admission.
- zone.pinning (deterministic per-pod zone assignment) is not included
in this PR — it is deferred to a later phase.
### Testing
Unit tests cover the rendering helpers (effectiveZoneSpread,
zoneSpreadTSCs) and the controller wiring in buildClusterValkeyNode.
envtest cases cover the CEL validation (the two-slot rules,
passthrough-vs-curated collisions, and the omitted-field fallback).
Manually tested on a kind cluster with zone-labelled workers:
```yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
labels:
topology.kubernetes.io/zone: az-a
- role: worker
labels:
topology.kubernetes.io/zone: az-b
- role: worker
labels:
topology.kubernetes.io/zone: az-c
```
**Applied below to spread pods in a shard across zones:**
```yaml
apiVersion: valkey.io/v1alpha1
kind: ValkeyCluster
metadata:
name: cluster-sample-topology-spread
spec:
shards: 3
replicas: 1
scheduling:
zone:
spread:
shard:
mode: Required
```
Templates on the pods:
```yaml
topologySpreadConstraints:
- labelSelector:
matchLabels:
valkey.io/cluster: cluster-sample-topology-spread
# Differs per shard
valkey.io/shard-index: "0"
maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
```
**Applied below to spread primaries across zones:**
```yaml
apiVersion: valkey.io/v1alpha1
kind: ValkeyCluster
metadata:
name: cluster-sample-topology-spread
spec:
shards: 3
replicas: 1
scheduling:
zone:
spread:
primaries:
mode: Required
```
Created these pods:
```
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
valkey-cluster-sample-topology-spread-0-0-0 2/2 Running 0 21s 10.244.2.4 valkey-operator-zone-test-worker2 <none> <none>
valkey-cluster-sample-topology-spread-0-1-0 2/2 Running 0 21s 10.244.1.5 valkey-operator-zone-test-worker3 <none> <none>
valkey-cluster-sample-topology-spread-1-0-0 2/2 Running 0 21s 10.244.3.5 valkey-operator-zone-test-worker <none> <none>
valkey-cluster-sample-topology-spread-1-1-0 2/2 Running 0 21s 10.244.3.4 valkey-operator-zone-test-worker <none> <none>
valkey-cluster-sample-topology-spread-2-0-0 2/2 Running 0 21s 10.244.1.6 valkey-operator-zone-test-worker3 <none> <none>
valkey-cluster-sample-topology-spread-2-1-0 2/2 Running 0 21s 10.244.2.5 valkey-operator-zone-test-worker2 <none> <none>
```
Templates on the primary pods:
```yaml
topologySpreadConstraints:
- labelSelector:
matchLabels:
valkey.io/cluster: cluster-sample-topology-spread
valkey.io/node-index: "0"
maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
```
### Checklist
Before submitting the PR make sure the following are checked:
- [x] This Pull Request is related to one issue.
- [x] Commit message explains what changed and why
- [x] Tests are added or updated.
- [x] Documentation files are updated.
- [ ] I have run pre-commit locally (`pre-commit run --all-files` or
hooks on commit)
---------
Signed-off-by: Joseph Heyburn <jdheyburn@gmail.com>1 parent 7b02994 commit 05010b1
11 files changed
Lines changed: 778 additions & 7 deletions
File tree
- api/v1alpha1
- config
- crd/bases
- samples
- docs
- internal/controller
- test/e2e
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
128 | 128 | | |
129 | 129 | | |
130 | 130 | | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
131 | 137 | | |
132 | 138 | | |
133 | 139 | | |
| |||
183 | 189 | | |
184 | 190 | | |
185 | 191 | | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
186 | 224 | | |
187 | 225 | | |
188 | 226 | | |
189 | 227 | | |
190 | 228 | | |
191 | 229 | | |
| 230 | + | |
| 231 | + | |
192 | 232 | | |
| 233 | + | |
| 234 | + | |
193 | 235 | | |
| 236 | + | |
| 237 | + | |
194 | 238 | | |
| 239 | + | |
| 240 | + | |
195 | 241 | | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
196 | 254 | | |
197 | 255 | | |
198 | 256 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
0 commit comments