feat: recover Cluster from a lost primary majority

A Cluster that loses more than half its primaries at once cannot heal
itself — voting a replica in needs a master quorum that no longer exists,
so it sits in cluster_state:fail with unserved slots indefinitely while its
healthy, current replicas wait for votes that never come.

The operator now drives recovery. Once the survivors are provably below a
voting quorum (healthy*2 <= total) and each dead primary is fenced from two
independent perspectives — the k8s API view of pod/node liveness AND a
direct data-path check — it waits out a 45s debounce so it never races a
failover gossip could still perform, then issues CLUSTER FAILOVER TAKEOVER
on the most up-to-date surviving replica of each shard. Re-created primaries
rejoin as replicas via their retained PVCs and the cluster self-heals.

Automatic for the Cache profile; opt-in for Durable via the
valkey.wellcake.io/quorum-takeover annotation, since a forced takeover can
drop acknowledged writes. Write safety rests on Valkey's own
minority-write-block (a primary partitioned from the majority stops serving
after cluster-node-timeout), not on proving process death — see ADR 0006.

Surfaced via status.quorumDownSince and failover_total{reason=
"cluster-takeover"}. Validated live on k3d: majority of primaries fenced,
recovered to cluster_state:ok with 0 data loss and the fenced primaries
rejoining as replicas.

Author: melancholictheory

CHANGELOG.md

@@ -6,6 +6,23 @@ follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+### Added
+- Cluster topology now recovers automatically from the loss of a **majority of
+  primaries** — the disaster gossip cannot fix on its own, because voting a
+  replica in needs a master quorum that no longer exists, so the cluster would
+  otherwise sit in `cluster_state:fail` with unserved slots indefinitely. The
+  operator confirms each dead primary is fenced from two independent
+  perspectives (the k8s API view of pod/node liveness AND a direct data-path
+  check), waits out a debounce so it never races a failover gossip could still
+  perform, then issues `CLUSTER FAILOVER TAKEOVER` on the most up-to-date
+  surviving replica of each shard. The re-created primaries rejoin as replicas
+  via their retained PVCs. Automatic for the Cache profile; opt-in for Durable
+  (via `valkey.wellcake.io/quorum-takeover: "true"`, since a forced takeover can
+  drop acknowledged writes). Surfaced through `status.quorumDownSince` and the
+  `failover_total{reason="cluster-takeover"}` metric. See ADR 0006. Validated
+  live on k3d: majority of primaries fenced, cluster recovered to `ok` with 0
+  data loss and the fenced primaries rejoining as replicas.
+
 ### Changed
 - Replicas now authenticate to their primary as a dedicated, least-privilege ACL
   user (`replicator`, granting only `+psync +replconf +ping` and no key access)

api/v1beta1/valkeycluster_types.go

@@ -584,6 +584,17 @@ type ValkeyClusterStatus struct {
 	// +optional
 	PrimaryDownSince *metav1.Time `json:"primaryDownSince,omitempty"`
 
+	// QuorumDownSince is when the operator first observed the Cluster topology
+	// stuck in cluster_state:fail with a majority of primaries unreachable —
+	// the state gossip cannot self-heal, because authorizing a replica failover
+	// needs a master quorum that no longer exists. It debounces operator-driven
+	// quorum recovery (CLUSTER FAILOVER TAKEOVER): the operator only intervenes
+	// after the cluster has been stuck for the recovery threshold, giving gossip
+	// time to promote replicas on its own where a quorum still exists. Cleared
+	// once the cluster reports cluster_state:ok again.
+	// +optional
+	QuorumDownSince *metav1.Time `json:"quorumDownSince,omitempty"`
+
 	// InternalEndpoint is the in-cluster client endpoint (host:port) of the
 	// client Service, for consumers to connect to. Surfaced here so wrappers
 	// (e.g. a Crossplane Composition) can project it without recomputing names.

api/v1beta1/zz_generated.deepcopy.go

@@ -5089,6 +5089,18 @@ spec:
                   prematurely failed over. Cleared once a master is reachable again.
                 format: date-time
                 type: string
+              quorumDownSince:
+                description: |-
+                  QuorumDownSince is when the operator first observed the Cluster topology
+                  stuck in cluster_state:fail with a majority of primaries unreachable —
+                  the state gossip cannot self-heal, because authorizing a replica failover
+                  needs a master quorum that no longer exists. It debounces operator-driven
+                  quorum recovery (CLUSTER FAILOVER TAKEOVER): the operator only intervenes
+                  after the cluster has been stuck for the recovery threshold, giving gossip
+                  time to promote replicas on its own where a quorum still exists. Cleared
+                  once the cluster reports cluster_state:ok again.
+                format: date-time
+                type: string
               readyReplicas:
                 description: ReadyReplicas is the count of ready Valkey pods.
                 format: int32

charts/valkey-operator/templates/crd/cache.wellcake.io_valkeyclusters.yaml

@@ -5085,6 +5085,18 @@ spec:
                   prematurely failed over. Cleared once a master is reachable again.
                 format: date-time
                 type: string
+              quorumDownSince:
+                description: |-
+                  QuorumDownSince is when the operator first observed the Cluster topology
+                  stuck in cluster_state:fail with a majority of primaries unreachable —
+                  the state gossip cannot self-heal, because authorizing a replica failover
+                  needs a master quorum that no longer exists. It debounces operator-driven
+                  quorum recovery (CLUSTER FAILOVER TAKEOVER): the operator only intervenes
+                  after the cluster has been stuck for the recovery threshold, giving gossip
+                  time to promote replicas on its own where a quorum still exists. Cleared
+                  once the cluster reports cluster_state:ok again.
+                format: date-time
+                type: string
               readyReplicas:
                 description: ReadyReplicas is the count of ready Valkey pods.
                 format: int32

config/crd/bases/cache.wellcake.io_valkeyclusters.yaml

@@ -0,0 +1,117 @@
+# ADR 0006 — Recovery from a lost primary majority (Cluster topology)
+
+- Status: **Implemented** for **Cluster** topology. Automatic for the **Cache**
+  profile; **opt-in for Durable** via the per-cluster annotation
+  `valkey.wellcake.io/quorum-takeover: "true"` (a forced takeover can drop
+  acknowledged writes, which the Durable profile promises to keep). Validated
+  live on k3d: a 3-shard cluster, majority of primaries fenced, recovered to
+  `cluster_state:ok` with **0 data loss** and the fenced primaries rejoining as
+  replicas. See "Validation" below.
+- Date: 2026-08-04
+- Context: ADR 0004 handles *planned* restarts and *single* primary loss (a
+  replica is promoted, reactively or proactively). It does not cover the
+  disaster where a Cluster loses the **majority of its primaries at once** — an
+  AZ/node-group outage, a bad drain, a correlated crash.
+
+## Context
+
+Valkey Cluster fails a primary over by **vote**: a replica asking to take its
+dead primary's slots needs acknowledgement from a majority of the *masters*.
+That is exactly what a majority-primary loss removes. With `N` primaries and more
+than `floor(N/2)` of them down, the survivors are not a quorum, so **no replica
+can be voted in**. The cluster sits in `cluster_state:fail` with unserved slots
+**indefinitely** — the surviving replicas of the dead primaries are healthy and
+current, but gossip will never promote them. This is a genuine SPOF for the
+Cluster topology (the AR1 gap): the data is there, but nothing brings it back.
+
+The one Valkey primitive that breaks the deadlock is `CLUSTER FAILOVER TAKEOVER`:
+a replica promotes itself **unilaterally**, bumping its `configEpoch` above every
+peer and claiming its primary's slots **without a vote**. It is powerful and
+correspondingly dangerous — run it while the old primary is still serving and you
+get two owners for the same slots (split-brain). Gossip therefore never issues it
+on its own; a human (or an operator) must decide the old primary is really gone.
+
+An operator is well placed to make that call, because it sees something gossip
+cannot: **pod and node liveness through the k8s API**, plus a direct out-of-band
+connection to every node.
+
+## Decision
+
+Add an operator-driven recovery path (`maybeRecoverClusterQuorum`, ahead of the
+allReady-gated scale/survey steps so it runs *during* the outage). It intervenes
+only when **all** of the following hold:
+
+1. **Enabled for this cluster** — Cache by default; Durable only with the opt-in
+   annotation. Availability-first vs durability-first is a per-workload choice.
+2. **Below a voting quorum** — from `CLUSTER NODES`, `healthyMasters*2 <=
+   totalMasters`. With a quorum intact, gossip can (and should) heal on its own,
+   or deliberately refuse under the Durable replica-validity factor; the operator
+   must not race or overrule it. A minority of primaries down is left alone.
+3. **Stuck for the debounce** — `cluster_state != ok` continuously for
+   `quorumRecoveryDownAfter` (45s). Gossip promotes a recoverable failure within
+   seconds; only an unrecoverable one stays stuck this long.
+4. **Each dead primary is fenced from two independent perspectives:**
+   - **k8s view** — the pod is missing, not Running, not Ready, or on a NotReady/
+     absent node; AND
+   - **data-path view** — the operator cannot reach it, or reaches it and it
+     reports `cluster_state:fail` (it has detected its own minority isolation).
+     A primary reachable **and** reporting `cluster_state:ok` is genuinely
+     serving and is **never** taken over.
+
+For each such shard the operator deletes the dead primary's pod (fencing it and
+letting the StatefulSet re-create it) and issues `CLUSTER FAILOVER TAKEOVER` on
+the shard's most up-to-date reachable replica. The re-created old primary rejoins
+through its retained data PVC, sees the higher `configEpoch`, and demotes itself
+to a replica — the cluster self-heals back to full redundancy.
+
+### Why this is not split-brain (the honest argument)
+
+Write safety does **not** rest on proving the old primary's process is dead — no
+k8s operator can do that without STONITH, and on a control-plane partition a
+pod on a NotReady node may still be running. It rests on a guarantee Valkey
+Cluster already makes: **a primary partitioned from the majority of masters for
+longer than `cluster-node-timeout` stops accepting writes** (it returns
+`CLUSTERDOWN`, because a replica on the majority side may have been promoted).
+The debounce (45s) is an order of magnitude above the operator's node-timeout
+(5s), so by the time a takeover fires, any still-running-but-isolated old primary
+has already stopped serving writes. The k8s fence and the data-path check are
+defence in depth on top of that: they decide *when* to intervene and refuse to
+overrule a primary that is demonstrably still serving.
+
+Residual, accepted limits: a minority primary may still answer **stale reads**
+until it rejoins (the Cache profile, the only one that recovers automatically,
+accepts this AP trade-off); and a pathological *asymmetric* partition is outside
+what any voteless takeover can fully rule out. Durable clusters opt in
+deliberately, or stay stuck for a human to resolve.
+
+## Consequences
+
+- The Cluster topology recovers automatically from a lost primary majority
+  (Cache), closing the AR1 SPOF, with a clear, auditable trail (structured logs
+  + the `failover_total{reason="cluster-takeover"}` metric + the
+  `status.quorumDownSince` debounce marker).
+- Durable clusters keep their no-silent-loss contract by default; recovering
+  them is a conscious opt-in.
+- A shard that lost **every** pod (primary and all replicas) is *not* recovered
+  by takeover — there is nothing to promote. That is a restore, out of scope
+  here, and is logged as such.
+- The feature is Cluster-only; Replication/Sentinel already fail over reactively
+  (ADR 0004) and have no quorum to lose.
+
+## Validation
+
+Live on a dedicated k3d cluster (3 shards, 1 replica each, Cache profile,
+Valkey 8.0):
+
+1. Loaded 500 keys; cordoned all nodes; force-deleted 2 of 3 primary pods so they
+   stayed Pending (fenced) while their replicas kept running — a true majority
+   loss (`cluster_state:fail`, `healthy=1/3`).
+2. The operator armed the debounce (`quorum lost and recoverable; starting
+   recovery debounce`), waited 45s, then issued `CLUSTER FAILOVER TAKEOVER` on
+   both surviving replicas.
+3. Result: `cluster_state:ok`, all 16384 slots served, `cluster_size:3`, **500/500
+   keys intact**. The takeover replicas carried `configEpoch` bumped above the
+   fenced primaries.
+4. Uncordoned: the two fenced primaries rescheduled (retained PVC) and rejoined
+   as **replicas** of the new primaries — full redundancy restored, no operator
+   action needed. `status.quorumDownSince` cleared on `cluster_state:ok`.

docs/adr/0006-majority-primaries-lost-recovery.md

@@ -388,6 +388,41 @@ The operator does not intervene. The Sentinel quorum elects a new primary;
 clients learn it via `SENTINEL get-master-addr-by-name`. The operator only
 maintains the StatefulSets, ConfigMaps, and Services.
 
+## Cluster: recovering from a lost primary majority (ADR 0006)
+
+A Cluster fails a single primary over by gossip vote, which needs a **majority of
+masters**. If it loses more than half its primaries at once (an AZ/node-group
+outage, a bad drain), the survivors are not a quorum and gossip **cannot** promote
+the healthy, current replicas of the dead primaries — the cluster stays in
+`cluster_state:fail` with unserved slots indefinitely.
+
+The operator recovers this automatically for the **Cache** profile. It only acts
+once the survivors are provably below a quorum, the cluster has been stuck for a
+45s debounce (so it never races a failover gossip could still do), and each dead
+primary is fenced from both the k8s API (pod/node not serving) and a direct
+data-path check. It then issues `CLUSTER FAILOVER TAKEOVER` on the best surviving
+replica of each shard. The re-created primaries rejoin as replicas via their
+retained PVCs — no manual step.
+
+```sh
+# watch it happen
+kubectl -n <ns> get valkeycluster <cluster> -o jsonpath='{.status.quorumDownSince}'  # set while stuck, cleared on recovery
+kubectl -n valkey-operator-system logs deploy/valkey-operator-controller-manager | grep 'quorum recovery'
+# metric: failover_total{reason="cluster-takeover"}
+```
+
+**Durable profile:** recovery is **off by default**, because a forced takeover can
+promote a slightly-behind replica and drop acknowledged writes — the opposite of
+the Durable contract. A stuck Durable cluster waits for a human. Opt in only when
+you accept that trade-off:
+
+```sh
+kubectl -n <ns> annotate valkeycluster <cluster> valkey.wellcake.io/quorum-takeover=true
+```
+
+A shard that lost **every** pod (primary and all its replicas) is not recovered by
+takeover — there is nothing to promote. Restore it from a backup instead.
+
 ## Cluster per-shard workload (ADR 0005, experimental)
 
 With `spec.perShardWorkload: true`, a Cluster is rendered as **one

docs/runbook.md

@@ -57,39 +57,23 @@ func (r *ValkeyClusterReconciler) reconcileCluster(ctx context.Context, vc *cach
 		}
 	}
 
-	// Scale-up: if the spec asks for more replicas than the cluster currently
-	// knows about (Status.LastAppliedReplicas), and all desired pods are
-	// already Ready, run a one-shot add-node Job. Slot rebalance is gated
-	// by spec.autoReshard.
-	if vc.Status.ClusterInitialized && allReady && vc.Status.LastAppliedReplicas < want {
-		result, err := r.runClusterScaleUp(ctx, vc, password)
-		if err != nil || !result.IsZero() {
-			return result, err
-		}
-	}
-
-	// Scale-down: if the spec wants fewer replicas than the cluster currently
-	// has (Status.LastAppliedReplicas), run the scale-down Job that reshards
-	// slots away from the leaving masters and del-nodes them. The StatefulSet
-	// is held at the old size by statefulSetReplicas() until this Job
-	// succeeds — otherwise pods owning slots would be deleted and lose data.
-	if vc.Status.ClusterInitialized && vc.Status.LastAppliedReplicas > want {
-		result, err := r.runClusterScaleDown(ctx, vc, password)
-		if err != nil || !result.IsZero() {
-			return result, err
-		}
+	// Quorum recovery (closes the Cluster-topology SPOF of AR1): a Cluster that
+	// has lost the majority of its primaries is stuck in cluster_state:fail —
+	// gossip cannot vote a failover without a master quorum. The operator detects
+	// it and drives CLUSTER FAILOVER TAKEOVER on surviving replicas after fencing
+	// the dead primaries via the k8s API. Automatic for Cache; opt-in for Durable
+	// (a forced takeover can drop acknowledged writes). This must run even when
+	// not all pods are Ready — an unready majority is exactly the outage it repairs
+	// — so it sits ahead of the allReady-gated scale/reshard/survey steps below.
+	if res, handled, rerr := r.maybeRecoverClusterQuorum(ctx, vc, password); handled {
+		return res, rerr
 	}
 
-	// Manual reshard request (valkey.wellcake.io/reshard): run a one-off rebalance
-	// once per distinct token. Only when the cluster is initialized and all
-	// pods are Ready so the rebalance sees a stable membership.
-	if vc.Status.ClusterInitialized && allReady {
-		if tok := vc.Annotations[reshardAnnotation]; tok != "" && tok != vc.Status.LastReshardToken {
-			result, err := r.runClusterReshard(ctx, vc, password, tok)
-			if err != nil || !result.IsZero() {
-				return result, err
-			}
-		}
+	// One-shot membership operations (scale-up add-node, scale-down reshard-away,
+	// manual reshard): each runs a Job and short-circuits the reconcile while in
+	// flight. Extracted to keep this function under the complexity budget.
+	if res, handled, rerr := r.maybeReconcileClusterScaling(ctx, vc, password, want, allReady); handled {
+		return res, rerr
 	}
 
 	// Proactive rolling restart (ADR 0004): when opted in and the cluster is in
@@ -137,6 +121,43 @@ func (r *ValkeyClusterReconciler) maybeDriveClusterRollout(
 	return ctrl.Result{}, false, nil
 }
 
+// maybeReconcileClusterScaling advances the one-shot membership operations —
+// scale-up (add-node), scale-down (reshard slots away + del-node) and a manual
+// reshard request — each of which runs a Job and short-circuits the reconcile
+// while in flight. Returns handled=true (with the Job's result) when one is in
+// progress or just started; handled=false when there is nothing to do and normal
+// reconciliation should continue. Scale-up and reshard require a stable, all-Ready
+// membership; scale-down is gated on the replica count alone because
+// statefulSetReplicas() holds the STS at its old size until the reshard completes.
+func (r *ValkeyClusterReconciler) maybeReconcileClusterScaling(
+	ctx context.Context, vc *cachev1beta1.ValkeyCluster, password string, want int32, allReady bool,
+) (ctrl.Result, bool, error) {
+	if !vc.Status.ClusterInitialized {
+		return ctrl.Result{}, false, nil
+	}
+	if allReady && vc.Status.LastAppliedReplicas < want {
+		result, err := r.runClusterScaleUp(ctx, vc, password)
+		if err != nil || !result.IsZero() {
+			return result, true, err
+		}
+	}
+	if vc.Status.LastAppliedReplicas > want {
+		result, err := r.runClusterScaleDown(ctx, vc, password)
+		if err != nil || !result.IsZero() {
+			return result, true, err
+		}
+	}
+	if allReady {
+		if tok := vc.Annotations[reshardAnnotation]; tok != "" && tok != vc.Status.LastReshardToken {
+			result, err := r.runClusterReshard(ctx, vc, password, tok)
+			if err != nil || !result.IsZero() {
+				return result, true, err
+			}
+		}
+	}
+	return ctrl.Result{}, false, nil
+}
+
 // ensureClusterWorkload reconciles the static prerequisites for a Cluster-topology
 // object — password Secret, headless+client Services, ConfigMap, StatefulSet, PDB
 // and NetworkPolicy — and returns the resolved password plus the live StatefulSet.