Recover a Cluster from a lost primary majority

[melancholictheory]

A Cluster fails a single primary over by gossip vote, which needs a majority of masters. Lose more than half the primaries at once (an AZ outage, a bad drain, a correlated crash) and the survivors aren't a quorum, so gossip can't vote any replica in. The cluster then sits in cluster_state:fail with unserved slots for as long as the outage lasts, even though the dead primaries' replicas are healthy and current. That is a real single point of failure for the Cluster topology: the data is right there, but nothing brings it back.

This adds an operator-driven recovery path for that case.

How it decides to act

The operator only steps in when all of these hold, so it never races a failover gossip could still do and never overrules a primary that might still be serving:

• Recovery is enabled for the cluster. Cache by default; Durable only with the valkey.wellcake.io/quorum-takeover: "true" annotation.
• The survivors are below a voting quorum: from CLUSTER NODES, healthyMasters*2 <= totalMasters. A minority of primaries down is left to gossip.
• The cluster has been stuck for a 45s debounce.
• Each dead primary is fenced from two independent perspectives: the k8s API (pod missing, not Running, not Ready, or on a NotReady node) and a direct data-path check (unreachable, or reachable but reporting cluster_state:fail). A primary that is reachable and reports cluster_state:ok is genuinely serving, and is never taken over.

It then deletes the dead primary's pod and runs CLUSTER FAILOVER TAKEOVER on the shard's most up-to-date reachable replica. The re-created primary rejoins through its retained PVC, sees the higher configEpoch, and demotes itself, so full redundancy comes back without any manual step.

Why it isn't split-brain

Write safety doesn't depend on proving the old primary's process is dead. No operator can do that without STONITH. It depends on a guarantee Valkey already makes: a primary partitioned from the majority for longer than cluster-node-timeout stops accepting writes. The 45s debounce is far longer than the 5s node-timeout, so by the time a takeover fires, a still-running but isolated old primary has already stopped serving writes. The k8s fence and the data-path check sit on top of that as defence in depth. A minority primary can still answer stale reads until it rejoins, which the Cache profile accepts and Durable opts into. The full argument is in ADR 0006.

A cross-model review shaped the safety design

An adversarial review pass went after the split-brain edge cases and changed the design. It led to the two-perspective fence instead of trusting a NotReady node on its own; the majority gate, so a minority failure is left to gossip and the Durable validity-factor refusal is never overruled; the Durable opt-in, because a forced takeover can drop acknowledged writes; validating CLUSTER NODES names against real pods, so a bare IP with no announce hostname can't look fenced; and clearing the debounce marker only on cluster_state:ok.

Validation

• Unit tests cover the fence truth table, the majority gate including the even-N boundary, the debounce, the profile and annotation gating, and the pre-dial fence filter.
• Full envtest and lint are clean.
• Live on a dedicated k3d cluster (3 shards, 1 replica each, Cache): loaded 500 keys, cordoned every node, then force-deleted 2 of 3 primary pods so they stayed fenced while their replicas kept running. The operator armed the debounce, waited 45s, and took over both replicas. Result: cluster_state:ok, 16384 slots served, 500/500 keys intact. After uncordoning, the fenced primaries rescheduled and rejoined as replicas, and status.quorumDownSince cleared.

Closes the AR1 Cluster SPOF.