Skip to content

Bug: SetupWithManager missing Owns(&appsv1.StatefulSet{}) — operator does not reconcile when StatefulSet pods or PVCs are deleted #403

Description

@aishyandapalli

The ValkeyReconciler creates a StatefulSet with controllerutil.SetControllerReference (setting the Valkey CR as the owner), but SetupWithManager only registers a watch on the Valkey custom resource itself via For(&hyperv1.Valkey{}). It does not call Owns(&appsv1.StatefulSet{}), so changes to the owned StatefulSet — including pod deletions, PVC losses, or replica count drift — do not trigger a reconciliation.

Root Cause

The upstream SetupWithManager at internal/controller/valkey_controller.go line 2768:

func (r *ValkeyReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&hyperv1.Valkey{}).
		Complete(r)
}

The controller only watches the Valkey CR and the changes on the stateful set are not reconciled.

Impact

When a pod or PVC belonging to the Valkey StatefulSet is deleted (e.g., due to node failure, manual intervention, or storage issues):

  1. No reconciliation is triggered. The operator does not notice that the StatefulSet's actual state has diverged from the desired state.
  2. The StatefulSet controller (kube-controller-manager) will recreate the pod, but the new pod comes up with a fresh Valkey instance — empty data, no cluster membership, and a new node ID.
  3. The Valkey cluster remains degraded because the operator doesn't run initCluster to re-meet the new node into the cluster, reassign slots, or reconfigure replication.
  4. Manual intervention is required — an operator restart or a no-op edit to the Valkey CR is needed to force a reconcile and heal the cluster.

This causes issues in scenarios like:

  • Node eviction / drain: Pods are rescheduled, PVCs may be lost if using local storage, and the cluster needs re-initialization.
  • PVC deletion: The StatefulSet recreates the pod with a new PVC, but without reconciliation the operator never detects the data loss or reconfigures the cluster topology.

Fix

Add Owns(&appsv1.StatefulSet{}) to SetupWithManager:

func (r *ValkeyReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&hyperv1.Valkey{}).
		Owns(&appsv1.StatefulSet{}).
		Complete(r)
}

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions