Storage spec validation blocks reconciliation and prevents cluster initialization

[MrChausson]

Problem Description

When the operator detects a mismatch between the Valkey CR's storage.spec and the existing StatefulSet's volumeClaimTemplates, it throws an error that blocks the entire reconciliation loop. This prevents cluster initialization from running, leaving pods in a failed cluster state.

Current Behavior

The operator's storage validation at valkey_controller.go:2652-2676 compares specs but:

1. Blocks reconciliation completely with error: volume claim template has changed and cannot be updated in a statefulset
2. Prevents initCluster() from running, so the cluster never forms
3. Doesn't specify what differs in the spec, making debugging difficult
4. Fails on default values - if the CR doesn't specify volumeMode, storageClassName, or accessModes, the comparison fails even though they match functionally

Reproduction Steps

1. Create a Valkey cluster with minimal storage spec:

apiVersion: hyperspike.io/v1
kind: Valkey
metadata:
  name: test-valkey
  namespace: default
spec:
  nodes: 3
  replicas: 0
  storage:
    spec:
      resources:
        requests:
          storage: 1Gi

2. Delete and recreate the Valkey CR (or update storage size)

3. Observe operator logs:

ERROR: volume claim template has changed and cannot be updated in a statefulset

4. Check cluster state:

kubectl exec test-valkey-0 -- valkey-cli cluster info
# cluster_state:fail
# cluster_known_nodes:1
# cluster_slots_assigned:0

Impact

• Cluster never initializes because reconciliation stops before initCluster()
• No clear error message about what specifically doesn't match
• Requires manual deletion of all resources to recover
• Makes updates/redeployments very fragile

Suggested Solutions

Option 1: Continue reconciliation with warning

Instead of blocking completely, log a warning and continue with existing storage:

if !cmp.Equal(currentPVCSpec, definedPVCSpec) {
    logger.Info("Storage spec differs from deployed StatefulSet, continuing with existing storage", 
        "current", currentPVCSpec, "desired", definedPVCSpec)
    // Don't return error - continue to initCluster
}

Option 2: Add admission webhook

Validate storage specs before allowing CR creation/updates to catch issues early.

Option 3: Auto-populate defaults

Set default values in the CR spec so comparisons work correctly:

if definedPVCSpec.VolumeMode == nil {
    fs := corev1.PersistentVolumeFilesystem
    definedPVCSpec.VolumeMode = &fs
}
if definedPVCSpec.StorageClassName == nil {
    definedPVCSpec.StorageClassName = &defaultStorageClass
}
// etc.

Option 4: Improve error message

Provide a clear diff showing what doesn't match:

return fmt.Errorf("volume claim template mismatch:\nCurrent: %+v\nDesired: %+v\nDiff: %s", 
    currentPVCSpec, definedPVCSpec, cmp.Diff(currentPVCSpec, definedPVCSpec))

Workaround

Specify complete storage spec including all defaults:

storage:
  spec:
    accessModes:
      - ReadWriteOnce
    volumeMode: Filesystem
    storageClassName: <your-storage-class>
    resources:
      requests:
        storage: 1Gi

Environment

• Operator version: v0.0.60
• Kubernetes: 1.35.0
• Issue occurs during reconciliation after StatefulSet already exists

Thank you @dmolik for this operator! Happy to test any fixes or provide more details if needed.