fix: RepairDisconnectedMasters does not heal failed followers, causing infinite Failed state after pod restarts

[svils]

Summary

Fixes #1692

• Broaden repairDisconnectedMasters → RepairDisconnectedNodes to handle both failed masters and failed slaves. For slaves, issue CLUSTER MEET (fix gossip) + CLUSTER REPLICATE (re-establish replication so the follower resolves its master's current IP from gossip).
• Add RepairStaleReplication to detect connected followers whose master_link_status is down and re-issue CLUSTER REPLICATE. This catches the subtler case where gossip is fine but replication is still pointed at a stale IP.
• Harden cluster config defaults to reduce the likelihood and blast radius of the problem:
  • tcp-keepalive 60s (Redis recommends this; faster dead connection detection)
  • cluster-node-timeout 15000ms, configurable via CLUSTER_NODE_TIMEOUT env var (more gossip recovery time before marking nodes failed)
  • cluster-allow-reads-when-down yes (keep clients unblocked while the operator repairs nodes)
  • Probe TimeoutSeconds=5 / FailureThreshold=5 (prevent Kubernetes from prematurely evicting pods during transient blips)

Both CLUSTER MEET and CLUSTER REPLICATE are idempotent and safe to retry.

Test plan

• Existing tests updated for the rename (RepairDisconnectedMasters → RepairDisconnectedNodes)
• New unit tests for replicationLinkUp (up, down, master node, empty)
• New unit tests for RepairStaleReplication (replication down → repaired, replication up → no-op)
• Updated statefulset_test.go expectations for new probe defaults
• All tests pass (go test ./internal/k8sutils/...)
• Deploy to a test cluster, restart follower pods, verify automatic recovery
• Deploy to a test cluster, restart leader pods, verify followers re-resolve master IP

[svils]

@shubham-cmyk hey, you asked me to open this PR back on #1692 — it's been about a month now with no review. we're running this in production across multiple EKS clusters and keep hitting these bugs on every node rotation.

this PR (#1705) fixes the follower repair loop, but while debugging in prod i ran into two more issues that are closely related:

• fix: Batch CLUSTER ADDSLOTS for single-leader RedisCluster to avoid exec URL limit #1706 — single-leader clusters get stuck in Bootstrap forever because ADDSLOTS exceeds the k8s exec URL limit
• fix: reset follower with stale state before add-node to prevent "Node is not empty" loop #1734 — after the leader-only CLUSTER RESET, add-node fails with "Node is not empty" because the follower still has stale state (same root cause as RedisCluster node cannot init/rejoin after cluster scale-in and scale-out due to already stored configuration #1407)

all three are straightforward fixes with tests, and they've been running stable in our production builds. would be great to get some eyes on these so they can land in the next release.

[joepizza1]

@shubham-cmyk @drivebyer can you please review and get this fix in ? We're facing this issue on few environmnets and it would be really helpful to have this fixed.