Operator stops sending heartbeat to hub ADX Cluster

[tpdownes]

I have a set of kubernetes cluster operating in a spoke region that runs an ADX-Mon operator pod to send heartbeats to an ADX Hub in another region. It suddenly stopped sending heartbeats yesterday afternoon. Notice that the final heartbeat reconciliation cycle did not complete as it is missing the last 2 entries you'd expect.

{"ts":"2026-04-05T17:28:29.457245Z","lvl":"INF","msg":"Reconciling"}
{"ts":"2026-04-05T17:28:29.457318Z","lvl":"INF","msg":"Reconciling ADXCluster adx-mon/adx-prod-eastus-1 (gene
ration 1)"}
{"ts":"2026-04-05T17:28:29.457332Z","lvl":"INF","msg":"ADXCluster adx-prod-eastus-1: executing partition clus
ter heartbeat"}
{"ts":"2026-04-05T17:28:29.457338Z","lvl":"INF","msg":"ADXCluster adx-prod-eastus-1: sending heartbeats to 1
federated clusters"}
{"ts":"2026-04-05T17:28:31.075749Z","lvl":"INF","msg":"Heartbeat sent to federated cluster https://adx-hub.fake.url"}
{"ts":"2026-04-05T17:28:31.075795Z","lvl":"INF","msg":"ADXCluster adx-prod-eastus-1: heartbeat cycle complete
, requeuing in 10m0s"}
...
{"ts":"2026-04-05T17:38:31.076158Z","lvl":"INF","msg":"Reconciling"}
{"ts":"2026-04-05T17:38:31.076243Z","lvl":"INF","msg":"Reconciling ADXCluster adx-mon/adx-prod-eastus-1 (gene
ration 1)"}
{"ts":"2026-04-05T17:38:31.076258Z","lvl":"INF","msg":"ADXCluster adx-prod-eastus-1: executing partition clus
ter heartbeat"}
{"ts":"2026-04-05T17:38:31.076265Z","lvl":"INF","msg":"ADXCluster adx-prod-eastus-1: sending heartbeats to 1
federated clusters"}

All logs in the operator are at level INF. Most of the other logs resemble this, where it appears to be logging watch operations.

{"ts":"2026-04-02T17:50:41.038642Z","lvl":"INF","msg":"Watch close","values":["reflector","sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114","type","*v1.DaemonSet","totalItems",150]}
{"ts":"2026-04-02T17:50:41.038758Z","lvl":"INF","msg":"Request Body","values":["body",""]}