operator: memleak with identities of deleted Pods retained in memory

[mloiseleur]

v1.2.3, hubble control plane, EKS + VPC CNI, 3 nodes / 169 pods.

retina-operator RSS grows linearly, 27 → 68 MiB over 15 days, no plateau.

The operator holds one entry per allocated identity in IdentityManager.labelIdentities and in the underlying CachingIdentityAllocator local keys. It is freed only when a Pod delete event is matched in the store (handlePodDelete); anything missed is retained for the process lifetime. Every CronJob Pod allocates a unique identity (job-name/controller-uid labels) — ~4500/day on this cluster.

Evidence: the operator still syncs identities of Jobs that finished days ago, whose CiliumIdentity is long gone:

level=warn caller=identitybackend/identity.go:184 msg="Unable update CRD identity information with a reference for this node" error="identity (id:\"5277\",key:\"[... k8s:job-name=nats-backup-29773815]\") does not exist"

The job-name suffix is minutes-since-epoch: that Job ran 2 days earlier, and the oldest one still referenced is 10 days old. The CiliumIdentity CRs themselves are GC'd correctly (252 live).

[mloiseleur]

Repro on kind (v1.2.3)

kind create cluster --name retina-repro
helm install retina oci://ghcr.io/microsoft/retina/charts/retina-hubble --version v1.2.3 -n kube-system \
  --set agent.tag=v1.2.3 --set agent.init.tag=v1.2.3 --set operator.tag=v1.2.3 \
  --set hubble.tls.auto.method=cronJob --set hubble.ui.enabled=false
# + metrics-server (with --kubelet-insecure-tls) for `kubectl top`

# short-lived Job pods, each with unique job-name/controller-uid labels
for i in $(seq 1 450); do
  kubectl create -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata: {name: churn-$i}
spec:
  ttlSecondsAfterFinished: 10
  template:
    spec:
      restartPolicy: Never
      containers: [{name: c, image: busybox:1.36, command: ["true"]}]
EOF
done

kubectl -n kube-system top pod -l control-plane=retina-operator:

state	operator RSS
idle baseline	23 MiB
after 150 Job pods	30 MiB
after 300	37 MiB
after 450	44 MiB

~7 MiB per 150 pods, linear, no plateau.

Worth noting: every one of those identities was released on the happy path — released identity due to no more references appears 452 times. Deleting what is left (kubectl delete ciliumidentities --all) does not give the memory back either: 36 MiB fourteen minutes later, against a 23 MiB baseline. So ~13 MiB is held for identities that exist neither in the cluster nor in the allocator's reference counts.

(Side observation: on this kind cluster the identity GC never marked anything in ~1 h — no Marking CRD identity for later deletion — hence the manual delete.)