Stale namespace in Tenant status breaks the whole ResourceQuota sync in a hot loop (status.spaces, new in 0.14.0)

[antoinemichea]

Bug description

After a Tenant namespace is deleted, its entry can survive in the Tenant status, and every
subsequent ResourceQuota event for that Tenant then fails the whole ResourceQuota sync in a hot
loop:

{"level":"error","logger":"capsule.ctrl.tenant","msg":"cannot sync ResourceQuotas",
 "tenant":"opencell",
 "error":"namespace \"opencell-continuous-migration-15992\": namespaces \"opencell-continuous-migration-15992\" not found"}

We measured ~10 of these per minute, indefinitely, on a Tenant with 98 namespaces and a busy CI
creating/deleting namespaces continuously.

Where it comes from

syncResourceQuotasForResourceQuota (internal/controllers/tenant/manager.go, around L478) is
registered as an event handler on ResourceQuota. It iterates over the namespaces recorded in the
Tenant status and does a Get per namespace; a single stale entry returns NotFound and the error
is propagated, so the entire ResourceQuota sync for that Tenant fails — not just the missing
namespace. Because any ResourceQuota event in any of the Tenant's namespaces triggers the handler,
the failure repeats forever.

The Tenant itself stays state: Active but its Ready condition is never refreshed
(generation == observedGeneration), so the stale status is not self-healing.

How we hit it (reproduction)

1. Tenant with many namespaces, CI creating short-lived namespaces.
2. Upgrade Capsule so that the manager pods roll (in our case 0.13.9 → 0.14.0). During the rollout,
   admission is briefly refused in tenant namespaces — the new binary registers
   generic.rules.mutating.projectcapsule.dev in capsule-dynamic-webhook while the old pod is
   still in the Service endpoints and does not serve that path, so the API server gets a 404 and
   fails closed.
3. A namespace created during that window, then deleted by CI, stays referenced in the Tenant
   status → the loop above starts and never stops.

What made it hard to diagnose (v0.14.0 specific)

There are two lists to look at, and the second one is new in 0.14.0:

• status.namespaces (list of strings)
• status.spaces (list of objects {name, uid, metadata, conditions}) ← new in 0.14.0

Removing the stale entry from status.namespaces alone changed nothing — the error kept naming the
same namespace, including after a full rollout restart of the manager, which initially led us
to suspect a stale cache. The actual source was status.spaces. After removing the entry from both
lists, the error stopped immediately (90s sliding window: 10 → 7 → 3 → 0) and the manager went back
to zero errors.

Workaround (status subresource only, so no GitOps drift):

kubectl patch tenant <t> --subresource=status --type=json -p \
  '[{"op":"test","path":"/status/spaces/<i>/name","value":"<stale-ns>"},
    {"op":"remove","path":"/status/spaces/<i>"}]'

Suggested fix

1. Treat IsNotFound as a skip in the per-namespace loop of syncResourceQuotasForResourceQuota
   instead of failing the whole sync — a namespace that no longer exists needs no quota.
2. Prune namespaces that no longer exist from the Tenant status (both status.namespaces and
   status.spaces) during reconciliation, so the condition is self-healing.

Environment

• Capsule 0.14.0 (Helm chart 0.14.0), 2 replicas, leader election on
• Kubernetes 1.35 (Talos), cert-manager for webhook TLS
• 4 Tenants, largest one 98 namespaces, ~104 tenant namespaces overall
• webhooks.hooks.globalresourcequotas.enabled: false (no GlobalResourceQuota in use)

[antoinemichea]

Fixed by #2104 in v0.14.1 — confirmed in production today.

Tested on the cluster where this originally happened (tenant with 97 namespaces, CI creating and
deleting namespaces continuously). Across two real namespace-deletion cycles:

• No more cannot sync ResourceQuotas loop. Before, a single stale entry made every subsequent
  ResourceQuota event fail the whole tenant sync — ~10 errors/minute, indefinitely.
• The Tenant status now self-heals: 2 stale entries were briefly present at +1/+2 min after a
  deletion and were gone by +3 min (status.size 98 → 97). Previously they persisted forever and I
  had to patch the status subresource by hand.

One detail worth recording for anyone hitting this on an older version: the stale entry had to be
removed from both status.namespaces and status.spaces (the latter is new in 0.14.0).
Cleaning only status.namespaces changed nothing, and neither did a full rollout restart of the
manager — which initially led me to suspect a stale cache. Good to see the regression tests in #2104
cover both lists.

Full measurements in #2102 (comment).
Thanks for the quick fix.