0.14.0: capsule/tenants controller never starts — *v1.ResourceQuota cache never syncs, so no Tenant is ever reconciled (silent, admission keeps working)

[antoinemichea]

Bug description

On 0.14.0 the capsule/tenants controller never starts its workers: its cache never becomes
synced, the manager exits after cacheSyncTimeout, the second replica takes the lease and dies the
same way. The result is that no Tenant is ever reconciled, while admission keeps working — so
the failure is silent.

{"level":"error","logger":"setup","msg":"problem running manager",
 "error":"failed to wait for capsule/tenants caches to sync kind source: *v1.ResourceQuota:
 timed out waiting for cache to be synced for kind source: *v1.ResourceQuota"}

Starting workers ... "controller":"capsule/tenants" never appears in the logs — not once in
~2 hours and 9 container restarts.

Impact

• observedGeneration stays behind generation: we changed spec.namespaceOptions.quota on three
  Tenants and the change was never observed (13/11, 3/1, 3/1).
• Tenant Ready conditions stay frozen on whatever they were.
• New namespaces get no RoleBindings and no ResourceQuota — the practical breaking point for us,
  since our CI creates tenant namespaces continuously.
• Admission keeps working (the webhook server starts independently of the cache), so from the
  outside the operator looks healthy. Nothing alerts.

Ruled out by measurement

Hypothesis	Check	Result
Too many objects	103 ResourceQuotas, 158 namespaces, 4 Tenants	not a volume issue
Missing RBAC	kubectl auth can-i list/watch/get resourcequotas --all-namespaces --as=system:serviceaccount:capsule-system:capsule	all yes
Watched API missing	deviceclasses, gatewayclasses, runtimeclasses, storageclasses, resourcequotas, limitranges	all served
Slow LIST	LIST per kind impersonating the controller SA (largest: 5188 RoleBindings)	all immediate
Slow nodes	moved both replicas to faster nodes	identical crashes
cacheInvalidation (chart forces 0h30m0s, API default is 24h)	set to 24h, clean restart	no change
cacheSyncTimeout too low	raised 4m → 10m	only delays the crash 4m → 10m

Only other signal: a single klog line "Unexpected error when reading response body" err="context deadline exceeded" (request.go:1196).

Decisive evidence

Rolling back to 0.13.9 with the same values, same nodes, same cluster, same object counts:
Starting workers ... "controller":"capsule/tenants" appears within seconds, all 4 Tenants go
Ready with generation == observedGeneration, zero errors, zero restarts. So this is specific to
0.14.0, not to our configuration.

Environment

• Capsule 0.14.0 (Helm chart 0.14.0) → rolled back to 0.13.9
• Kubernetes 1.35 (Talos), cert-manager for webhook TLS, 2 replicas, leader election on
• manager.options: clientConnectionQPS: 100, clientConnectionBurst: 150, --workers=1
• 4 Tenants (largest: 97 namespaces), ~103 tenant namespaces, 5188 RoleBindings cluster-wide
• webhooks.hooks.globalresourcequotas.enabled: false (no GlobalResourceQuota in use)

Happy to provide full manager logs or run any instrumented build — we can reproduce by simply
bumping back to 0.14.0.

Possibly related to the stale-namespace issue we filed separately: #2101.

[oliverbaehler]

@antoinemichea pls test latest release

[antoinemichea]

Thanks for the very fast turnaround — issue filed in the evening, fix released the next morning.

Tested v0.14.1 in production. Both issues are fixed. Rolled out on a 4-tenant cluster
(largest: 97 namespaces, ~105 tenant namespaces total, continuous CI creating and deleting
namespaces), Kubernetes 1.35, 2 replicas with leader election.

#2102 — controller now starts: CONFIRMED

Check	v0.14.0	v0.14.1
Starting workers … "controller":"capsule/tenants"	never (2 h, 9 restarts)	within seconds
capsule/tenant-resourcequotas (new)	n/a	present and running
timed out waiting for cache to be synced	every 4–10 min	0 in 3.1 h
Container restarts	9	0 in 3.1 h
Tenants	Ready frozen, observedGeneration stuck	4/4 Ready, generation == observedGeneration
New namespace gets RBAC	no	yes — 33 RoleBindings + ResourceQuota + NetworkPolicy

The dedicated controller also explains what we saw: the synchronous ResourceQuota work inside the
Tenant informer handler was starving the cache fill — a deadlock, not slowness. That is why the
symptoms looked absurd (only 103 ResourceQuotas, RBAC fine, every LIST immediate, yet the cache
never synced).

#2101 — stale namespace no longer wedges the sync: CONFIRMED

Observed two real namespace-deletion cycles (11:44 and 13:24 UTC):

• No cannot sync ResourceQuotas loop at all. Before: ~10 errors/minute, indefinitely, because
  any ResourceQuota event in any of the tenant's namespaces re-triggered the handler.
• The status now self-heals. In the second cycle, 2 stale entries were briefly visible in the
  Tenant status at +1 and +2 min, and were gone by +3 min (status.size 98 → 97). Previously
  they stayed forever and required manually patching the status subresource (both
  status.namespaces and status.spaces).

Residual errors: 3 in 3.1 hours, all transient

Reconciler error on sync ResourceQuotas (11:44, 12:48) and once on
cannot sync rolebindings items: namespace … not found (11:44) — all at namespace-deletion moments,
all retried and resolved by controller-runtime. That is healthy backoff behaviour, nothing like the
previous unbounded loop.

One observation, NOT a regression claim

The first RoleBinding on a brand-new tenant namespace appeared later than on 0.13.9:
+99 s and +195 s (0.14.1) vs +15 s to +62 s across six namespaces on 0.13.9. I want to be
explicit that this is not a regression report: my 0.14.1 sample is only 2 namespaces, created 15 s
apart, 15 minutes after a manager restart that was catching up 97 namespaces — a deliberately
unfavourable context, and we run with --workers=1. The full set of 33 RoleBindings actually
completed faster than on 0.13.9. I'll keep measuring and will open a separate issue only if it
reproduces under normal conditions.

Possible upgrade note for others (not a bug)

Rolling 0.13.x → 0.14.x briefly refuses admission in tenant namespaces: the new binary registers
generic.rules.mutating.projectcapsule.dev in capsule-dynamic-webhook as soon as it starts, while
the old pod is still in the Service endpoints and does not serve that path — so the API server gets
a 404 and fails closed (dispatcher.go:225 Failed calling webhook, failing closed). Measured ~3 s
on the first attempt and ~6 s on this one. maxUnavailable does not help: the endpoint is
ready, it just belongs to a version that ignores the path. Harmless for us (a few transient CNPG
Failed to update lease), but it may be worth a line in the upgrade notes, since it lands on
failurePolicy: Fail webhooks covering */* in tenant namespaces.

Happy to run anything else. Thanks again for the dedicated controller — it is the right fix.