ApplyFromCache never re-fetches provider components for a fixed version once a cache Secret exists (stale cache is self-perpetuating)

[Melkamhawy97]

/kind bug

What steps did we take and what happened?

When an InfrastructureProvider's on-cluster cache holds stale component manifests for a given spec.version, the operator never re-fetches from fetchConfig.url and keeps re-applying the stale objects on every reconcile.

The reconcile phase list begins with ApplyFromCache (v0.25.0 ref):

r.ReconcilePhases = []PhaseFn{
    reconciler.ApplyFromCache,      // <-- runs first
    reconciler.PreflightChecks,
    reconciler.InitializePhaseReconciler,
    reconciler.DownloadManifests,   // <-- never reached when ApplyFromCache short-circuits
    ...
    reconciler.Install,
    reconciler.ReportStatus,
    reconciler.Finalize,
}

ApplyFromCache (L438) compares a hash of the provider spec + the cache Secret contents against the applied-spec-hash annotation. When it matches, it re-applies the cached objects and returns early (logging "Applying provider configuration from cache" / "Applied all objects from cache"), so DownloadManifests never runs and the remote is never consulted.

Consequences we observed:

If the cache Secret contains stale/incorrect components for a version, every reconcile re-applies the stale objects.
Manually correcting the live objects (e.g. fixing an RBAC ClusterRole) is reverted on the next reconcile.
Deleting the cache ConfigMap alone does not help, the operator short-circuits on the cache Secret before reaching DownloadManifests, so it never re-fetches and the ConfigMap is not recreated.
The only ways to recover are deleting the cache Secret (forces a cache miss → re-fetch) or bumping spec.version.

Reproduction (on a kind cluster, operator v0.25.0):

1. Install an InfrastructureProvider at version vX via fetchConfig.url. Let it finish (Finalize writes applied-spec-hash on the cache Secret infrastructure--vX-cache).

2. Make the live components drift from the cache (e.g. kubectl edit a cached object, or arrange for the cache to hold older content than the remote serves), without changing spec.version.

3. Trigger a reconcile (annotate the provider). Observed: ApplyFromCache short-circuits; logs Applied all objects from cache; the cached (stale) state is re-applied; no re-fetch.

4. kubectl delete configmap infrastructure--vX and reconcile. Observed: still short-circuits on the Secret; ConfigMap is not recreated; no re-fetch.

5. kubectl delete secret infrastructure--vX-cache and reconcile. Observed: cache miss → DownloadManifests runs → components re-fetched from the URL → ConfigMap + Secret recreated → live objects corrected. Durable across subsequent reconciles and operator restarts.

Some way to reconcile the cache against the source without manual Secret deletion or a version bump. Concretely, any of:

• A documented/supported mechanism to force a re-fetch for a fixed version.

• ApplyFromCache detecting that the live cluster state has drifted from the cache and re-applying (or re-fetching) accordingly, rather than relying solely on the spec/cache hash.

• Deleting the cache ConfigMap (or some annotation) triggering a re-fetch.

• At minimum, documenting that the cache is content-blind to the remote for a fixed version, and that the cache Secret (not just the ConfigMap) must be removed to force a refresh.

Environment

• Cluster API Operator version: v0.25.0

• Cluster API version: v1.10.6

• Kubernetes version: v1.30 (EKS); also reproduced on a local kind cluster

• Provider: an InfrastructureProvider using spec.fetchConfig.url

[k8s-ci-robot]

This issue is currently awaiting triage.

If CAPI Operator contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.