Work-agent continues applying ManifestWork on spoke cluster even after deletion from hub

[ncr38]

Describe the bug
Even after ManifestWork from hub cluster was deleted , i could observe work-agent continuously doing SSA on spoke clusters.

To Reproduce
Occurred for once.

Expected behaviour
Server side apply stops on spoke cluster once the ManifestWork is deleted from HUB.

Environment ie: OCM version, Kubernetes version and provider:
Used GKE Clusters as hub and spoke with Kubernetes version : 1.33.5-gke.2118001.
Hosting OCM in singelton mode.
OCM HUB and Spoke images version 1.1.2.

Additional context

Event : ManifestWork Delete from hub cluster

I deleted a parent object which owned ManifestWork which lead to garbage collection of the ManifestWork.

{
"protoPayload": {
"@type": "type.googleapis.com/google.cloud.audit.AuditLog",
"authenticationInfo": {
"principalEmail": "system:serviceaccount:kube-system:generic-garbage-collector"
},
"authorizationInfo": [
{
"granted": true,
"permission": "io.open-cluster-management.work.v1.manifestworks.delete",
"resource": "work.open-cluster-management.io/v1/namespaces/ld-cell-01/manifestworks/stocks-live-data-krakend-change-rollout-mw"
}
],
"methodName": "io.open-cluster-management.work.v1.manifestworks.delete",
"requestMetadata": {
"callerIp": "172.16.0.3",
"callerSuppliedUserAgent": "kube-controller-manager/v1.33.5 (linux/amd64) kubernetes/f9258cc/system:serviceaccount:kube-system:generic-garbage-collector"
},
"resourceName": "work.open-cluster-management.io/v1/namespaces/ld-cell-01/manifestworks/stocks-live-data-krakend-change-rollout-mw",
"serviceName": "k8s.io",
"status": {
"code": 0
}
},
"insertId": "b0e4ceea-66da-499e-af86-016bdaa1fd5a",
"resource": {
"type": "k8s_cluster",
"labels": {
"cluster_name": "investcentral-hub01",
"location": "asia-south1",
"project_id": "gw-backend-production"
}
},
"timestamp": "2026-02-20T07:23:47.107796Z",
"labels": {
"authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding "system:controller:generic-garbage-collector" of ClusterRole "system:controller:generic-garbage-collector" to ServiceAccount "generic-garbage-collector/kube-system"",
"authentication.kubernetes.io/credential-id": "JTI=86861461-ec71-4758-973e-5e644cf0a0f6",
"authorization.k8s.io/decision": "allow"
},
"logName": "projects/gw-backend-production/logs/cloudaudit.googleapis.com%2Factivity",
"operation": {
"id": "b0e4ceea-66da-499e-af86-016bdaa1fd5a",
"producer": "k8s.io",
"first": true,
"last": true
},
"receiveTimestamp": "2026-02-20T07:24:00.655947342Z"
}

Event : ServerSideApplied on spoke clusters

There were continuous patches on the spoke cluster from work-agent even after 36 minutes of deleting ManifestWork on hub cluster.
I am running klusterlet agents in singelton mode.
Attaching log for last patch done by work-agent service account:

{
"protoPayload": {
"@type": "type.googleapis.com/google.cloud.audit.AuditLog",
"authenticationInfo": {
"principalEmail": "system:serviceaccount:open-cluster-management-agent:klusterlet-work-sa"
},
"authorizationInfo": [
{
"granted": true,
"permission": "io.argoproj.v1alpha1.rollouts.patch",
"resource": "argoproj.io/v1alpha1/namespaces/stocks-common-livedata/rollouts/stocks-live-data-krakend-rollout"
}
],
"methodName": "io.argoproj.v1alpha1.rollouts.patch",
"requestMetadata": {
"callerIp": "10.184.18.145",
"callerSuppliedUserAgent": "work-agent"
},
"resourceName": "argoproj.io/v1alpha1/namespaces/stocks-common-livedata/rollouts/stocks-live-data-krakend-rollout",
"serviceName": "k8s.io",
"status": {
"code": 0
}
},
"insertId": "bc10cc6f-6ec6-4f4e-888c-7b03fa20e269",
"resource": {
"type": "k8s_cluster",
"labels": {
"project_id": "gw-backend-production",
"cluster_name": "gw-prod-live-data-cell-01",
"location": "asia-south1"
}
},
"timestamp": "2026-02-20T08:05:53.109931Z",
"labels": {
"authentication.kubernetes.io/credential-id": "JTI=31bbeffa-ddfe-41fb-839e-3da2ec8e1e4c",
"authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding "open-cluster-management:klusterlet-work:execution-admin" of ClusterRole "admin" to ServiceAccount "klusterlet-work-sa/open-cluster-management-agent"",
"authorization.k8s.io/decision": "allow"
},
"logName": "projects/gw-backend-production/logs/cloudaudit.googleapis.com%2Factivity",
"operation": {
"id": "bc10cc6f-6ec6-4f4e-888c-7b03fa20e269",
"producer": "k8s.io",
"first": true,
"last": true
},
"receiveTimestamp": "2026-02-20T08:06:09.075817491Z"
}

Screenshot of log query showing last Manifestwork deletion from hub :

Screenshot of log query showing AppliedManifestWork getting created after ManifestWork Deletion:

Actions taken to recover from situation:

1. I tried restarting klusterlet-agent on spoke cluster, it didn't work.
2. I downscaled klusterlet-agent on the spoke cluster and it stopped continuous patched to the rollout object.
3. Restarted GRPC Server on hub cluster and deleted AppliedManifestWork resource on spoke cluster which was not getting deleted due to a finalizer applied by work-agent . I removed the finalizer and it was deleted. Patch stopped after this.

[qiujian16]

are you using grpc driver?

[ncr38]

Yes

[skeeey]

cloud you try upgrading the image to the latest, we've fixed some deletion-related bugs (e.g. open-cluster-management-io/sdk-go#198) since 1.1.2

[qiujian16]

/assign @skeeey

[rokej]

@ncr38 Did the newer image resolve the problem you faced?

[ciaranRoche]

We hit what looks like this exact behaviour in Maestro e2e (gRPC driver as well) and managed to root cause it from the agent logs. Full write-up in open-cluster-management-io/sdk-go#232.

Short version: there's a race between ManifestWorkAgentClient.Patch and the agent store's HandleReceivedResource. If the delete event lands between the patch's final store read and its store write, the patch writes back a pre-delete copy of the work and the DeletionTimestamp is erased from the agent's in-memory store. The finalize controller has usually already cleaned the spoke by then, but the work finalizer never gets removed, so the next controller requeue sees a live work with finalizers and recreates the AppliedManifestWork and reapplies the manifests. That matches the continuous SSA you saw after the hub side was deleted, and the AppliedManifestWork coming back.

Worth noting the race is still present on current main, so upgrading past sdk-go#198 won't fix this one. We're proposing a fix in the linked issue and are planning to send a PR.