Completed HookSucceeded resource leaks hook finalizer when live object is missing from cache #29464

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

A completed sync hook can be left in the cluster with
argocd.argoproj.io/hook-finalizer when its live object is temporarily
missing from Argo CD's reconciliation cache.

The sync operation is reported as Succeeded, even though:

• The hook still exists in Kubernetes.
• Its Argo CD hook finalizer remains.
• Its HookSucceeded deletion policy was not applied.

If the Application is later deleted with cascading deletion, Kubernetes marks
the hook as terminating, but the leaked hook finalizer prevents its deletion.
Argo CD's Application deletion loop then waits indefinitely for that resource.

The relevant task appears in the final sync snapshot as:

PostSync/0 hook apps/Deployment:<namespace>/maintenance-worker
nil->obj
(Synced,Succeeded,deployment.apps/maintenance-worker created)

nil->obj means task.liveObj is nil. Kubernetes object history confirms that
the Deployment still existed at this point and contained:

finalizers:
  - argocd.argoproj.io/hook-finalizer

The current implementation silently skips cleanup in this state.

removeHookFinalizer returns success without querying Kubernetes:

func (sc *syncContext) removeHookFinalizer(
    ctx context.Context,
    task *syncTask,
) error {
    if task.liveObj == nil {
        return nil
    }
    // ...
}

Completed hooks are also only selected for deletion when their cached live
object is non-nil:

hooksPendingDeletionSuccessful := tasks.Filter(func(task *syncTask) bool {
    return task.isHook() &&
        task.liveObj != nil &&
        !task.running() &&
        task.deleteOnPhaseSuccessful()
})

The operation can therefore report success without verifying whether the
completed hook still exists.

The same logic is present in v3.4.8, v3.5.2, and current master.

PR #26286 added a live API lookup for missing resources in runningTasks, but
a completed hook does not enter that path.

To Reproduce

The controller-level race is concurrency-dependent. A deterministic unit-test
reproduction is described below.

Controller reproduction

1. Configure Argo CD with annotation-based resource tracking and multiple
   operation processors. The observed environment used:

configs:
  cm:
    application.resourceTrackingMethod: annotation

controller:
  operationProcessors: 50

2. Create a Helm chart containing the same named Deployment as both a
   pre-upgrade and post-upgrade hook:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: maintenance-worker
  annotations:
    helm.sh/hook: pre-upgrade,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
spec:
  replicas: 1
  selector:
    matchLabels:
      app: maintenance-worker
  template:
    metadata:
      labels:
        app: maintenance-worker
    spec:
      containers:
        - name: worker
          image: nginx:alpine

3. Add a PostSync Job that runs longer than the Deployment takes to become
   healthy:

apiVersion: batch/v1
kind: Job
metadata:
  name: post-sync-delay
  annotations:
    helm.sh/hook: post-upgrade
    helm.sh/hook-delete-policy: hook-succeeded
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: delay
          image: busybox:1.36
          command: ["sh", "-c", "sleep 60"]

4. Create and synchronize more Applications concurrently than the configured
   operation processor count. Use a separate namespace for each Application.

5. Wait for the operations to report Succeeded.

6. Search for completed hooks that retained the Argo CD finalizer:

kubectl get deployments -A -o json |
jq -r '
  .items[]
  | select(.metadata.name == "maintenance-worker")
  | select(
      (.metadata.finalizers // [])
      | index("argocd.argoproj.io/hook-finalizer")
    )
  | [
      .metadata.namespace,
      .metadata.name,
      .metadata.creationTimestamp,
      (.metadata.deletionTimestamp // "-")
    ]
  | @tsv
'

7. Inspect the affected Application's controller logs. Its final Tasks
   snapshot contains:

maintenance-worker nil->obj (Synced,Succeeded,...)

8. Confirm that the Deployment exists in Kubernetes and still has:

finalizers:
  - argocd.argoproj.io/hook-finalizer

9. Delete the Application using cascading deletion.

The Deployment receives a deletionTimestamp but cannot disappear. The
Application remains terminating while Argo CD repeatedly waits for its managed
resources to be deleted.

Deterministic unit-test reproduction

Adapt TestSync_HooksDeletedAfterSyncSucceeded in
gitops-engine/pkg/sync/sync_context_test.go:

1. Create a completed hook with:
   • HookDeletePolicyHookSucceeded
   • argocd.argoproj.io/hook-finalizer
2. Keep that hook in the fake dynamic Kubernetes client, proving that the API
   object exists.
3. Set its ReconciliationResult.Live entry to nil, modelling a temporary
   reconciliation-cache miss.
4. Preserve the successful hook result using WithInitialState.
5. Call syncCtx.Sync().

Current behavior:

• The operation becomes OperationSucceeded.
• No API GET is made for the missing cached object.
• The finalizer is not removed.
• The delete request is not made.
• The hook remains in the fake Kubernetes client.

One way to extend the existing test is to keep hook3 in
fakeDynamicClient, but replace its entry in ReconciliationResult.Live
with nil. The assertion should verify that the hook is still present after
Sync(), demonstrating the current bug.

Expected behavior

A temporary reconciliation-cache miss should not cause completed-hook cleanup
to be skipped.

When a completed hook has task.liveObj == nil, Argo CD should perform a live
Kubernetes API GET before deciding that cleanup is unnecessary:

• If the object exists, attach it to the task, remove the Argo CD hook
  finalizer, and apply its hook deletion policy.
• If the object is genuinely NotFound, treat cleanup as complete.
• If the API lookup fails, retry or fail the operation rather than silently
  reporting success.

The sync operation should not report Succeeded while a completed
HookSucceeded resource still exists with Argo CD's hook finalizer.

Screenshots

Not applicable.

Version

argocd-application-controller: v3.4.5
BuildDate: 2026-07-09T16:15:22Z
GitCommit: 564b94973b284b8de98da7cee6eeade2cb941e46

The affected installation used Argo CD Helm chart 10.2.1.

The relevant cleanup logic was also checked in v3.4.8, v3.5.2, and current
master, where it remains unchanged.