Skip to content

Server-side diff broken with Kustomize + Helm #27259

@gregbrowndev

Description

@gregbrowndev

Describe the bug

When an Application uses Kustomize + Helm, the server-side diff strategy seems temperamental. I believe that previous diffs leaves behind the Kustomize downloaded charts/ directories on the ArgoCD server which causes subsequent diffs to fail.

To Reproduce

The same error can be reproduced locally:

  1. Use kustomize build --enable-helm . to render a Kustomize + Helm application

  2. Observe that Kustomize downloads the chart to a local charts/ directory

  3. Create an ArgoCD diff locally with argocd:

    argocd app diff my-application \
      --server-side-generate \
      --local "$PWD" \
      --refresh
  4. Observe error (redacted for brevity):

    {
      "level": "fatal",
      "msg": "rpc error: code = Unknown desc = `kustomize build /tmp/[redacted] --enable-helm  failed exit status 1: Error: Error: template: synthetic-monitoring-agent/templates/serviceaccount.yaml:5:11: executing \"synthetic-monitoring-agent/templates/serviceaccount.yaml\" at \u003cinclude \"synthetic-monitoring-agent.serviceAccountName\" .\u003e: error calling include: template: no template \"synthetic-monitoring-agent.serviceAccountName\" associated with template \"gotpl\"\n\n' with env=[HELM_CONFIG_HOME=/tmp/kustomize-helm-3418589331/helm HELM_CACHE_HOME=/tmp/kustomize-helm-3418589331/helm/.cache HELM_DATA_HOME=/tmp/kustomize-helm-3418589331/helm/.data] (is 'helm' installed?): exit status 1",
      "time": "2026-04-09T16:03:50+01:00"
    }

This error goes away if you either:

  1. Delete the charts/ directory in your local source tree
  2. Commit charts/ to git (not verified but I suspect it would because...)
  3. Set the --local-include "*.*" flag in the diff command (I found this fix was reported several years ago here: argocd 2.5: --server-side-generate fails with helm charts ? #11315 (reply in thread))

The problem here is --local uploads your local directory, but --local-include defaults to *.{yaml,yml,json}, the charts/ directory is only partially uploaded, missing the _helpers.tpl template in particular.

All this is relevant because I suspect when using the server side diff strategy, these charts/ directories are also being retained between diffs on the ArgoCD server itself.

This issue (#711) seems to suggest ArgoCD uses git clean -fdx between manifest generation. The -x being important since git clean -fd wouldn't clean up any files excluded by .gitignore (so if we're ignoring the charts directory they wouldn't be cleaned up).

Maybe whats happened in my case is I've run argocd app diff locally, uploading a partial charts/ directory which is then cached and isn't getting cleaned up by subsequent runs, breaking my ArgoCD server. (Might be wrong tho, assuming the user uploaded directory is cached in the same way ArgoCD caches git directories for an app...). The only other explanation is that ArgoCD isn't doing git clean -fdx between runs.

Expected behavior

Server-side diff whether run by ArgoCD or invoked by the argocd CLI should not be affected by Kustomize's charts/ directories from previous invocations.

Screenshots

Version

argocd: v3.3.6+998fb59
  BuildDate: 2026-03-27T14:09:03Z
  GitCommit: 998fb59dc355653c0657908a6ea2f87136e022d1
  GitTreeState: clean
  GoVersion: go1.25.5
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v3.3.6
  BuildDate: 2026-03-27T13:41:15Z
  GitCommit: 998fb59dc355653c0657908a6ea2f87136e022d1
  GitTreeState: clean
  GitTag: v3.3.6
  GoVersion: go1.25.5
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.8.1 2026-02-09T16:15:27Z
  Helm Version: v3.19.4+g7cfb6e4
  Kubectl Version: v0.34.0
  Jsonnet Version: v0.21.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingtriage/pendingThis issue needs further triage to be correctly classified

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions