Describe the bug
When an Application uses Kustomize + Helm, the server-side diff strategy seems temperamental. I believe that previous diffs leaves behind the Kustomize downloaded charts/ directories on the ArgoCD server which causes subsequent diffs to fail.
To Reproduce
The same error can be reproduced locally:
-
Use kustomize build --enable-helm . to render a Kustomize + Helm application
-
Observe that Kustomize downloads the chart to a local charts/ directory
-
Create an ArgoCD diff locally with argocd:
argocd app diff my-application \
--server-side-generate \
--local "$PWD" \
--refresh
-
Observe error (redacted for brevity):
{
"level": "fatal",
"msg": "rpc error: code = Unknown desc = `kustomize build /tmp/[redacted] --enable-helm failed exit status 1: Error: Error: template: synthetic-monitoring-agent/templates/serviceaccount.yaml:5:11: executing \"synthetic-monitoring-agent/templates/serviceaccount.yaml\" at \u003cinclude \"synthetic-monitoring-agent.serviceAccountName\" .\u003e: error calling include: template: no template \"synthetic-monitoring-agent.serviceAccountName\" associated with template \"gotpl\"\n\n' with env=[HELM_CONFIG_HOME=/tmp/kustomize-helm-3418589331/helm HELM_CACHE_HOME=/tmp/kustomize-helm-3418589331/helm/.cache HELM_DATA_HOME=/tmp/kustomize-helm-3418589331/helm/.data] (is 'helm' installed?): exit status 1",
"time": "2026-04-09T16:03:50+01:00"
}
This error goes away if you either:
- Delete the
charts/ directory in your local source tree
- Commit
charts/ to git (not verified but I suspect it would because...)
- Set the
--local-include "*.*" flag in the diff command (I found this fix was reported several years ago here: argocd 2.5: --server-side-generate fails with helm charts ? #11315 (reply in thread))
The problem here is --local uploads your local directory, but --local-include defaults to *.{yaml,yml,json}, the charts/ directory is only partially uploaded, missing the _helpers.tpl template in particular.
All this is relevant because I suspect when using the server side diff strategy, these charts/ directories are also being retained between diffs on the ArgoCD server itself.
This issue (#711) seems to suggest ArgoCD uses git clean -fdx between manifest generation. The -x being important since git clean -fd wouldn't clean up any files excluded by .gitignore (so if we're ignoring the charts directory they wouldn't be cleaned up).
Maybe whats happened in my case is I've run argocd app diff locally, uploading a partial charts/ directory which is then cached and isn't getting cleaned up by subsequent runs, breaking my ArgoCD server. (Might be wrong tho, assuming the user uploaded directory is cached in the same way ArgoCD caches git directories for an app...). The only other explanation is that ArgoCD isn't doing git clean -fdx between runs.
Expected behavior
Server-side diff whether run by ArgoCD or invoked by the argocd CLI should not be affected by Kustomize's charts/ directories from previous invocations.
Screenshots
Version
argocd: v3.3.6+998fb59
BuildDate: 2026-03-27T14:09:03Z
GitCommit: 998fb59dc355653c0657908a6ea2f87136e022d1
GitTreeState: clean
GoVersion: go1.25.5
Compiler: gc
Platform: darwin/arm64
argocd-server: v3.3.6
BuildDate: 2026-03-27T13:41:15Z
GitCommit: 998fb59dc355653c0657908a6ea2f87136e022d1
GitTreeState: clean
GitTag: v3.3.6
GoVersion: go1.25.5
Compiler: gc
Platform: linux/amd64
Kustomize Version: v5.8.1 2026-02-09T16:15:27Z
Helm Version: v3.19.4+g7cfb6e4
Kubectl Version: v0.34.0
Jsonnet Version: v0.21.0
Describe the bug
When an Application uses Kustomize + Helm, the server-side diff strategy seems temperamental. I believe that previous diffs leaves behind the Kustomize downloaded
charts/directories on the ArgoCD server which causes subsequent diffs to fail.To Reproduce
The same error can be reproduced locally:
Use
kustomize build --enable-helm .to render a Kustomize + Helm applicationObserve that Kustomize downloads the chart to a local
charts/directoryCreate an ArgoCD diff locally with
argocd:argocd app diff my-application \ --server-side-generate \ --local "$PWD" \ --refreshObserve error (redacted for brevity):
{ "level": "fatal", "msg": "rpc error: code = Unknown desc = `kustomize build /tmp/[redacted] --enable-helm failed exit status 1: Error: Error: template: synthetic-monitoring-agent/templates/serviceaccount.yaml:5:11: executing \"synthetic-monitoring-agent/templates/serviceaccount.yaml\" at \u003cinclude \"synthetic-monitoring-agent.serviceAccountName\" .\u003e: error calling include: template: no template \"synthetic-monitoring-agent.serviceAccountName\" associated with template \"gotpl\"\n\n' with env=[HELM_CONFIG_HOME=/tmp/kustomize-helm-3418589331/helm HELM_CACHE_HOME=/tmp/kustomize-helm-3418589331/helm/.cache HELM_DATA_HOME=/tmp/kustomize-helm-3418589331/helm/.data] (is 'helm' installed?): exit status 1", "time": "2026-04-09T16:03:50+01:00" }This error goes away if you either:
charts/directory in your local source treecharts/to git (not verified but I suspect it would because...)--local-include "*.*"flag in the diff command (I found this fix was reported several years ago here: argocd 2.5: --server-side-generate fails with helm charts ? #11315 (reply in thread))The problem here is
--localuploads your local directory, but--local-includedefaults to*.{yaml,yml,json}, thecharts/directory is only partially uploaded, missing the_helpers.tpltemplate in particular.All this is relevant because I suspect when using the server side diff strategy, these
charts/directories are also being retained between diffs on the ArgoCD server itself.This issue (#711) seems to suggest ArgoCD uses
git clean -fdxbetween manifest generation. The-xbeing important sincegit clean -fdwouldn't clean up any files excluded by.gitignore(so if we're ignoring the charts directory they wouldn't be cleaned up).Maybe whats happened in my case is I've run
argocd app difflocally, uploading a partialcharts/directory which is then cached and isn't getting cleaned up by subsequent runs, breaking my ArgoCD server. (Might be wrong tho, assuming the user uploaded directory is cached in the same way ArgoCD caches git directories for an app...). The only other explanation is that ArgoCD isn't doinggit clean -fdxbetween runs.Expected behavior
Server-side diff whether run by ArgoCD or invoked by the
argocdCLI should not be affected by Kustomize'scharts/directories from previous invocations.Screenshots
Version