Generate Kustomize overlays dynamically / reference a Kustomize base (Managing optional "overrides" for Kustomize Applications)

[VannTen]

Hello,

I'm using an ApplicationSet to deploy several application type, including Kustomize.
The way we designed the ApplicationSet allow for several level of overrides:

app_repo/
├── clusters
│   ├── aa
│   ├── bb
│   └── cc
├── prlvl
│   ├── pp
│   ├── pr
│   └── q
└── zones
    ├── east
    └── west

For Kustomize App, the Application spec template is roughly like this:

  template:
        metadata: {}
        spec:
          destination: {}
          project: default
          source:
            kustomize:
              components:
              - zones/{{ values.zone }}
              - prlvl/{{ values.prlvl }}
              - clusters/{{ values.cluster }}
              ignoreMissingComponents: true
            path: '{{ values.application }}'

So the target of kustomize build is the app root and all level of kustomization are components.

This mostly works, but today I stumbled upon a problem, which, IMO, comes from this scheme not following the spirit of kustomize.

In Kustomize, components are loaded/executed after resources but before any transformers (replacements, patches, etc) from what I understand.

This means that trying to use them as overrides fails if there is a conflict between the "root application" and the override.

For instance, I'm converting my argocd instance to be handled by that appset. Consider the following:

# app_repo/kustomization.yaml

[...]
resources:
- <argo upstream manifest>

[...]

patches:
  - target:
      kind: ConfigMap
      name: argocd-cm
    patch: |-
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cm
      data:
        dex.config: |
          connectors:
            - [ ... ]

# app_repo/clusters/aa/kustomization.yaml

patches:
  - target:
      kind: ConfigMap
      name: argocd-cm
    patch: |-
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cm
      data:
        dex.config: null # let's assume we want to delete the dex config only there and keeps it in all other instances.

This will result on the dex.config key not being overriden on the aa instance because the patches of the components is applied before the one in the kustomization.

Conceptually, I think the outermost "overrides" should instead be a kustomize overlay, where argocd would add components if needed, and referencing the "root application" as base.

However, creating the kustomization.yaml in each clusters directory is not practical and kinda defeat the purpose of optional overrides.

So, IMO, this would need some new capabilities in argocd Kustomize support, namely:

1. Generate an empty kustomization.yaml if missing (opt-in to avoid masking bugs/typo)
2. Ability to add a base to a kustomization.yaml (just like components)

The template above would then become

  template:
        metadata: {}
        spec:
          destination: {}
          project: default
          source:
            kustomize:
              components:
              - zones/{{ values.zone }}
              - prlvl/{{ values.prlvl }}
              ignoreMissingComponents: true
            path:  clusters/{{ values.cluster }}
            base: '{{ values.application }}'
            createKustomizationYamlIfMissing: true

This is all a bit fuzzy on the details, which is why I'm only opening a discussion. Especially, if this is already possible, I'd love to hear it !