installNamespace not respected when using AddonTemplate

[arturshadnik]

I am trying to create and deploy a custom addon using the AddonTemplate API and clusteradm. My current flow is:

• Create 2 KinD clusters
• Join them in a hub and spoke configuration using clusteradm init and clusteradm join
• Run clusteradm addon create --name=test-addon --version=v0.3.0 --filename=<path-to-manifests>, using the following manifests:

apiVersion: apps/v1                                                                                                                                                                                                                                                                       
kind: Deployment                                                                                                                                                                                                                                                                          
metadata:                                                                                                                                                                                                                                                                                 
  name: test-addon                                                                                                                                                                                                                                                                 
spec:                                                                                                                                                                                                                                                                                     
  replicas: 1                                                                                                                                                                                                                                                                             
  selector:                                                                                                                                                                                                                                                                               
    matchLabels:                                                                                                                                                                                                                                                                          
      app: test-addon                                                                                                                                                                                                                                                               
  template:                                                                                                                                                                                                                                                                               
    metadata:                                                                                                                                                                                                                                                                             
      labels:                                                                                                                                                                                                                                                                             
        app: test-addon                                                                                                                                                                                                                                                           
    spec:                                                                                                                                                                                                                                                                                 
      containers:                                                                                                                                                                                                                                                                         
      - image: nginx:latest                                                                                                                                                                                                                                                               
        imagePullPolicy: IfNotPresent                                                                                                                                                                                                                                                     
        name: test-addon                                                                                                                                                                                                                                                             
        ports:                                                                                                                                                                                                                                                                            
        - containerPort: 8080

This creates an AddonTemplate containing the manifests, and ClusterManagementAddon with the following spec:

spec:                                                                                                                                                                                                                                                                                             
  addOnConfiguration: {}                                                                                                                                                                                                                                                                          
  addOnMeta: {}                                                                                                                                                                                                                                                                                   
  installStrategy:                                                                                                                                                                                                                                                                                
    type: Manual                                                                                                                                                                                                                                                                                  
  supportedConfigs:                                                                                                                                                                                                                                                                               
  - defaultConfig:                                                                                                                                                                                                                                                                                
      name: test-addon-v0.3.0                                                                                                                                                                                                                                                                     
    group: addon.open-cluster-management.io                                                                                                                                                                                                                                                       
    resource: addontemplates

• Run clusteradm addon enable --names=test-addon --cluster=spoke --namespace=test (tried both without and with the namespace being created ahead of time)
• A ManagedClusterAddon is created with as follows (note that spec.installNamespace and status.ConfigReference are correctly set).

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  creationTimestamp: "2025-07-24T16:12:29Z"
  generation: 1
  name: test-addon
  namespace: spoke
  ownerReferences:
  - apiVersion: addon.open-cluster-management.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ClusterManagementAddOn
    name: test-addon
    uid: e1e12e89-f353-49d7-b5f7-1ce113fa88c4
  resourceVersion: "14436"
  uid: 28345f09-cadd-4aec-a6de-42860de55806
spec:
  installNamespace: test
status:
  conditions:
  - lastTransitionTime: "2025-07-24T16:12:29Z"
    message: Configurations configured
    reason: ConfigurationsConfigured
    status: "True"
    type: Configured
  - lastTransitionTime: "2025-07-24T16:12:29Z"
    message: progressing... work is not ready
    reason: Progressing
    status: "True"
    type: Progressing
  - lastTransitionTime: "2025-07-24T16:12:29Z"
    message: Registration of the addon agent is configured
    reason: SetPermissionApplied
    status: "True"
    type: RegistrationApplied
  - lastTransitionTime: "2025-07-24T16:12:29Z"
    message: failed to apply the manifests of addon
    reason: AddonManifestAppliedFailed
    status: "False"
    type: ManifestApplied
  - lastTransitionTime: "2025-07-24T16:12:29Z"
    message: 1 of 1 resources are not available
    reason: WorkNotApplied
    status: "False"
    type: Available
  configReferences:
  - desiredConfig:
      name: test-addon-v0.3.0
      specHash: 2bfdf0baf5a3a3c72c18df65d5fba65d6b0e9e84388347e9fba0ef4b7845d3ff
    group: addon.open-cluster-management.io
    lastObservedGeneration: 1
    name: test-addon-v0.3.0
    resource: addontemplates
  healthCheck:
    mode: Customized
  namespace: open-cluster-management-agent-addon
  supportedConfigs:
  - group: addon.open-cluster-management.io
    resource: addondeploymentconfigs
  - group: addon.open-cluster-management.io
    resource: addontemplates

After this, the Deployment never comes up on the spoke. klusterlet-work-agent repeatedly logs the following:

I0724 16:34:20.565061       1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"open-cluster-management-agent", Name:"klusterlet-work-agent", UID:"a86ef894-5076-494f-bdcc-87dcbe6ce799", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'Dep
loyment Created' Created /test-addon-3 because it was missing                                                                                                                                                                                                                                     
E0724 16:34:20.566399       1 manifestwork_controller.go:171] Reconcile work addon-test-addon-deploy-0 fails with err: [the server does not allow this method on the requested resource]                                                                                                          
E0724 16:34:20.566918       1 base_controller.go:277] "Unhandled Error" err="\"ManifestWorkAgent\" controller failed to sync \"addon-test-addon-deploy-0\", err: the server does not allow this method on the requested resource"

On the hub, the cluster-manager-addon-manager-controller logs the following:

I0724 16:41:16.863442       1 addon_config.go:56] "Addon deployment config is nil, return an empty string for agent install namespace" addonNamespace="spoke" addonName="test-addon"

This suggests that the namespace is defaulting to open-cluster-management-agent-addon.

When i try with the following manifests:

apiVersion: v1
kind: Namespace
metadata:
  name: test
---
apiVersion: apps/v1                                                                                                                                                                                                                                                                       
kind: Deployment                                                                                                                                                                                                                                                                          
metadata:                                                                                                                                                                                                                                                                                 
  name: test-addon
  namespace: test                                                                                                                                                                                                                                                                      
spec:                                                                                                                                                                                                                                                                                     
  replicas: 1                                                                                                                                                                                                                                                                             
  selector:                                                                                                                                                                                                                                                                               
    matchLabels:                                                                                                                                                                                                                                                                          
      app: test-addon                                                                                                                                                                                                                                                                   
  template:                                                                                                                                                                                                                                                                               
    metadata:                                                                                                                                                                                                                                                                             
      labels:                                                                                                                                                                                                                                                                             
        app: test-addon                                                                                                                                                                                                                                                              
    spec:                                                                                                                                                                                                                                                                                 
      containers:                                                                                                                                                                                                                                                                         
      - image: nginx:latest                                                                                                                                                                                                                                                               
        imagePullPolicy: IfNotPresent                                                                                                                                                                                                                                                     
        name: test-addon                                                                                                                                                                                                                                                              
        ports:                                                                                                                                                                                                                                                                            
        - containerPort: 8080

Both the namespace and deployment are created successfully, despite the cluster-manager-addon-manager-controller still logging the same message as before:

k get deploy -n test
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
test-addon   1/1     1            1           59s

I0724 16:41:16.863442       1 addon_config.go:56] "Addon deployment config is nil, return an empty string for agent install namespace" addonNamespace="spoke" addonName="test-addon"

I also noticed that some of the sample addons (eg. https://github.com/open-cluster-management-io/addon-contrib/blob/main/argocd-agent-addon/charts/argocd-addon/templates/argocd-addon-template.yaml#L6868) bundle the namespace with the manifests.

What I would like clarified:
Is this the only way to target a specific namespace when using the AddonTemplate API?
Why does the klusterler-work-agent fail to create a Deployment unless the namespace is hardcoded in the manifests, and created alongside the deployment?
Is the expectation that even when using clusteradm addon create and enable, an AddonDeploymentConfig needs to be created out-of-band?
Why do both AddonDeploymentConfig and ManagedClusterAddOn expose a namespace field? Which one takes precedence? How are either of them actually configured in the ManifestWork (how does the klusterler-work-agent know which namespace the manifests should be installed into)

---

clusteradm version
client          version :v1.0.0-0-g1e1db3e
server release  version :v1.33.1
default bundle  version :1.0.0

kind version
kind v0.29.0 go1.24.4 darwin/arm64

[arturshadnik]

Additional note:
I have also tried to create an AddonDeploymentConfig with spec.AgentInstallNamespace, and reference this config inside of my ManagedClusterAddOn spec. This made the addon manager errors stop, but still the addon was not installed inside the spoke cluster, with klusterlet-work-agent again reporting the [the server does not allow this method on the requested resource] error.

[zhujian7]

Hi @arturshadnik thanks for reporting this!

Currently, the template type addons ignore the managedclusteraddon.spec.installNamespace, because we are going to deprecate the managedclusteraddon.spec.installNamespace, some reasons see here

But the AddonDeploymentConfig.spec.AgentInstallNamespace should work, the error you encountered the server does not allow this method on the requested resource seems related to the topic work agent permission, see here, you can try to grant the work-agent the permissions that it needs.

[zhujian7]

And currently, the clusteradm addon enable --names=test-addon --cluster=spoke --namespace=test namespace does not work for the template-type addons, I think it is a bug, I am not sure if we should create an addonDeploymentConfig to set the installNamespace for template-type addons, @qiujian16 WDYT?

/type bug

[qiujian16]

yes It is a bug in clusteradm actually. I think we should update clusteradm so it always use addonDeploymentConfig to set the install namespace rather than the managedclusteraddon spec.

/kind bug

[openshift-ci]

@qiujian16: The label(s) kind/bug cannot be applied, because the repository doesn't have them.

Details

In response to this:

yes It is a bug in clusteradm actually.

/kind bug

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[arturshadnik]

Got it, thank you! I already opened a related ticket in clusteradm open-cluster-management-io/clusteradm#501. Will check the link you provided re. RBAC issues.

[TylerGillson]

Regarding the RBAC issues, the logs would ideally indicate clearly which group / resource / verb was denied.

These logs are not actionable:

E0724 16:34:20.566399       1 manifestwork_controller.go:171] Reconcile work addon-test-addon-deploy-0 fails with err: [the server does not allow this method on the requested resource]                                                                                                          
E0724 16:34:20.566918       1 base_controller.go:277] "Unhandled Error" err="\"ManifestWorkAgent\" controller failed to sync \"addon-test-addon-deploy-0\", err: the server does not allow this method on the requested resource"

[zhujian7]

@TylerGillson, please check the status of the addon-test-addon-deploy-0 manifestwork in the namespace on the hub cluster, it should show the resources.