Agent installed using addon-framework not able to list the configmap on Hub cluster

[kkothule]

Hi Team,

We are using the Hosted control plane clusters
We deployed the addons using the addon-framework on the HCP cluster

Initially all the addons are healthy and working fine. We upgrade the framework from -0.11.0 -> 1.0.0

One of the addon on the spoke/HCP cluster is not able to list the configmap on the Hub which is working fine initially

W0114 11:01:36.558461       1 reflector.go:569] pkg/mod/k8s.io/client-go@v0.32.2/tools/cache/reflector.go:251: failed to list *v1.ConfigMap: Unauthorized
E0114 11:01:36.558504       1 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.32.2/tools/cache/reflector.go:251: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized"

below are the code snippet which are used to deploy the addon

func newAgentController(
	spokeKubeClient kubernetes.Interface,
	addonClient addonv1alpha1client.Interface,
	configmapInformers corev1informers.ConfigMapInformer,
	clusterName string,
	addonName string,
	addonNamespace string,
) factory.Controller {
	c := &agentController{
		spokeKubeClient:    spokeKubeClient,
		addonClient:        addonClient,
		clusterName:        clusterName,
		addonName:          addonName,
		addonNamespace:     addonNamespace,
		hubConfigMapLister: configmapInformers.Lister(),
		testInstall:      test.NewTestInstall(),
	}
	return factory.New().WithInformersQueueKeysFunc(
		func(obj runtime.Object) []string {
			key, _ := cache.MetaNamespaceKeyFunc(obj)
			return []string{key}
		}, configmapInformers.Informer()).
		WithSync(c.sync).ToController("agent-controller")
}

hubKubeInformerFactory := informers.NewSharedInformerFactoryWithOptions(hubKubeClient, 10*time.Minute, informers.WithNamespace(o.SpokeClusterName))

	// create an agent controller
	agent := newAgentController(
		spokeKubeClient,
		addonClient,
		hubKubeInformerFactory.Core().V1().ConfigMaps(),
		o.SpokeClusterName,
		o.AddonName,
		o.AddonNamespace,
	)

All the addons are deployed in open-cluster-management-agent-addon

[1:12 PM]kk ~/tmp  $ kubectl get secret -n open-cluster-management-agent-addon
NAME                                                                      TYPE                             DATA   AGE
builder-dockercfg-2prp6                                                   kubernetes.io/dockercfg          1      6d16h
cluster-proxy-ca                                                          Opaque                           1      6d16h
cluster-proxy-dockercfg-h7h4x                                             kubernetes.io/dockercfg          1      6d16h
cluster-proxy-hub-kubeconfig                                              Opaque                           4      6d16h
cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert   Opaque                           3      6d16h
cluster-proxy-service-proxy-server-certificates                           Opaque                           2      6d16h
default-dockercfg-k5x7z                                                   kubernetes.io/dockercfg          1      6d16h
deployer-dockercfg-x7nsj                                                  kubernetes.io/dockercfg          1      6d16h
agent-sa-dockercfg-2bgjb                                           kubernetes.io/dockercfg          1      2d21h
backup-hub-kubeconfig                                              Opaque                           4      2d20h
backupagent-sa-dockercfg-h7q6l                                     kubernetes.io/dockercfg          1      6d16h
base-hub-kubeconfig                                                Opaque                           4      2d20h
klusterlet-addon-workmgr-dockercfg-6vz72                                  kubernetes.io/dockercfg          1      6d16h
klusterlet-addon-workmgr-log-dockercfg-29rzl                              kubernetes.io/dockercfg          1      6d16h
managed-serviceaccount-dockercfg-wcxzh                                    kubernetes.io/dockercfg          1      6d16h
managed-serviceaccount-hub-kubeconfig                                     Opaque                           4      6d16h
open-cluster-management-image-pull-credentials                            kubernetes.io/dockerconfigjson   1      6d16h
work-manager-hub-kubeconfig

The hub config is in place

base-hub-kubeconfig                                                Opaque                           4      2d20h

Agent spec deployed

kind: Pod
apiVersion: v1
metadata:
  generateName: agent-7656bddd86-
  annotations:
    k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.1.166/23"],"mac_address":"0a:58:0a:84:01:a6","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.65.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.1.166/23","gateway_ip":"10.132.0.1","role":"primary"}}'
    k8s.v1.cni.cncf.io/network-status: |-
      [{
          "name": "ovn-kubernetes",
          "interface": "eth0",
          "ips": [
              "10.132.1.166"
          ],
          "mac": "0a:58:0a:84:01:a6",
          "default": true,
          "dns": {}
      }]
    openshift.io/scc: nonroot-v2
    seccomp.security.alpha.kubernetes.io/pod: runtime/default
  resourceVersion: '2901397'
  name: agent-7656bddd86-swhht
  uid: 6e1fd1af-b4a9-4191-92c4-1f399a51fbdf
  creationTimestamp: '2026-01-16T09:49:12Z'
  managedFields:
    - manager: kube-controller-manager
      operation: Update
      apiVersion: v1
      time: '2026-01-16T09:49:12Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:generateName': {}
          'f:labels':
            .: {}
            'f:app': {}
            'f:pod-template-hash': {}
          'f:ownerReferences':
            .: {}
            'k:{"uid":"433feeba-ca88-4dbd-9922-d197c0e903cf"}': {}
        'f:spec':
          'f:volumes':
            .: {}
            'k:{"name":"hub-config"}':
              .: {}
              'f:name': {}
              'f:secret':
                .: {}
                'f:defaultMode': {}
                'f:secretName': {}
          'f:containers':
            'k:{"name":"agent"}':
              'f:image': {}
              'f:volumeMounts':
                .: {}
                'k:{"mountPath":"/var/run/hub"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
              'f:terminationMessagePolicy': {}
              .: {}
              'f:resources':
                .: {}
                'f:limits':
                  .: {}
                  'f:cpu': {}
                  'f:ephemeral-storage': {}
                  'f:memory': {}
                'f:requests':
                  .: {}
                  'f:cpu': {}
                  'f:ephemeral-storage': {}
                  'f:memory': {}
              'f:args': {}
              'f:securityContext':
                .: {}
                'f:allowPrivilegeEscalation': {}
                'f:capabilities':
                  .: {}
                  'f:drop': {}
                'f:privileged': {}
                'f:readOnlyRootFilesystem': {}
                'f:runAsNonRoot': {}
                'f:runAsUser': {}
                'f:seccompProfile':
                  .: {}
                  'f:type': {}
              'f:terminationMessagePath': {}
              'f:imagePullPolicy': {}
              'f:name': {}
          'f:dnsPolicy': {}
          'f:serviceAccount': {}
          'f:restartPolicy': {}
          'f:schedulerName': {}
          'f:terminationGracePeriodSeconds': {}
          'f:serviceAccountName': {}
          'f:enableServiceLinks': {}
          'f:securityContext':
            .: {}
            'f:runAsNonRoot': {}
    - manager: np-1-g8l4b-jvs65
      operation: Update
      apiVersion: v1
      time: '2026-01-16T09:49:12Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:k8s.ovn.org/pod-networks': {}
      subresource: status
    - manager: multus-daemon
      operation: Update
      apiVersion: v1
      time: '2026-01-16T09:49:13Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:k8s.v1.cni.cncf.io/network-status': {}
      subresource: status
    - manager: kubelet
      operation: Update
      apiVersion: v1
      time: '2026-01-16T09:49:18Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          'f:conditions':
            'k:{"type":"ContainersReady"}':
              .: {}
              'f:lastProbeTime': {}
              'f:lastTransitionTime': {}
              'f:status': {}
              'f:type': {}
            'k:{"type":"Initialized"}':
              .: {}
              'f:lastProbeTime': {}
              'f:lastTransitionTime': {}
              'f:status': {}
              'f:type': {}
            'k:{"type":"PodReadyToStartContainers"}':
              .: {}
              'f:lastProbeTime': {}
              'f:lastTransitionTime': {}
              'f:status': {}
              'f:type': {}
            'k:{"type":"Ready"}':
              .: {}
              'f:lastProbeTime': {}
              'f:lastTransitionTime': {}
              'f:status': {}
              'f:type': {}
          'f:containerStatuses': {}
          'f:hostIP': {}
          'f:hostIPs': {}
          'f:phase': {}
          'f:podIP': {}
          'f:podIPs':
            .: {}
            'k:{"ip":"10.132.1.166"}':
              .: {}
              'f:ip': {}
          'f:startTime': {}
      subresource: status
  namespace: open-cluster-management-agent-addon
  ownerReferences:
    - apiVersion: apps/v1
      kind: ReplicaSet
      name: agent-7656bddd86
      uid: 433feeba-ca88-4dbd-9922-d197c0e903cf
      controller: true
      blockOwnerDeletion: true
  labels:
    app: agent
    pod-template-hash: 7656bddd86
spec:
  restartPolicy: Always
  serviceAccountName: agent-sa
  priority: 0
  schedulerName: default-scheduler
  enableServiceLinks: true
  terminationGracePeriodSeconds: 30
  preemptionPolicy: PreemptLowerPriority
  nodeName: np-1-g8l4b-jvs65
  securityContext:
    seLinuxOptions:
      level: 's0:c6,c5'
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
    - resources:
        limits:
          cpu: '1'
          ephemeral-storage: 500Mi
          memory: 500Mi
        requests:
          cpu: 200m
          ephemeral-storage: 200Mi
          memory: 128Mi
      terminationMessagePath: /dev/termination-log
      name: agent
      securityContext:
        capabilities:
          drop:
            - ALL
        privileged: false
        runAsUser: 1000
        runAsNonRoot: true
        readOnlyRootFilesystem: false
        allowPrivilegeEscalation: false
        seccompProfile:
          type: RuntimeDefault
      imagePullPolicy: Always
      volumeMounts:
        - name: hub-config
          mountPath: /var/run/hub
        - name: kube-api-access-cshrw
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      terminationMessagePolicy: File
      image: 'hcp-manager:0.0.1'
      args:
        - base
  serviceAccount: agent-sa
  volumes:
    - name: hub-config
      secret:
        secretName: base-hub-kubeconfig
        defaultMode: 420
    - name: kube-api-access-cshrw
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
          - configMap:
              name: openshift-service-ca.crt
              items:
                - key: service-ca.crt
                  path: service-ca.crt
        defaultMode: 420
  dnsPolicy: ClusterFirst
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/memory-pressure
      operator: Exists
      effect: NoSchedule
status:
  containerStatuses:
    - restartCount: 0
      started: true
      ready: true
      name: agent
      state:
        running:
          startedAt: '2026-01-16T09:49:19Z'
      volumeMounts:
        - name: hub-config
          mountPath: /var/run/hub
        - name: kube-api-access-cshrw
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          readOnly: true
          recursiveReadOnly: Disabled
      imageID: 'hcp-manager@sha256:adcff828d2a866ce209d6f003e361ce3441164f76e9982c32f21ff57471c77f6'
      image: 'hcp-manager:0.0.1'
      lastState: {}
      containerID: 'cri-o://465d3049e3014ac32b3c1f400d3b4fd9b1c5fa1c4e9bbdf013f1a83cc4091f1b'
  qosClass: Burstable
  hostIPs:
    - ip: 10.129.2.40
  podIPs:
    - ip: 10.132.1.166
  podIP: 10.132.1.166
  hostIP: 10.129.2.40
  startTime: '2026-01-16T09:49:13Z'
  conditions:
    - type: PodReadyToStartContainers
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2026-01-16T09:49:20Z'
    - type: Initialized
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2026-01-16T09:49:13Z'
    - type: Ready
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2026-01-16T09:49:20Z'
    - type: ContainersReady
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2026-01-16T09:49:20Z'
    - type: PodScheduled
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2026-01-16T09:49:12Z'
  phase: Running

[zhujian7]

Hi @kkothule , the addon-framework does not automatically grant permissions to access ConfigMaps on the hub cluster. If your agent needs this permission, you need to set the RegistrationOption.PermissionConfig to create the necessary RBAC resources.

Please see the example here: https://github.com/open-cluster-management-io/addon-framework/blob/main/examples/helloworld/helloworld.go#L31

The RBAC implementation example is here: https://github.com/open-cluster-management-io/addon-framework/blob/main/examples/rbac/rbac.go

If you're able to share your code, I'd be happy to help review it.

[kkothule]

@zhujian7 We have the AddonRBAC configured

func AddonRBAC(kubeConfig *rest.Config) agent.PermissionConfigFunc {
	return func(cluster *clusterv1.ManagedCluster, addon *addonapiv1alpha1.ManagedClusterAddOn) error {
		kubeclient, err := kubernetes.NewForConfig(kubeConfig)
		if err != nil {
			return err
		}

		groups := agent.DefaultGroups(cluster.Name, addon.Name)

		role := &rbacv1.Role{
			ObjectMeta: metav1.ObjectMeta{
				Name:      fmt.Sprintf("open-cluster-management:%s:agent", addon.Name),
				Namespace: cluster.Name,
			},
			Rules: []rbacv1.PolicyRule{
				{Verbs: []string{"get", "list", "watch"}, Resources: []string{"configmaps"}, APIGroups: []string{""}},
				{Verbs: []string{"get", "list", "watch"}, Resources: []string{"secrets"}, APIGroups: []string{""}},
				{Verbs: []string{"get", "list", "watch"}, Resources: []string{"managedclusteraddons"}, APIGroups: []string{"addon.open-cluster-management.io"}},
			},
		}

		binding := &rbacv1.RoleBinding{
			ObjectMeta: metav1.ObjectMeta{
				Name:      fmt.Sprintf("open-cluster-management:%s:agent", addon.Name),
				Namespace: cluster.Name,
			},
			RoleRef: rbacv1.RoleRef{
				APIGroup: "rbac.authorization.k8s.io",
				Kind:     "Role",
				Name:     fmt.Sprintf("open-cluster-management:%s:agent", addon.Name),
			},
			Subjects: []rbacv1.Subject{
				{Kind: "Group", APIGroup: "rbac.authorization.k8s.io", Name: groups[0]},
			},
		}

		_, err = kubeclient.RbacV1().Roles(cluster.Name).Get(context.TODO(), role.Name, metav1.GetOptions{})
		switch {
		case errors.IsNotFound(err):
			_, createErr := kubeclient.RbacV1().Roles(cluster.Name).Create(context.TODO(), role, metav1.CreateOptions{})
			if createErr != nil {
				return createErr
			}
		case err != nil:
			return err
		}

		_, err = kubeclient.RbacV1().RoleBindings(cluster.Name).Get(context.TODO(), binding.Name, metav1.GetOptions{})
		switch {
		case errors.IsNotFound(err):
			_, createErr := kubeclient.RbacV1().RoleBindings(cluster.Name).Create(context.TODO(), binding, metav1.CreateOptions{})
			if createErr != nil {
				return createErr
			}
		case err != nil:
			return err
		}

		return nil
	}
}

func NewRegistrationOption(kubeConfig *rest.Config, addonName, agentName string) *agent.RegistrationOption {
	return &agent.RegistrationOption{
		CSRConfigurations: agent.KubeClientSignerConfigurations(addonName, agentName),
		CSRApproveCheck:   utils.DefaultCSRApprover(agentName),
		PermissionConfig:  AddonRBAC(kubeConfig),
		Namespace:         apis.InstallationNamespace,
	}
}

[zhujian7]

It looks ok. Could you check the addon manager logs on the hub cluster? and check if the roles & rolebindins you defined in the AddonRBAC are created successfully on the hub?

[kkothule]

@zhujian7 , We tried deleting the base-hub-kubeconfig on spoke cluster, But it is not getting recreated

[zhujian7]

ok, Please check the addon manager logs on the hub cluster in the open-cluster-management-hub ns and the klusterlet-agent(or registration-agent) logs on the managed cluster in the open-cluster-management-agent ns.