[backport v2.14] HelmOp: Secret cloned to downstream namespace cannot be updated failing on immutable type field

[rancherbot]

This is a backport issue for #5252, automatically created via GitHub Actions workflow initiated by @p-se

Original issue body:

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Deploying a HelmOp resource that pulls an OCI chart from a private registry (e.g., dp.apps.rancher.io) and requires secrets on both the upstream (chart pull) and downstream (image pull) clusters causes the bundle controller to enter an infinite reconcile loop.

The fleet-controller logs the following error every ~5 seconds indefinitely:

INFO  bundle  requeue event: failed to clone secret fleet-default/rancher-registry-auth to downstream cluster namespace: failed to create or update source secret cluster-fleet-default-downstream1-32d08b529583/rancher-registry-auth: Secret "rancher-registry-auth" is invalid: type: Invalid value: "fleet.cattle.io/bundle-helmops-access/v1alpha1": field is immutable

From investigation, the secret is successfully created in the downstream cluster namespace on the first attempt. However, on subsequent reconcile cycles Fleet attempts to update the same value of the type field (fleet.cattle.io/bundle-helmops-access/v1alpha1). Kubernetes rejects this because a Secret's type field is immutable once set. The controller treats this as a retryable error and loops indefinitely.

Symptoms observed:

• BUNDLEDEPLOYMENTS-READY 0/0 on the HelmOp resource
• kubectl describe helmop kafka-app shows no status conditions (generation=1, observedGeneration=0)
• The secret copy exists in the downstream cluster namespace but the deployment never proceeds
• No error is surfaced via kubectl describe helmop, only visible in fleet-controller pod logs

Expected Behavior

Not sure about that. Fleet should successfully reconcile the HelmOp and deploy the Helm chart to the downstream cluster. Likely the secret cloned to the downstream cluster namespace should not have its type field modified after initial creation.

Steps To Reproduce

kubectl create secret docker-registry rancher-registry-auth
--docker-server=dp.apps.rancher.io
--docker-username=
--docker-password=
-n fleet-default

 2. ```bash
  kubectl create secret docker-registry application-collection \
    --docker-server=dp.apps.rancher.io \
    --docker-username=<user> \
    --docker-password=<token> \
    -n default

3. kubectl apply -f helmop_kafka.yaml

apiVersion: fleet.cattle.io/v1alpha1
  kind: HelmOp
  metadata:
    name: kafka-app
    namespace: fleet-default
  spec:
    helm:
      repo: oci://dp.apps.rancher.io/charts/apache-kafka
      chart: ""
      version: "1.3.2"
      releaseName: kafka
      values:
        replicas: 1
        global:
          imagePullSecrets:
            - application-collection
    helmSecretName: rancher-registry-auth
    downstreamResources:
      - kind: Secret
        name: application-collection
      - kind: Secret
        name: rancher-registry-auth
    targets:
      - name: foo
        clusterName: downstream1

4. observe helmop kafka-app
5. kubectl logs -n cattle-fleet-system deployment/fleet-controller -c fleet-controller

Environment

- Architecture: dev - commit 6992451
- Fleet Version: Multi-cluster
- Cluster: k3d 
  - Provider: k3d                                         
  - Options: NA
  - Kubernetes Version: v1.31.13+k3s1   
openSUSE, kernel 7.0.11-1-default

Logs

2026-06-08T15:43:41Z  INFO  bundle  requeue event: failed to clone secret fleet-default/rancher-registry-auth to downstream cluster namespace: failed to create or
  update source secret cluster-fleet-default-downstream1-32d08b529583/rancher-registry-auth: Secret "rancher-registry-auth" is invalid: type: Invalid value:
  "fleet.cattle.io/bundle-helmops-access/v1alpha1": field is immutable  {"controller": "bundle", "Bundle": {"name":"kafka-app","namespace":"fleet-default"}, ...}
  2026-06-08T15:43:46Z  DEBUG bundle  Reconciling bundle, checking targets, calculating changes, building objects  {"generation": 1, "observedGeneration": 0}
  2026-06-08T15:43:46Z  INFO  bundle  Unchanged bundledeployment  {"bundledeployment": "kafka-app", "deploymentID": ":8c95fa0a..."}
  2026-06-08T15:43:46Z  INFO  bundle  requeue event: failed to clone secret fleet-default/rancher-registry-auth ...
  [pattern repeats every ~5 seconds]

Anything else?

cattle-fleet-system-fleet-controller-79dd7888c9-thxcg-1780933602506710134.log

[sbulage]

System Information

Rancher Version	Fleet Version
v2.14-54f30b06da6420191628fbf8ea726738f971f032-head	0.15.4 (Custom built)

Verification

• Initially, I wasnot able to reproduce the issue clearly.

• Used @0xavi0's example to reproduce this issue.

  Details

  ---
  apiVersion: v1
  kind: Secret
  metadata:
    name: rancher-registry-auth
    namespace: fleet-default
  type: kubernetes.io/dockerconfigjson   # This bug will prevent to change the type of the secret.
  stringData:
    .dockerconfigjson: |
      {"auths":{"registry.example.com":{"username":"my-test-user","password":"pass","auth":"XXXXXXX"}}}
  ---
  apiVersion: fleet.cattle.io/v1alpha1
  kind: HelmOp
  metadata:
    name: test-clone-secret-issue-reproduce
    namespace: fleet-default
  spec:
    helmSecretName: rancher-registry-auth
  
    downstreamResources:
      - kind: Secret
        name: rancher-registry-auth
  
    helm:
      repo: https://stefanprodan.github.io/podinfo
      chart: podinfo
      version: 6.7.1
  
    targets:
      - clusterSelector: {}
  

• Need to create the secret before creating helmop

• After creating secrets first and then creating helmops, verified that:

  • No change in the type of secrets.
  • No error messages in fleet-controller container logs.

See below screenshots/logs for more details.

Fleet Controller logs error message

sbulage@localhost:~/fleet_gh/fleet> kubectl -n cattle-fleet-system logs deploy/fleet-controller -f | grep -i "immutable"
Defaulted container "fleet-controller" out of: fleet-controller, fleet-cleanup, fleet-agentmanagement
2026-06-26T13:50:25Z	INFO	bundle	requeue event: failed to clone secret fleet-default/rancher-registry-auth to downstream cluster namespace: failed to create or update source secret cluster-fleet-default-downstream1-32d08b529583/rancher-registry-auth: Secret "rancher-registry-auth" is invalid: type: Invalid value: "fleet.cattle.io/bundle-helmops-access/v1alpha1": field is immutable	{"controller": "bundle", "controllerGroup": "fleet.cattle.io", "controllerKind": "Bundle", "Bundle": {"name":"repro-5252","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "repro-5252", "reconcileID": "5aa88263-a299-4197-b646-c2bc7c3a2148", "manifestID": "", "bundledeployment": "repro-5252"}
2026-06-26T13:50:25Z	INFO	bundle	requeue event: failed to clone secret fleet-default/rancher-registry-auth to downstream cluster namespace: failed to create or update source secret cluster-fleet-default-downstream1-32d08b529583/rancher-registry-auth: Secret "rancher-registry-auth" is invalid: type: Invalid value: "fleet.cattle.io/bundle-helmops-access/v1alpha1": field is immutable	{"controller": "bundle", "controllerGroup": "fleet.cattle.io", "controllerKind": "Bundle", "Bundle": {"name":"repro-5252","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "repro-5252", "reconcileID": "db05adc8-cf3e-4df1-9a07-a62e029a2eb5", "manifestID": "", "bundledeployment": "repro-5252"}
2026-06-26T13:50:26Z	INFO	bundle	requeue event: failed to clone secret fleet-default/rancher-registry-auth to downstream cluster namespace: failed to create or update source secret cluster-fleet-default-downstream1-32d08b529583/rancher-registry-auth: Secret "rancher-registry-auth" is invalid: type: Invalid value: "fleet.cattle.io/bundle-helmops-access/v1alpha1": field is immutable	{"controller": "bundle", "controllerGroup": "fleet.cattle.io", "controllerKind": "Bundle", "Bundle": {"name":"repro-5252","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "repro-5252", "reconcileID": "6a9520f3-0808-4ddd-8d88-268a95bf896b", "manifestID": "", "bundledeployment": "repro-5252"}
2026-06-26T13:50:26Z	INFO	bundle	requeue event: failed to clone secret fleet-default/rancher-registry-auth to downstream cluster namespace: failed to create or update source secret cluster-fleet-default-downstream1-32d08b529583/rancher-registry-auth: Secret "rancher-registry-auth" is invalid: type: Invalid value: "fleet.cattle.io/bundle-helmops-access/v1alpha1": field is immutable	{"controller": "bundle", "controllerGroup": "fleet.cattle.io", "controllerKind": "Bundle", "Bundle": {"name":"repro-5252","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "repro-5252", "reconcileID": "fbd925ab-96f0-42aa-a9d5-85d06f496020", "manifestID": "", "bundledeployment": "repro-5252"}
2026-06-26T13:50:26Z	INFO	bundle	requeue event: failed to clone secret fleet-default/rancher-registry-auth to downstream cluster namespace: failed to create or update source secret cluster-fleet-default-downstream1-32d08b529583/rancher-registry-auth: Secret "rancher-registry-auth" is invalid: type: Invalid value: "fleet.cattle.io/bundle-helmops-access/v1alpha1": field is immutable	{"controller": "bundle", "controllerGroup": "fleet.cattle.io", "controllerKind": "Bundle", "Bundle": {"name":"repro-5252","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "repro-5252", "reconcileID": "392e4e07-d53b-413d-891d-a098737866ef", "manifestID": "", "bundledeployment": "repro-5252"}
2026-06-26T13:50:31Z	INFO	bundle	requeue event: failed to clone secret fleet-default/rancher-registry-auth to downstream cluster namespace: failed to create or update source secret cluster-fleet-default-downstream1-32d08b529583/rancher-registry-auth: Secret "rancher-registry-auth" is invalid: type: Invalid value: "fleet.cattle.io/bundle-helmops-access/v1alpha1": field is immutable	{"controller": "bundle", "controllerGroup": "fleet.cattle.io", "controllerKind": "Bundle", "Bundle": {"name":"repro-5252","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "repro-5252", "reconcileID": "feef06ad-1dd6-4658-b269-af96f44d1b9e", "manifestID": "", "bundledeployment": "repro-5252"}
2026-06-26T13:50:36Z	INFO	bundle	requeue event: failed to clone secret fleet-default/rancher-registry-auth to downstream cluster namespace: failed to create or update source secret cluster-fleet-default-downstream1-32d08b529583/rancher-registry-auth: Secret "rancher-registry-auth" is invalid: type: Invalid value: "fleet.cattle.io/bundle-helmops-access/v1alpha1": field is immutable	{"controller": "bundle", "controllerGroup": "fleet.cattle.io", "controllerKind": "Bundle", "Bundle": {"name":"repro-5252","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "repro-5252", "reconcileID": "196a5d23-1fd9-4393-98cb-e19ef3aa8589", "manifestID": "", "bundledeployment": "repro-5252"}

Screenshot showing HelmOp after upgrade, no error in the fleet-controller pod logs.