Fleet rollout gets stuck in combination with Harvester. The original issue was moved to #4283.
This is the context provided by @ibrokethecloud in this ticket.
I suspect the issue is arising from the change in behaviour on when createOrUpdate bundledeployment fails
v0.13.0...v0.14.0-alpha.3#diff-38f599d4b6f84abae0fd9c6582e85b27f3f8a52774a565742e207ef52302274eL362
The controller now logs the error and continues with processing subsequent targets.
As a result of this for existing bundleDeployments their UID is no longer tracked in bundleDeploymentUIDs
https://github.com/rancher/fleet/blob/pkg/apis/v0.14.0-alpha.3/internal/cmd/controller/reconciler/bundle_controller.go#L388-L389
When the target loop finishes, the bundle controller reconciles orphaned bundles
https://github.com/rancher/fleet/blob/pkg/apis/v0.14.0-alpha.3/internal/cmd/controller/reconciler/bundle_controller.go#L409-L410
The cleanupOrphanedBundleDeployments skips bundles being tracked by bundleDeploymentUIDs, however due to error during create/update the existing bundleDeployment is not being tracked and is subsequently deleted before being re-created
Simple fix would be to revert back to v0.13.0 behaviour to return when createOrUpdate of bundledeployment fails.
From the logs we can see this behaviour
{"level":"info","ts":"2025-10-28T03:40:00Z","logger":"bundle","msg":"failed to create a bundledeployment, skipping and requeuing: failed to get content resource: Content.fleet.cattle.io \"s-73fdc1d64412860a11f29556ec85bb6519fe44054bfcff8e9ed6845719b53\" not found","controller":"bundle","controllerGroup":"fleet.cattle.io","controllerKind":"Bundle","Bundle":{"name":"mcc-harvester","namespace":"fleet-local"},"namespace":"fleet-local","name":"mcc-harvester","reconcileID":"26140e93-6314-470e-8df1-1c7ba31c2051","manifestID":"s-73fdc1d64412860a11f29556ec85bb6519fe44054bfcff8e9ed6845719b53","bundledeployment":"mcc-harvester"}
{"level":"error","ts":"2025-10-28T03:40:00Z","logger":"bundle","msg":"deleting orphaned bundle deployments","controller":"bundle","controllerGroup":"fleet.cattle.io","controllerKind":"Bundle","Bundle":{"name":"mcc-harvester","namespace":"fleet-local"},"namespace":"fleet-local","name":"mcc-harvester","reconcileID":"26140e93-6314-470e-8df1-1c7ba31c2051","manifestID":"s-73fdc1d64412860a11f29556ec85bb6519fe44054bfcff8e9ed6845719b53","bundle":"mcc-harvester","error":"admission webhook \"validator.harvesterhci.io\" denied the request: Delete bundledeployment cluster-fleet-local-local-1a3d67d0a899/mcc-harvester is prohibited","stacktrace":"github.com/rancher/fleet/internal/cmd/controller/reconciler.(*BundleReconciler).Reconcile\n\t/home/runner/_work/fleet/fleet/internal/cmd/controller/reconciler/bundle_controller.go:410\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:296"}
Fleet rollout gets stuck in combination with Harvester. The original issue was moved to #4283.
This is the context provided by @ibrokethecloud in this ticket.
I suspect the issue is arising from the change in behaviour on when
createOrUpdatebundledeployment failsv0.13.0...v0.14.0-alpha.3#diff-38f599d4b6f84abae0fd9c6582e85b27f3f8a52774a565742e207ef52302274eL362
The controller now logs the error and continues with processing subsequent targets.
As a result of this for existing
bundleDeploymentstheir UID is no longer tracked inbundleDeploymentUIDshttps://github.com/rancher/fleet/blob/pkg/apis/v0.14.0-alpha.3/internal/cmd/controller/reconciler/bundle_controller.go#L388-L389
When the target loop finishes, the bundle controller reconciles orphaned bundles
https://github.com/rancher/fleet/blob/pkg/apis/v0.14.0-alpha.3/internal/cmd/controller/reconciler/bundle_controller.go#L409-L410
The
cleanupOrphanedBundleDeploymentsskips bundles being tracked bybundleDeploymentUIDs, however due to error during create/update the existingbundleDeploymentis not being tracked and is subsequently deleted before being re-createdSimple fix would be to revert back to v0.13.0 behaviour to return when
createOrUpdateof bundledeployment fails.From the logs we can see this behaviour