Orphaned Bundledeployment Lifecycle Secrets Block Installation

[manno]

I am installing 50 helm-style bundles, deleting the gitrepo and re-creating the gitrepo. The second time the installation will get stuck, unless I wait for all lifecycle secrets to disappear. I am using Fleet v0.13.2-rc.2. I am testing with 500 clusters (25k bundledeployments&secrets).

kubectl apply -n fleet-default -f - <<EOF
---
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
  name: scale-50-single
spec:
  repo: https://github.com/manno/fleet-experiments
  branch: main
  paths:
    - scale-50-bundles/ten
    - scale-50-bundles/twenty
    - scale-50-bundles/thirty
    - scale-50-bundles/fourty
    - scale-50-bundles/fifty
  targetNamespace: scale-50-single
  targets:
    - clusterSelector: {}
EOF

After removing a GitRepo:

• the gitrepo is removed
• fleet-controller removes finalizers from content resources, then it removes content resources
  • there are lot's of conflicts in the logs, but it works.
  • example only creates three (!) content resources total (s-9d774f37b4a... is used by 48 bundledeployments)
  • fleet-controller-76cb85cf48-qshmp fleet-controller {"level":"error","msg":"Reconciler error","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"scale-50-single-scale-50-bundles-twenty-seventeen","namespace":"cluster-fleet-default-d0-k3k-downstream002-downstream0161-5314b"},"namespace":"cluster-fleet-default-d0-k3k-downstream002-downstream0161-5314b","name":"scale-50-single-scale-50-bundles-twenty-seventeen","error":"Content.fleet.cattle.io \"s-9d774f37b4a7562f0f91c3acd6409fdee41209cb35d55f76aa1f381ebbb1f\" not found",...}
• fleet-controller removes finalizers from bundledeployments, then it removes bundledeployments
• fleet-controller removes finalizer from bundles, then bundles are removed
• lifecycle secrets are left behind so that k8s garbage collection can remove them (orphaned owner?)
  • kubectl get secrets -A --field-selector type=fleet.cattle.io/bundle-deployment/v1alpha1
  • this takes easily 30 minutes for 25000 secrets

Ideas:

• Installation should deal with existing lifecycle secrets and update the owner?
• Deinstallation/clean up should not rely on k8s garbage selection
• Removing finalizers should not print that many errors
• GitRepo finalizer should not be removed until deletion cascade (bundles, bundledeployments) are gone

[weyfonk]

Created #4192 for remaining work, to be done as a follow-up. Closing this.