Vitess operator trying to run vtbackup-init pods for old shards after resharding worfklow finished and old shard tablets still available before yaml apply to remove it

[GenLN]

Vitess latest operator 2.16.0 after reshard workflow completed, removing definition for old shards so it should be removed all vttablet pods of those, we get suddenly spawning vtbackup-init pods for old shards... operator stuck and not removing old shards because of those init pods trying to run.

{"level":"error","ts":"2026-01-23T10:58:03Z","msg":"Reconciler error","controller":"vitessshard-controller","object":{"name":"dev-vitess2-loadtest-c0-x-22f2ea3f","namespace":"vitess-test"},"namespace":"vitess-test","name":"dev-vitess2-loadtest-c0-x-22f2ea3f","reconcileID":"1af80deb-a82f-438b-aaec-28ef444974f9","error":"object is being deleted: pods \"dev-vitess2-loadtest-c0-x-vtbackup-init-c1bee57e\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:341\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:249"}
{"level":"info","ts":"2026-01-23T10:58:03Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"vitessshard-controller","object":{"name":"dev-vitess2-loadtest-c0-x-22f2ea3f","namespace":"vitess-test"},"namespace":"vitess-test","name":"dev-vitess2-loadtest-c0-x-22f2ea3f","reconcileID":"72271ce3-0eff-4e94-a6d3-0e4e82dae22a"}
{"level":"error","ts":"2026-01-23T10:58:03Z","msg":"Reconciler error","controller":"vitessshard-controller","object":{"name":"dev-vitess2-loadtest-c0-x-22f2ea3f","namespace":"vitess-test"},"namespace":"vitess-test","name":"dev-vitess2-loadtest-c0-x-22f2ea3f","reconcileID":"72271ce3-0eff-4e94-a6d3-0e4e82dae22a","error":"object is being deleted: pods \"dev-vitess2-loadtest-c0-x-vtbackup-init-c1bee57e\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:341\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:249"}
{"level":"info","ts":"2026-01-23T10:58:03Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"vitessshard-controller","object":{"name":"dev-vitess2-loadtest-c0-x-22f2ea3f","namespace":"vitess-test"},"namespace":"vitess-test","name":"dev-vitess2-loadtest-c0-x-22f2ea3f","reconcileID":"bfad7523-ac33-4603-8842-dc6290976ff2"}
{"level":"error","ts":"2026-01-23T10:58:03Z","msg":"Reconciler error","controller":"vitessshard-controller","object":{"name":"dev-vitess2-loadtest-c0-x-22f2ea3f","namespace":"vitess-test"},"namespace":"vitess-test","name":"dev-vitess2-loadtest-c0-x-22f2ea3f","reconcileID":"bfad7523-ac33-4603-8842-dc6290976ff2","error":"object is being deleted: pods \"dev-vitess2-loadtest-c0-x-vtbackup-init-c1bee57e\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:341\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:249"}```

[GenLN]

Also after old vttablets removed to fix serving metric (vttablet_is_in_srv_keyspace) we need to manually refresh tablet state of primary as metric return not serving state which is not true looking at vtadmin

[GenLN]

replication:
  initializeBackup: false

This one set in shard template stop spawning vtbackup init pods, but when old shards removed from yaml, operator doesn't delete those tablets... metric vttablet_is_in_srv_keyspace is showing old tablet primary as serving and new tablet primary as not serving... vtadmin sees and shows only new shards and shows all of them serving state.... refresh tablet state on new prims helps to update metric... but old shards are still there.....I'm not sure does this metric somehow affect operator reconcile or something?