Don't halt cluster-wide autoscaling when one node group TargetSize() fails - #10168
Don't halt cluster-wide autoscaling when one node group TargetSize() fails#10168yunluyl wants to merge 1 commit into
Conversation
…fails getTargetSizes used to discard the whole map on the first error, and UpdateNodes treated that as fatal. One broken node group then blocked scale-up/down for every other group. Skip failing groups, keep partial results, and only abort UpdateNodes when no target sizes are available.
|
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: yunluyl The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
This issue is currently awaiting triage. If SIG Autoscaling contributors determines this is a relevant issue, they will accept it by applying the The DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Welcome @yunluyl! |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Hi @yunluyl. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
What type of PR is this?
/kind bug
/area cluster-autoscaler
What this PR does / why we need it:
getTargetSizescurrently returns an empty map on the firstTargetSize()error, andUpdateNodestreats that as fatal. One broken node group then blocks scale-up and scale-down for every other node group until it is removed.This was hit in production on Azure/AKS when an AgentPool had no backing VMSS (
could not find vmss: ...on every reconcile) and froze the whole cluster for ~17.5 hours. The same path is provider-agnostic.This PR:
TargetSize()fails, returning partial resultsUpdateNodescontinue when at least one node group succeededUpdateNodesif every node group fails (nothing useful to scale)This matches the rest of CA (
Recalculate,ScaleUpToNodeGroupMinSize, and other directTargetSize()callers already log-and-continue).Related Azure-only draft: #10166. This change is the provider-agnostic fix for #10132.
Which issue(s) this PR fixes:
Fixes #10132
Special notes for your reviewer:
The snippet in #10132 still returned
errors.Join(...), whichUpdateNodeswould have treated as fatal and discarded the partial map. This PR also changesUpdateNodesto continue when the map is non-empty.Happy to cherry-pick to
cluster-autoscaler-release-1.30through1.36— the failing function is byte-identical across those branches.Does this PR introduce a user-facing change?
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:
Made with Cursor