Increased reconciler errors about nonexistent offerings upon nodeclaim deletion

[micmathew]

Description

Observed Behavior:

The following reconciler error occurs very frequently across multiple nodepools:

attempted to remove nonexistent offering from nodepool "<nodepool_name>" (instance, "<instance_type>", zone, "", capacity, "spot")

Expected Behavior:

No reconciler errors with this message.

Reproduction Steps (Please include YAML):

I haven’t been able to reliably reproduce this with a consistent set of steps, but it occurs frequently in clusters with high churn (e.g., scaling up/down), and much less often in clusters with minimal churn.

Workaround: Patched the error to emit a warning instead to minimize alerts.

Versions:

• Chart Version: 1.10.0
• Kubernetes Version (kubectl version): 1.35

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jigisha620]

Hi @micmathew,
This is a known race condition in the cost reconciler's DeleteNodeClaim path that was fixed in v1.11.0 via #2944. The issue was that a read lock was released before a write lock was acquired, allowing concurrent deletions to remove the same offering between the existence check and the actual removal. Upgrading to v1.11.0 should resolve this. Let me know if you continue to see it after upgrading.

[jigisha620]

/triage accepted