fix: use container status resources over desired spec resources for cpu/memory resource metrics during in-place pod vertical scaling

[kondracek-nr]

Kubernetes 1.33 introduced in-place pod vertical scaling (beta, on by default; GA in 1.35). With this feature, Pod.Spec.Containers[i].Resources becomes the desired state rather than the actual state. The actual applied resources live in Pod.Status.ContainerStatuses[i].Resources, which is only populated once the kubelet has successfully enacted the allocation.

This means that during an active resize, cpuRequestedCores, memoryRequestedBytes, cpuLimitCores, and memoryLimitBytes — and the utilization ratios derived from them — were reporting the target values before they were applied to the running container.

Description

Change
In fetchContainersData, prefer Pod.Status.ContainerStatuses[i].Resources (actual applied state) over Pod.Spec.Containers[i].Resources (desired state), with a fallback to Spec when Status resources are nil. Sidecar init containers (RestartPolicy: Always) are handled the same way via Pod.Status.InitContainerStatuses.

Backward compatibility

• Pre-1.33 clusters: ContainerStatus.Resources is never populated, so the nil fallback always fires. No behavior change.
• 1.33+ clusters, no resize in progress: Spec and Status are identical. No behavior change.
• 1.33+ clusters, resize in progress: Values differ. We now report the currently-enforced allocation rather than the pending target.

Why change existing metric semantics vs. adding new metrics
Upstream kube-state-metrics is likely taking an additive approach (new kube_pod_container_actual_resource_* metrics) because changing existing metric semantics would silently break dashboards across their entire user base with no path to coordinate consumers.

Our situation is different: we control both the metrics and the dashboards that consume them, and cpuRequestedCores semantically means "what is currently being enforced on the container." Using the desired state for utilization calculations (cpuUsageCores / cpuRequestedCores) produces a ratio that doesn't reflect the container's actual resource envelope during a resize. The value change is also narrow in scope — it only occurs during the window between a resize being requested and the kubelet applying it.

Desired-state metrics (cpuRequestedCoresDesired etc.) and resize condition metrics (PodResizePending, PodResizeInProgress) are left as follow-up work, as is parity with the OTel collector chart (blocked on upstream KSM shipping the new actual-resource metrics).

Related

• Jira: NR-523430
• KSM tracking issue: Expose actual pod CPU/memory request from status.containerStatuses.resources (Kubernetes 1.33) kubernetes/kube-state-metrics#2665
• KSM PRs: #2702, #2773

Type of change

• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• New feature / enhancement (non-breaking change which adds functionality)
• Security fix
• Bug fix (non-breaking change which fixes an issue)

Checklist:

• Add changelog entry following the contributing guide
• Documentation has been updated
• This change requires changes in testing:
  • unit tests
  • E2E tests

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.94%. Comparing base (633dc0f) to head (ab7fbb1).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1433      +/-   ##
==========================================
+ Coverage   76.89%   76.94%   +0.04%     
==========================================
  Files          53       53              
  Lines        3787     3795       +8     
==========================================
+ Hits         2912     2920       +8     
  Misses        708      708              
  Partials      167      167              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.