Kubernetes v1.35 Release Highlights

[graz-dev]

This is a discussion to collect the Release Highlights for the v1.35 release cycle.

Release Highlights are an essential part of the Kubernetes release cycle as they provide a birds-eye view of improvements, breaking or user-facing changes that are coming as part of the upcoming release. The release Comms team is asking for SIGs to comment on this discussion with items that we should highlight in release communications for v1.35.
For each submission, please include a link to the KEP, the stage the feature is moving to, and a brief description of the KEP that will be used as an outline for the blog post content.

The deadline for v1.35 Release Highlights is Friday 28th November.

Here is the discussion thread from the last release cycle.

cc @kubernetes/release-team-leads
cc @arujjval @chadmcrowell @aakankshabhende @SwathiR03

[neolit123]

hello, this is a SIG cluster lifecycle highlight for kubeadm related to this PR:

• vendor: update system-validators to v1.12.1 kubernetes#134744

the PR already has an action required release note, but additional signal via highlights could add more visitibility.
proposing this highlight text, please adjust as see fit or omit this highlight if you think the action required is sufficient!

SIG Cluster Lifecycle

With the release of 1.35, kubeadm will start validating the host cgroups and kubelet versions. It will throw a preflight error if cgroups v1 is detected on the host and if the detected target kubelet version is 1.35 or newer.

To allow cgroups v1 with kubeadm and kubelet version 1.35 or newer, you must:

• Ignore the error from the SystemVerifcation preflight check by kubeadm.
• Edit the kube-system/kubelet-config ConfigMap and add the failCgroupV1: false field, before upgrading.

This change is necessary after the deprecation of cgroups v1 in the kubelet for 1.35 and the feature being disabled by default starting this release.

cc @pacoxu

[pacoxu]

+1

[kannon92]

For sig-node, I'd like to mention deprecation of cgroup v1 (kubernetes/enhancements#5573).

The KEP is setting failCGroupV1 to true meaning that on cgroup v1 nodes kubelet will not start by default.

This means that nodes will not start on a cgroup v1 in our default behavior.
This is putting cgroup v1 into a deprecated state.

cc @haircommander @SergeyKanzhelev

I'm sure there are other items we should highlight for sig-node also though.

[SergeyKanzhelev]

1. The KEP Expose Node Topology Labels via Downward API enhancements#4742 will copy topology labels to all Pods. It is useful, but may be unexpected as everybody will see a lot of new labels on their Pods
2. Long standing requested feature of not marking pods not-ready during kubelet restart was implemented: Fix inconsistent container ready state after kubelet restart enhancements#4781. No more endpoints flaking (ready->unready->ready) during kubelet maintenance any longer.
3. ExecProbeTimeout Feature gate was locked to default. There is no more ways to revert to old behavior of ignoring timeouts for exec probes.
4. Kubernetes recommends to upgrade the runc (1.3.2+) and crun (1.23+) dependency (if used) with k8s 1.35 and cgroupv2 as we expect ecosystem adopting the new cpu.weight calculating formula to be used. Blog to be published: Blog post: New conversion from cgroup v1 CPU shares to v2 CPU weight website#52793 and issue: Conversion of cgroup v1 CPU shares to v2 CPU weight causes workloads to have low CPU priority kubernetes#131216.

I think those are worth mention in highlights. We will have way more things for release blog

[haircommander]

Also in addition to the cgroupv1 deprecation, 1.35 is the last release that will support containerd 1.x versions

[richabanker]

For sig-instrumentation,

1. Flagz for Kubernetes Components enhancements#4828 The /flagz endpoint now provides a versioned, structured JSON response, making flag information machine-readable across control plane components (kube-apiserver, kube-scheduler, kube-controller-manager, kubelet, kube-proxy) while retaining the plain-text format.
2. Statusz for Kubernetes Components enhancements#4827 The /statusz endpoint now has a structured v1alpha1 JSON response available, in addition to the plain-text format. This enables machine-readable status when the correct Accept header is used across control plane components (kube-apiserver, kube-scheduler, kube-controller-manager, kubelet, kube-proxy)

@pohly should we add something for Contextual Logging kubernetes/enhancements#3077 ?

[pohly]

No, it was just some incremental improvement.

[michaelasp]

Some sig-apimachinery highlights are

1. Move Storage Version Migrator in-tree enhancements#4192, StorageVersionMigration is now beta and can be enabled to run storage migration for GroupResources using supported APIs.
2. Comparable Resource Version enhancements#5504 Resource Versions are comparable client-side, meaning that clients are now able to compare objects of the same GroupResource together in terms of relative age. Conformance tests now include checks for resource version comparability of all K8s builtin API Objects and custom resources.

[lukasmetzner]

For SIG cloud provider,

I would like to mention KEP-5237 "CCM: watch-based route controller reconciliation using informers" (kubernetes/enhancements#5237).

We introduce an alpha-stage feature gate, CloudControllerManagerWatchBasedRoutesReconciliation, for k8s.io/cloud-provider. When enabled, it changes the cloud-controller-manager’s routes controller from a fixed-interval reconciliation loop to an informer-based mechanism. Reconciliation is triggered when nodes are added, removed, or when certain node properties change. This reduces the number of calls made to cloud provider APIs.

This might not be the full list of SIG cloud provider. I will clarify in the next biweekly, if anything else should be added.

[lukasmetzner]

@danwinship Correct me if I am wrong, but I think this should be a separate comment and is not related to KEP 5237 😅

[graz-dev]

Please @danwinship could you post it again as a separate comment?
Thankss

[danwinship]

oops, sorry, wrong text box 😬

[xing-yang]

Here are highlights from SIG Storage:

• Service Account Tokens via Secrets Field feature moved to Beta in v1.35. This feature adds CSI driver opt-in mechanism for receiving service account tokens via secrets field instead of volume context.
• Mutable Volume Attach Limits feature was enabled by default while staying in Beta in v1.35. This feature allows CSI drivers to dynamically adjust the maximum number of attachable volumes at runtime.

cc @saad-ali @msau42 @jsafrane

[sanposhiho]

We've done a lot, but I'd focus on new things as a highlight here.

SIG-Scheduling:

• Gang scheduling (alpha): It supports an alpha Workload API that supports the gang scheduling feature. This Workload would be a core initiative to make our scheduler more "workload aware". ...and other components too for a long term!
• Opportunistic batching (beta): It improves the performance to schedule similar pods. For now, we aim at ML workload cluster where there could be many similar workloads and a high performance scheduling is required, but we're going to expand it to wider use cases towards the future.
• Extended Toleration Operators for Threshold-Based Placement (alpha): newly supports Gt/Lt operators in tolerations for a numeric comparison.

cc @macsko @dom4ha @wojtek-t

[danwinship]

For SIG Network,

KEP-3015, PreferSameNode Traffic Distribution is now GA, which lets you tell kube-proxy that clients connecting to a Service (like DNS, for instance), should be redirected to an endpoint on the same node as the client whenever possible, to minimize latency.

[adrianmoisey]

KEP-5495 marks the ipvs kube-proxy mode as deprecated.
This is the first step of the eventual ipvs removal. The details and timeline of that are still being figured out in KEP-5344

[soltysh]

Highlights for SIG Apps:

• maxUnavailable for StatefulSets graduates to Beta. It allows setting the maximum number of pods that can be unavailable during the update.
• Promote DeploymentReplicaSetTerminatingReplicas to Beta. It exposes information about terminating replicas under .status.terminatingReplicas.
• Job API managed-by mechanism goes Stable. It allows using 3rd party controllers for managing the Job resource.
• Introduce Mutable Container Resources when Job is suspended. It allows controlling the resource requirements (CPU, memory, and GPU requests/limits) of a Job.

[soltysh]

Highlights for SIG CLI:

• kuberc allows setting credential plugin lists, which control which client-go credential plugins are allowed to execute.
• kubectl alpha kuberc sub-command provides handy helper to manage and view your kuberc file.
• KYAML output format goes beta. KYAML provides a new output format for kubectl, you can now use it by default specifying -okyaml.

[omerap12]

Highlights for SIG Autoscaling:

configurable hpa tolerance (beta), which allow users to optionally override the default workload autoscaling tolerance on a per-HPA bases.

/cc @kubernetes/sig-autoscaling-leads

[ritazh]

Highlights for SIG Auth:

• Pod Certificates (Beta) Graduates to Beta; enables workloads to request X.509 certificates directly from the API server
• Ensure secret pulled images (Beta) Graduates to Beta and enabled by default; kubelet tracks private image pulls to prevent incorrect pods from pulling images without the proper creds
• Constrained Impersonation (Alpha) New Alpha feature; introduces finer-grained controls for impersonation, so that any impersonator can impersonate in a more restricted way

/cc @kubernetes/sig-auth-leads

[liggitt]

Thanks for seeding!

kubernetes/enhancements#4872 New Alpha feature; improves security by validating the Common Name (CN) against the node name during API server connections to prevent rogue nodes from using certificates issued for IPs they do not own.

this one didn't actually land in 1.35...

---

kubernetes/enhancements#5284 (Alpha) New Alpha feature; introduces finer-grained controls for impersonation, so that any impersonator can impersonate in a more restricted way to prevent impersonators from performing any action that they couldn’t perform on their own.

Should we wait until beta to highlight?

---

kubernetes/enhancements#2535 going to beta and enabled by default should be highlighted, either by us or by sig-node (we were heavily involved in the design and implementation).

[ritazh]

removed 4872, added 2535, keeping alpha stuff since other sigs also included alpha things...

[graz-dev]

Hi all, the Release Announcement PR is open kubernetes/website#53498
It mentions all the KEPs listed in the discussion and some additions. Could you please review your respective KEPs for technical accuracy? Additionally, please let us know if anything else needs to be added to the KEPs section, particularly any specific changes that were delivered as part of v1.35.

cc @soltysh @xing-yang @aramase @macsko @SergeyKanzhelev
@ritazh @liggitt @omerap12 @adrianmoisey @danwinship @sanposhiho @lukasmetzner @michaelasp @pohly @richabanker @haircommander @kannon92 @pacoxu @neolit123