Kubernetes v1.33 Release Highlights

[rytswd]

This is a discussion to collect the Release Highlights for the 1.33 release cycle.

Release Highlights are an essential part of the Kubernetes release cycle as they provide a birds-eye view of improvements, breaking or user-facing changes that are coming as part of the upcoming release. The release Comms team asks all the SIGs to comment on this discussion ticket with items that we should highlight in the release communications for 1.33.
For each submission, please include a link to KEP, the stage the feature is moving to, and a brief description of KEP that will be used as an outline for the blog post content.

The deadline for 1.33 Release Highlights is Tuesday, 25th March 2025.

Note

Some references for the previous releases:

• 1.32 discussion thread
• 1.31 discussion thread

cc @kubernetes/release-team-leads
cc @aakankshabhende @aibarbetta @Udi-Hofesh @sn3hay

[jm-franc]

SIG Autoscaling (FYI @gjtempleton):

• Configurable tolerance for Horizontal Pod Autoscalers to dampen scaling reactions to small metric variations (blog post). (Alpha feature.)

[marosset]

SIG Windows

• Promoting DSR support in Windows kube-proxy to beta KEP #5100
• Withdrawing HostNetwork support for Windows pods (previously alpha) KEP #3503
• Windows unit tests have been cleaned up and are now running against all kubelet PRs in k/k Issue #130149

[marosset]

cc @aravindhp @jsturtevant

[aojea]

SIG Network Highlights:

• Dynamic Resource Allocation (DRA) for Network Interfaces reaches Beta KEP-4817:
  • The standardized reporting of network interface data via DRA is now in beta. This enables more native Kubernetes network integrations, simplifying the development and management of networking devices.
• Multiple Service CIDRs graduates to GA KEP-1880 :
  • The new Services ClusterIP address allocator, based on IPAddress objects, is now GA and removes previous allocator limitations, allowing for dynamic service CIDR configuration and expanding the scope for network integrations using the new IPAddress API objects.
• kube-proxy nftables Backend is now GA KEP-3866:
  • The nftables backend for kube-proxy is now GA, adding a new implementation that significantly improves performance and scalability for Services implementation within Kubernetes clusters.
• Topology Aware Routing using trafficDistribution: PreferClose Graduates to GA (KEP-4444, KEP-2433)
  • The Service trafficDistribution field, enabling topology aware routing with the PreferClose option, graduates to GA. This provides a stable and standardized way for users to express their preference for routing traffic to endpoints within the same zone. This GA status also signifies the stabilization of EndpointSlice Hints (KEP-2433), which are leveraged by trafficDistribution to achieve this topology-aware routing functionality.

[danwinship]

I don't think we can say that nftables kube-proxy is "replacing" iptables, since it's not the default, for either old or new users.

[liam-mackie]

We also have an in-progress blog post that would work well for topology aware routing going GA, if you're open to considering it?

[aojea]

@danwinship yeah, now I read it that way, I was more trying to mean, that nftables replaces iptables but not the proxy, I will reword

@liam-mackie absolutely, I don't know how I missed it, sorry about it

[aojea]

@gauravkghildiyal can you provide a few lines about topology aware routing ?

[gauravkghildiyal]

For Topology Aware Routing:

---

• Topology Aware Routing using trafficDistribution: PreferClose Graduates to GA (KEP-4444, KEP-2433)
  • The Service trafficDistribution field, enabling topology aware routing with the PreferClose option, graduates to GA. This provides a stable and standardized way for users to express their preference for routing traffic to endpoints within the same zone. This GA status also signifies the stabilization of EndpointSlice Hints (KEP-2433), which are leveraged by trafficDistribution to achieve this topology-aware routing functionality.

[xing-yang]

SIG Storage highlights:

New Features:

• Volume Populator feature moved to GA in v1.33. This feature allows generic data populators by permitting any object to be the data sources for a PVC.
• Always Honor PersistentVolume Reclaim Policy moved to GA in v1.33. This feature prevents volumes from being leaked by honoring the PV Reclaim policy after the Bound PVC is also deleted.

Deprecations & Removals:

• In 1.33, we propose to Remove in-tree gitRepo volume driver code. The gitRepo volume type has been deprecated for a long time. There are Security concerns: gitRepo volume types can be exploited to gain remote code execution as root on the nodes. There are alternatives such as git-sync and initContainers. We aren't proposing removing gitVolumes from the Kubernetes API, meaning pods with gitRepo volumes will be admitted by kube-apiserver but kubelets with the feature-gate GitRepoVolumeDriver set to false will not run them and return an appropriate error to the user. This allows users to opt-in to re-enabling the driver for 3 versions to give them enough time to fix workloads. Finally in the 1.39 release, the feature gate in kubelet and in-tree plugin code will be removed.

cc @saad-ali @msau42 @jsafrane

[soltysh]

SIG CLI highlights:

• Separate kubectl user preferences from cluster configs - adds a .kuberc file with user preferences allowing kubectl customization. This feature is alpha in 1.33 so KUBECTL_KUBERC=true environment variable is required to use it.
• Add subresource support to kubectl finally reaches GA in 1.33. This adds a new --subresource flag to the following kubectl commands: get, patch, edit, apply and replace. It allows fetching and updating subresources like status, scale, etc.

[sftim]

+1 for kuberc, even as alpha

[soltysh]

SIG Apps highlights:

• Backoff Limit Per Index For Indexed Jobs reaches GA in 1.33. It allows counting failures per index rather than entire Job.
• Job success/completion policy reaches GA in 1.33. It adds a new API to define when a Job can be declared as succeeded based on the number of succeeded pods.

[sftim]

A blog article with examples about Job success/completion policy - or even a two-parter with different examples - could really help people understand the impact of this newly GA feature.

The concept is a bit abstract but something like a DES challenge cracker ("stop when you brute force the challenge") is easy to imagine. Maybe a voting example too, where the job output isn't stable but a solution can be accepted once there is an outright majority consensus on an answer?

Anyway, I recommend a small mention in the release highlights and a big push on post release comms. If folks like this idea, you also have to sell it to release comms, this is just my opinion of what will work.

[rytswd]

Both of the features have an open PR for Feature Blog, and I do agree these changes would really benefit from more real world examples in a blog 👍

[pohly]

SIG Scheduling/Node/Auth/Networking + WG Device Management:

For Dynamic Resource Allocation (DRA) we added several new features and enhanced existing ones:

• Partitionable Devices enables hardware vendors to advertise different, potentially conflicting hardware configurations (= "partitioning" a large device). The scheduler then chooses configurations based on demand. This happens automatically and is transparent to users.
• Device taints and tolerations enable DRA drivers and admins to limit the usage of devices, similar to node taints. Scheduling of pods which depend on those devices can be paused while a taint exists and/or pods using a tainted device can be evicted.
• Prioritized alternatives in device requests provide some additional flexibility to the user to specify that a request may be satisfied in different ways, including allocating nothing at all if some specific hardware is not available.
• ResourceClaim status with network interface data graduated to beta.
• Usage of admin access can be limited to users in certain namespaces via a standardized namespace label.
• In the kubelet, seamless upgrade support for the first time allows a kubelet plugin that is deployed as a DaemonSet to use a rolling update mechanism. In the case of DRA that prevents deleting and recreating per-node ResourceSlices reliably. There's now a 30 second grace period before the kubelet cleans up after unregistering a driver, which may help when a driver does not use a rolling update.
• Last but not least, some changes take the core API one step closer to GA:
  • v1beta2 simplifies the API.
  • Regular users with namespaced cluster edit role assigned have permission to use DRA.

[sftim]

+1 for seamless upgrades for plugin drivers / extended resources / DRA claimable resources.

[mbianchidev]

The following SIGs have nothing to highlight for this release:

• Lifecycle, see thread
• Cloud provider, see thread
• K8s Infra, see thread
• Testing, see thread
• UI, see thread
• Security, see thread

The following SIGs have yet to reply:

• Instrumentation, see thread
• Multi-cluster, see thread
• Scalability, see thread
• API Machinery, see thread

Thanks to SIG Autoscaling, Windows, Network, Storage, CLI, Apps, Scheduling, Node, Auth, Networking, WG Management, Instrumentation and WG Structured Logging for providing the updates!

[pohly]

SIG Instrumentation + WG Structured Logging:

• The k8s.io/client-go APIs were extended or (in as few cases as possible) modified to support contextual logging. The remaining changes are pending and hopefully will follow in 1.35.

[dgrisonnet]

• Z-pages introduced in 1.32 as an Alpha feature to the kube-apiserver with statusz and flagz are now available on all Kubernetes components.

[rytswd]

Just to clarify my understanding, statusz and flagz are both available as alpha but seem to be out of v1.33 scope. I suppose those should not be a part of the release highlight this time?

[dgrisonnet]

Just to clarify my understanding, statusz and flagz are both available as alpha but seem to be out of v1.33 scope.

I am not sure what you mean by that.
Statusz and flagz were introduced in 1.32, but only in the kube-apiserver. Now they are available across all the Kubernetes components since 1.33 so I think it is worth mentioning this improvement in 1.33. This is beta criteria that was worked on this release, but the feature wasn't ready for graduation just yet.

[rytswd]

Ah I see, thanks for the clarification. Where can I (and other readers) find such release details of statusz and flagz? I was mainly checking out KEPs, and it wasn't obvious to me...

[dgrisonnet]

Some of it is detailed in https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/4827-component-statusz/README.md#graduation-criteria, but you made me realized that it is incorrectly reflected on the KEP issue, I'll look into addressing that. Now that the feature is available across all components for users to try out, we will probably get some additional feedback by 1.34 and be able to graduate the feature to beta at that time.

[dom4ha]

SIG Scheduling highlights:

• Pop pod from the backoffQ when the activeQ is empty KEP-5142
  • Improves the overall scheduler performance when scheduling pods need to go through the unschedulable state.
• Asynchronous preemption in the scheduler graduated to Beta KEP-4832
  • Reduces the number of api-server calls inside the scheduling loop.
• MatchLabelKeys in Pod Affinity and Pod Anti Affinity graduated to GA KEP-3633
  • Allows to update a group of pods with Pod Affinity to a newer version without breaking the affinity constraints.
• NodeInclusionPolicyInPodTopologySpread graduated to GA KEP-3094
  • Takes taints/tolerations into consideration when calculating PodTopologySpread skew

[rayandas]

SIG Release highlights:

• In v1.33, the Release Notes and Docs sub-teams have merged into a single unified Docs team. This change streamlined efforts around documentation and release notes while ensuring better alignment between both deliverables.
• Improvements to the Docs team's processes have enabled them to take on additional responsibilities, which has contributed to a smoother and more efficient release cycle. Overall, the v1.33 release went smoothly, and the newly merged team integration was seamless.

[stlaz]

SIG Auth highlights:

New features:

• Ensure Secret Images implemented as alpha. It provides authorization to container image pulls for images already present on the node when these are requested with the IfNotPresent or Never image pull po licy in a Pod.
• Service Account Image Pull Credentials implemented as alpha. The feature enables using ServiceAccount tokens to be used for container image pulls.
• DRA Admin Access implemented as alpha. Usage of admin access can be limited to users in certain namespaces via a standardized namespace label.

Feature graduation:

• Bound ServiceAccount Token Improvements goes GA. This feature enables binding ServiceAccount tokens to Node objects and also improves ServiceAccount token tracking in audit logs.
• Cluster Trust Bundles moves to beta. Cluster Trust Bundles make it easy to publish X.509 trust anchors and then consume them in cluster workloads.

[haircommander]

SIG Node Highlights:

Pod configuration:

• Container Stop Signals allowing configurable stop signals specified at the pod API goes alpha.
• Fine-grained SupplementalGroups control adds support for enabling policies on how supplemental groups are handled. This feature went beta.
• Tune CrashLoopBackOff adds kubelet level policies on how CrashLoopBackoff is handled and has graduated to beta.
• Support for mounting images as volumes goes to beta. This is useful for storing AI models in registries and mounting them as volumes in pods.
• Expose node topology information to pods through downward API moves to alpha.
• Support User Namespaces in pods + add ProcMount option are two features that enable better pod isolation, and both of them are on-by-default beta.

Pod Lifecycle

• Support for In-Place Update of Pod Resources goes beta.
• Support for Sidecar Containers goes GA.
• Better tracking of pod status through use of Pod Generation (alpha).
• The sleep action for PreStop hook (and the corresponding KEP for enabling the zero value for it) adds an additional option in the pod lifecycle to add a sleep action before PreStop hooks and has moved to beta.

Resource Management:

• Three CPU manager policies have made progress: Split L3 cache to alpha 2, CPUManager policy to distribute CPUs across NUMA nodes went to beta, and options to reject non SMT-aligned workloads went GA.
• Kubelet feature for reporting Pressure Stall Information in stats and metrics is now alpha.

[sftim]

+1 for Linux userns

[sftim]

+1 for sidecar containers (needs a lot of comms)

[sftim]

+1 for vertical resize of existing Pods ("In-place update of Pod resources")

[rytswd]

Hi @haircommander , just double checking about kubernetes/enhancements#4603 -- in the thread here, you mentioned that this would graduate to beta, but looking at the KEP, I can see this is tracked as alpha for v1.33 (and marked as "alpha2" in the issue description). Could you confirm if we are talking about the right KEP here, and if this is something we would want to highlight?

[haircommander]

oops you're right! it is still in alpha2

[deads2k]

SIG API Machinery highlights:

New features:
Declarative Validation is beta and runs by default. It provides more reliable and consistent validation. Watch the declarative_validation_mismatch_total metric and let us know if it is above zero.

[sftim]

Do we need to mention this as a highlight? It's mostly an implementation detail AIUI.

[rytswd]

As far as I can see, there is no update to the underlying KEP-4153. Could you help me understand how this should be a part of the v1.33 release highlight? Perhaps did you mean to refer to KEP-5073?

[jpbetz]

Correct, kubernetes/enhancements#5073 is the KEP we want here. It superceeds the other one.

[rytswd]

Thanks for the confirmation @jpbetz!

[rytswd]

Just making sure to update all with the Release Highlights input:

No Notable Changes

• Docs, see thread
• Cloud provider, see thread
• K8s Infra, see thread
• Lifecycle, see thread
• Multicluster, confirmed via group DM
• Scalability, confirmed via group DM
• Security, see thread
• Testing, see thread
• UI, see thread

Highlights Provided

• API Machinery
• Apps
• Auth
• Autoscaling
• CLI
• Instrumentation
• Network
• Node
• Release
• Scheduling
• Storage
• Windows

Highlights with Alpha

• Auth
  • Ensure secret pulled images enhancements#2535
  • Projected service account tokens for Kubelet image credential providers enhancements#4412
  • DRA: AdminAccess for ResourceClaims and ResourceClaimTemplates enhancements#5018
• Autoscaling
  • Configurable tolerance for Horizontal Pod Autoscalers enhancements#4951
• CLI
  • Separate kubectl user preferences from cluster configs enhancements#3104
• Device Management
  • Partitionable Devices
    • DRA: Partitionable Devices enhancements#4815
  • Device taints and tolerations
    • DRA: device taints and tolerations enhancements#5055
  • Prioritized alternatives in device requests
    • DRA: Prioritized Alternatives in Device Requests enhancements#4816
  • AdminAccess for ResourceClaims and ResourceClaimTemplates
    • DRA: Resource Claim Status with possible standardized network interface data enhancements#4817
• Node
  • Pod configuration
    • Container Stop Signals enhancements#4960
    • Tune CrashLoopBackOff enhancements#4603
    • Expose Node Topology Labels via Downward API enhancements#4742
  • Pod lifecycle
    • Pod Generation enhancements#5067
  • Resource management
    • Split L3 Cache Topology Awareness in CPU Manager enhancements#5109
    • Support PSI based on cgroupv2 enhancements#4205

Highlights with Beta

• API Machinery
  • Declarative Validation Of Kubernetes Native Types With validation-gen enhancements#5073
• Auth
  • ClusterTrustBundles (previously Trust Anchor Sets) enhancements#3257
• Device Management
  • ResourceClaim status with network interface data
    • DRA: Resource Claim Status with possible standardized network interface data enhancements#4817
• Network
  • Dynamic Resource Allocation (DRA) for Network Interfaces reaches Beta
    • DRA: Resource Claim Status with possible standardized network interface data enhancements#4817
• Node
  • Pod configuration
    • Fine-grained SupplementalGroups control enhancements#3619
    • VolumeSource: OCI Artifact and/or Image enhancements#4639
    • Better pod isolation, on-by-default beta
      • Support User Namespaces in pods enhancements#127
      • add ProcMount option enhancements#4265
  • Pod lifecycle
    • In-Place Update of Pod Resources enhancements#1287
    • Sleep action with zero value before PreStop hook
      • Introducing Sleep Action for PreStop Hook enhancements#3960
      • Allow zero value for Sleep Action of PreStop Hook enhancements#4818
  • Resource management
    • Add CPUManager policy option to distribute CPUs across NUMA nodes instead of packing them enhancements#2902
• Scheduling
  • Pop pod from backoffQ when activeQ is empty enhancements#5142
  • Asynchronous preemption in the scheduler enhancements#4832
• Windows
  • Support for Direct Service Return (DSR) and overlay networking in Windows kube-proxy  enhancements#5100

Highlights with GA

• Apps
  • Backoff Limit Per Index For Indexed Jobs enhancements#3850
  • Job success/completion policy enhancements#3998
• Auth
  • Bound service account token improvements enhancements#4193
• CLI
  • Add subresource support to kubectl enhancements#2590
• Network
  • Multiple Service CIDRs graduates to GA
    • Multiple Service CIDRs enhancements#1880
  • kube-proxy nftables Backend is now GA
    • nftables kube-proxy backend enhancements#3866
  • Topology Aware Routing using trafficDistribution: PreferClose Graduates to GA
    • Traffic Distribution for Services enhancements#4444
    • Topology Aware Routing enhancements#2433
• Node
  • Pod lifecycle
    • Sidecar Containers  enhancements#753
  • Resource management
    • node: cpumanager: add options to reject non SMT-aligned workload enhancements#2625
• Scheduling
  • Introduce MatchLabelKeys to Pod Affinity and Pod Anti Affinity enhancements#3633
  • Take taints/tolerations into consideration when calculating PodTopologySpread skew enhancements#3094
• Storage
  • Volume Populator feature moved to GA
    • Generic data populators enhancements#1495
  • Always Honor PersistentVolume Reclaim Policy moved to GA
    • Always Honor PersistentVolume Reclaim Policy enhancements#2644

Miscellaneous highlights

• Device Management
  • DRA: seamless kubelet plugin upgrade kubernetes#128964
  • DRA: add user rbac kubernetes#130738
• Instrumentation
  • k8s.io/client-go APIs were extended or (in as few cases as possible) modified to support contextual logging
    • contextual logging enhancements#3077
• Release
  • Release Notes and Docs sub-teams have merged into a single unified Docs team.
  • Improvements to the Docs team's processes
• Windows
  • Stabilize unit tests on Windows and promote ci-kubernetes-unit-windows-master to release-informing kubernetes#130149

Highlights with Deprecation / Withdrawal / Removal

• Storage
  • Remove gitRepo volume driver enhancements#5040
• Windows
  • Host network support for Windows pods enhancements#3503

Highlights pending clarification

[rytswd]

I went ahead updating the heading to "miscellaneous highlights" to capture code changes as well as non-code changes👍

[aibarbetta]

@rytswd two small comments:

• Pod Generation enhancements#5067 is present in both the alpha and GA lists, probably a copy paste typo
• Tune CrashLoopBackOff enhancements#4603 was reported as beta but appears to be alpha2

Thanks for this summary!

[rytswd]

Thanks for the great catch, @aibarbetta! The one to be in GA was about sidecar container (surely something we cannot miss! 🫠).
As for the CrashLoopBackOff, it was actually highlighted as beta in the discussion thread above. However, I think you are right that this is still alpha, and I've corrected it for now. I'll drop a note in the thread above to double check as well.

[rytswd]

I have updated the highlight from SIG API Machinery to be in the beta, as well as reordering all the items based on alphabetic order. This should be final for the v1.33, and we will use this as the main input for the release announcement.

Thank you everyone for their input! 🥳