Change default container runtime to containerd - #23562
Conversation
The same value was named rtime in start, cr in docker-env and configureNodes, and runtimeName in validateCNI. cr is also used for cruntime.Manager and for a command.Runner, so a string named cr is easy to misread. containerRuntime cannot be the local: start_flags.go already uses that identifier for the --container-runtime flag key. Call the string crName, matching delete.go and addons.go. Leave cr for the runtime manager. One name makes it possible to find and review every place that chooses or checks the runtime before changing the default from docker to containerd.
validateGPUs treated an empty runtime as docker, the current default. That check would accept --gpus after the default changes to containerd, even though GPUs only work with the docker runtime. Pass the existing cluster into validateFlags so getContainerRuntime can resolve the name, and require docker driver plus docker runtime. Keep a test that empty runtime is rejected so the old assumption cannot return.
validateCNI ran only when --container-runtime was on the command line, and it used the raw viper value. An unset runtime is "", which was treated as docker, so --cni=false was skipped. Runtime from config, env, or an existing profile was never checked. Always run the check with getContainerRuntime so CNI is required for every non-docker runtime, including the implicit default.
validateBareMetal compared the raw viper value to defaultRuntime(), and skipped the warning when that value was empty. Unset was treated as docker, so runtime from config, env, or an existing profile was never checked. In commit 0af762e ("none driver: Add warnings about --cpus, --memory, and --container-runtime") the check was runtime != "docker". None with anything but docker was untested. In commit c4800a6 ("Make the default container runtime dynamic") the comparison became empty-or-defaultRuntime(k8sVersion), with a comment that only the default had been tested after dockershim removal. That assumed the product default would vary by Kubernetes version. In commit 50e9147 ("fix lint errors") defaultRuntime dropped the unused k8sVersion argument because it still always returned docker. The comment stayed, but the real assumption never changed: warn unless the runtime is docker. Pass resolved container runtime so the warning uses the actual name, and compare to docker again. After the product default switches to containerd, none with the implicit default will warn; none with docker will not.
cruntime.New maps "" to docker. That is not "use the product default". Until c4800a6 (2022, "Make the default container runtime dynamic") start wrote the raw --container-runtime flag into the profile, and the flag default is "". Those clusters are docker. Later starts store a real name via getContainerRuntime, so new containerd support never persisted "". Keep "" as docker when the product default switches to containerd.
TestMirrorCountry and TestGenerateCfgFromFlagsHTTPProxyHandling passed
constants.DefaultContainerRuntime ("") as the runtime name. Production
always passes getContainerRuntime(). Empty still becomes docker in
cruntime.New, so those tests would keep exercising docker after the
product default switches to containerd.
Pass defaultRuntime() so the tests follow the product default. Clarify
TestValidRuntime and TestName that "" is the pre-2022 stored empty
value, not the current default.
setContainerRuntimeOptions sent empty, docker, and unknown names through the default branch to GenerateDockerOptions. Empty looked like "use the product default", which would be wrong when that default becomes containerd. Name "", "docker" as its own case, matching cruntime.New. Pre-2022 profiles stored the empty flag default and are docker. Behavior is unchanged; unknown names still take the default branch.
4254f10 to
8f72313
Compare
8f72313 to
b1c09e2
Compare
|
/retest |
There was a problem hiding this comment.
This works well on my end — built successfully and all tests pass. I also verified it locally on my Windows dev box, where minikube start and minikube.exe start --container-runtime=docker successfully brings up a cluster. I've shared a pic of the runs.
A few edge cases worth a look — left as separate comments below.

|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: bobsira, nirs The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
getContainerRuntime used "" as one sentinel for "user did not set a runtime" and for a stored profile value. After copying from the old config, a second compare still saw "" and called defaultRuntime(), so pre-2022 profiles that persisted "" for docker would switch to containerd. The actual fix is to return docker for that stored empty value. Rewrite the function around the three sources so they stay distinct: user value, existing profile, then the product default. Add a table test for flag, existing profile, and legacy empty values.
updateExistingConfigFromFlags wrote KubernetesConfig.ContainerRuntime twice when the flag was passed. The generic updateStringFromFlag copied viper.GetString(containerRuntime), then later in the same function getContainerRuntime overwrote that field. The first write was wrong: an empty or unnormalized flag must not be stored. The resolved name from getContainerRuntime is the only value that belongs in the cluster config. Keep only that write.
When --container-runtime is omitted, integration tests return docker the same way defaultRuntime does. Those two values must change together when the product default switches to containerd. Point each function at the other so that commit cannot update only one.
New clusters used docker when --container-runtime was unset. Starting v1.39.0 that default is containerd. Return containerd from defaultRuntime and from the integration ContainerRuntime helper so tests that omit the flag match. Drop the v1.39 change warning, its constant, and the error_spam allowlist entry. Existing profiles keep the runtime stored in the cluster config.
The handbook still described docker as the default, which is no longer true for new clusters. State that the default is containerd, and show --container-runtime=docker as the way to opt in. Drop the redundant flag from the rootless docker example; that flow only works with containerd.
These starts built their own argv and never appended StartArgs(), so CI's --container-runtime never reached minikube start. After the default became containerd they would run containerd even on *-docker-* jobs. Append StartArgs() to the custom network, existing network, subnet, and static-ip starts.
Jobs named *-docker-* set cruntime: docker but never put it on the start command, so they used the product default. After the default became containerd those jobs would have run containerd. Add extra-start-args: --container-runtime=docker to: - docker-docker-ubuntu24.04-x86 - docker-docker-ubuntu24.04-arm - baremetal-docker-ubuntu-24.04-x86 - baremetal-docker-ubuntu-24.04-arm Run the podman job with cri-o, the documented rootful runtime, instead of docker. Rename podman-docker-ubuntu-24.04-x86 to podman-crio-ubuntu-24.04-x86 and pass --container-runtime=cri-o. Containerd and rootless jobs already passed --container-runtime=containerd.
The Windows Hyper-V TestFunctional run only set --driver=hyperv, so it used the product default. After the default became containerd that job would no longer test docker. Add --container-runtime=docker to --minikube-start-args.
The smoke jobs did not pass a runtime flag, so they used the product default. Smoke has one start per driver and platform, so that run should exercise containerd, not docker. Pass --container-runtime=containerd and rename the jobs from *-docker-* to *-containerd-*. Keep the flag explicit so the job name and start args stay in sync.
b1c09e2 to
db730d4
Compare
|
/retest |
|
/retest-required |
|
@nirs: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Roslaan001
left a comment
There was a problem hiding this comment.
@nirs , This is well thought out PR, I left some comments. and here are some too
-
Renaming
docker-docker-ubuntu-24.04-x86→docker-containerd-…andpodman-docker-… →podman-crio-… will strand any of those names configured as required checks in test-infra branch protection. -
site/content/en/docs/contrib/roadmap.en.md:25still has - [ ] Change the default container runtime from "docker" to "containerd" unchecked (dropping this here because the file is not in the diff) -
PR title typo: "continer" instead of "container".
| driver: podman | ||
| cruntime: docker | ||
| cruntime: crio | ||
| extra-start-args: --container-runtime=cri-o |
There was a problem hiding this comment.
This is the only place in the repo that uses the cri-o spelling; hack/prow/integration_docker_crio_linux_x86.sh:23 and integration_kvm_crio_linux_x86.sh:23 both set CONTAINER_RUNTIME=crio. test/integration/main_test.go:194 returns the raw flag value, so ContainerRuntime() here is cri-o, which matches none of the crio keys the tests use. Inside TestFunctional, which is what this job runs:
functional_test.go:1276 (CacheCmd/cache/cache_reload): case containerd, crio doesn't match, binary stays "*, and the command becomes ssh sudo *" rmi→ subtest error.
functional_test.go:477 (ImageCommands/ImageSaveDaemon): the crio localhost/ prefix isn't applied → docker image inspect fails → t.Fatalf.functional_test.go:2154 (NotActiveRuntimeDisabled):disableMap[cri-o] is nil, so the loop never runs; the check silently asserts nothing.functional_test.go:1355 (checkSaneLogs): drops the crio assertion.
net_test.go:238 is the only spot that normalizes the two spellings. crio is a valid flag value and matches this job's own cruntime: crio on line 120 so I suggest --container-runtime=crio.
|
|
||
| This works only with the default container runtime (containerd). | ||
|
|
||
| Unlike Podman driver, it is not necessary to set the `rootless` property of minikube (`minikube config set rootless true`). |
There was a problem hiding this comment.
I think this overstates it because cruntime/crio.go:184 enableRootless is explicitly "for running CRI-O in Rootless Docker," and cruntime/docker.go:136 has a rootless path too. The previous "recommended" wording was accurate. Also, per this PR's own CI rationale ("keep the flag explicit"), dropping --container-runtime=containerd from the example on line 47 breaks it for anyone with minikube config set container-runtime docker.
There was a problem hiding this comment.
We can replace with recommends like it was before, and add back the --container-runtime flag.
| | [image build command](/docs/handbook/pushing/#8-building-images-to-in-cluster-container-runtime) | all | ok | no | yes | | ||
|
|
||
| * Note 1: The default container-runtime on minikube is `docker`. | ||
| * Note 1: The default container-runtime on minikube is `containerd`. |
There was a problem hiding this comment.
I think it's worth adding that method 1 (docker-env) now needs --container-runtime=docker. on a default cluster minikube docker-env takes the nerdctld branch and prints "highly experimental feature", and method 1 is still what this page leads with.
There was a problem hiding this comment.
Right we need a note about this.
| {"nvidia", "docker", "docker", ""}, | ||
| {"all", "docker", "", ""}, | ||
| {"nvidia", "docker", "", ""}, | ||
| {"all", "docker", "", "The gpus flag can only be used with the docker driver and docker container-runtime"}, |
There was a problem hiding this comment.
The {"nvidia", "docker", "", ""} and {"amd", "docker", "", ""} rows were deleted rather than flipped to the error expectation, only all keeps the guard. I think it's cheap to keep all three.
These jobs are not required - but even if they are we cannot keep wrong name - the settings must be updated.
The roadmap is stale for years, fixing it is a no-goal for this change.
Fixed |
This prepares start and related code so the default container runtime
can change without treating an unset flag as docker, then switches the
default to containerd.
crNameso every place that chooses orchecks the runtime is easy to find, and so future work does not mix
that string up with a
cruntime.Manageror a command runner.getContainerRuntimeinto flagchecks (
--gpus, none-driver warning, CNI) instead of the raw vipervalue. Empty from
viper.Getis the auto sentinel (""), not docker.validateCNI, not only when--container-runtimeis onthe command line, so runtime from config or
MINIKUBE_CONTAINER_RUNTIMEis checked too.
""incruntime.Newand provision options isdocker: that is the pre-2022 profile format, not the current product
default. Do not map it to containerd.
ContainerRuntimeingetContainerRuntimeasdocker (pre-2022 profiles), not as unset. Unset from the user still
means the product default. Rewrite the function around those three
sources so a later default change cannot migrate those clusters.
viper.GetString(containerRuntime)intothe cluster config.
updateExistingConfigFromFlagsalready overwrotethat field with
getContainerRuntime; the first write was wrong.defaultRuntimeand the integrationContainerRuntimehelperat each other, and have generateClusterConfig tests pass
defaultRuntime()instead of"".and the error-spam allowlist entry for it.
--container-runtime=dockeris the opt-in.--container-runtimeon every GitHub functional job (dockerjobs were named
*-docker-*but used the product default). Keep theHyper-V extra job on docker. Smoke has one start per driver/platform,
so those jobs now pass
--container-runtime=containerdand arerenamed
*-containerd-*. The flag stays explicit so the job nameand start args stay in sync.
StartArgs()in the kic custom-network tests so CI's--container-runtimereachesminikube start.Existing profiles keep the runtime stored in the cluster config.
Docker is unchanged as a runtime. Users who want docker can still
use it; they must pass
--container-runtime=docker(or set it inconfig/env). Only the implicit default for new clusters changes.
Prow already passed
--driverand--container-runtimeon everyintegration job.
Example run - default runtime (containerd)
Example run - docker runtime
Fixes #22601