Skip to content

Change default container runtime to containerd - #23562

Open
nirs wants to merge 16 commits into
kubernetes:masterfrom
nirs:default-container-runtime
Open

Change default container runtime to containerd#23562
nirs wants to merge 16 commits into
kubernetes:masterfrom
nirs:default-container-runtime

Conversation

@nirs

@nirs nirs commented Aug 25, 2026

Copy link
Copy Markdown
Member

This prepares start and related code so the default container runtime
can change without treating an unset flag as docker, then switches the
default to containerd.

  • Rename the runtime string to crName so every place that chooses or
    checks the runtime is easy to find, and so future work does not mix
    that string up with a cruntime.Manager or a command runner.
  • Pass the resolved runtime from getContainerRuntime into flag
    checks (--gpus, none-driver warning, CNI) instead of the raw viper
    value. Empty from viper.Get is the auto sentinel (""), not docker.
  • Always run validateCNI, not only when --container-runtime is on
    the command line, so runtime from config or MINIKUBE_CONTAINER_RUNTIME
    is checked too.
  • Document that a stored "" in cruntime.New and provision options is
    docker: that is the pre-2022 profile format, not the current product
    default. Do not map it to containerd.
  • Treat a stored empty ContainerRuntime in getContainerRuntime as
    docker (pre-2022 profiles), not as unset. Unset from the user still
    means the product default. Rewrite the function around those three
    sources so a later default change cannot migrate those clusters.
  • On restart, stop copying raw viper.GetString(containerRuntime) into
    the cluster config. updateExistingConfigFromFlags already overwrote
    that field with getContainerRuntime; the first write was wrong.
  • Point defaultRuntime and the integration ContainerRuntime helper
    at each other, and have generateClusterConfig tests pass
    defaultRuntime() instead of "".
  • Default new clusters to containerd. Remove the v1.39 change warning
    and the error-spam allowlist entry for it.
  • Update the handbook so the documented default is containerd, and
    --container-runtime=docker is the opt-in.
  • Pass --container-runtime on every GitHub functional job (docker
    jobs were named *-docker-* but used the product default). Keep the
    Hyper-V extra job on docker. Smoke has one start per driver/platform,
    so those jobs now pass --container-runtime=containerd and are
    renamed *-containerd-*. The flag stays explicit so the job name
    and start args stay in sync.
  • Append StartArgs() in the kic custom-network tests so CI's
    --container-runtime reaches minikube start.

Existing profiles keep the runtime stored in the cluster config.

Docker is unchanged as a runtime. Users who want docker can still
use it; they must pass --container-runtime=docker (or set it in
config/env). Only the implicit default for new clusters changes.

Prow already passed --driver and --container-runtime on every
integration job.

Example run - default runtime (containerd)

% minikube start -d vfkit
😄  minikube v1.38.1 on Darwin 26.6.2 (arm64)
✨  Using the vfkit driver based on user configuration
🌐  Automatically selected the vmnet-shared network
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔥  Creating vfkit VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
📦  Preparing Kubernetes v1.36.4 on containerd 2.3.4 ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Example run - docker runtime

% minikube start -d vfkit --container-runtime docker
😄  minikube v1.38.1 on Darwin 26.6.2 (arm64)
✨  Using the vfkit driver based on user configuration
🌐  Automatically selected the vmnet-shared network
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔥  Creating vfkit VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.36.4 on Docker 28.5.2 ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Fixes #22601

nirs added 6 commits August 25, 2026 22:27
The same value was named rtime in start, cr in docker-env and
configureNodes, and runtimeName in validateCNI. cr is also used for
cruntime.Manager and for a command.Runner, so a string named cr is
easy to misread.

containerRuntime cannot be the local: start_flags.go already uses
that identifier for the --container-runtime flag key.

Call the string crName, matching delete.go and addons.go. Leave cr
for the runtime manager.

One name makes it possible to find and review every place that chooses
or checks the runtime before changing the default from docker to
containerd.
validateGPUs treated an empty runtime as docker, the current default.
That check would accept --gpus after the default changes to containerd,
even though GPUs only work with the docker runtime.

Pass the existing cluster into validateFlags so getContainerRuntime can
resolve the name, and require docker driver plus docker runtime. Keep a
test that empty runtime is rejected so the old assumption cannot
return.
validateCNI ran only when --container-runtime was on the command line,
and it used the raw viper value. An unset runtime is "", which was
treated as docker, so --cni=false was skipped. Runtime from config,
env, or an existing profile was never checked.

Always run the check with getContainerRuntime so CNI is required for
every non-docker runtime, including the implicit default.
validateBareMetal compared the raw viper value to defaultRuntime(),
and skipped the warning when that value was empty. Unset was treated
as docker, so runtime from config, env, or an existing profile was
never checked.

In commit 0af762e ("none driver: Add warnings about --cpus,
--memory, and --container-runtime") the check was runtime !=
"docker". None with anything but docker was untested.

In commit c4800a6 ("Make the default container runtime dynamic")
the comparison became empty-or-defaultRuntime(k8sVersion), with a
comment that only the default had been tested after dockershim
removal. That assumed the product default would vary by Kubernetes
version.

In commit 50e9147 ("fix lint errors") defaultRuntime dropped the
unused k8sVersion argument because it still always returned docker.
The comment stayed, but the real assumption never changed: warn
unless the runtime is docker.

Pass resolved container runtime so the warning uses the actual name,
and compare to docker again. After the product default switches to
containerd, none with the implicit default will warn; none with docker
will not.
cruntime.New maps "" to docker. That is not "use the product default".
Until c4800a6 (2022, "Make the default container runtime dynamic")
start wrote the raw --container-runtime flag into the profile, and the
flag default is "". Those clusters are docker.

Later starts store a real name via getContainerRuntime, so new
containerd support never persisted "". Keep "" as docker when the
product default switches to containerd.
TestMirrorCountry and TestGenerateCfgFromFlagsHTTPProxyHandling passed
constants.DefaultContainerRuntime ("") as the runtime name. Production
always passes getContainerRuntime(). Empty still becomes docker in
cruntime.New, so those tests would keep exercising docker after the
product default switches to containerd.

Pass defaultRuntime() so the tests follow the product default. Clarify
TestValidRuntime and TestName that "" is the pre-2022 stored empty
value, not the current default.
@kubernetes-prow kubernetes-prow Bot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 25, 2026
@kubernetes-prow
kubernetes-prow Bot requested review from medyagh and prezha August 25, 2026 21:24
@kubernetes-prow kubernetes-prow Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 25, 2026
@nirs
nirs requested review from afbjorklund and spowelljr August 25, 2026 22:03
setContainerRuntimeOptions sent empty, docker, and unknown names through
the default branch to GenerateDockerOptions. Empty looked like "use the
product default", which would be wrong when that default becomes
containerd.

Name "", "docker" as its own case, matching cruntime.New. Pre-2022
profiles stored the empty flag default and are docker. Behavior is
unchanged; unknown names still take the default branch.
@nirs
nirs force-pushed the default-container-runtime branch from 4254f10 to 8f72313 Compare August 25, 2026 22:14
@nirs
nirs requested a review from bobsira August 25, 2026 22:18
@nirs
nirs force-pushed the default-container-runtime branch from 8f72313 to b1c09e2 Compare August 25, 2026 22:41
@nirs

nirs commented Aug 26, 2026

Copy link
Copy Markdown
Member Author

/retest

@bobsira bobsira left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works well on my end — built successfully and all tests pass. I also verified it locally on my Windows dev box, where minikube start and minikube.exe start --container-runtime=docker successfully brings up a cluster. I've shared a pic of the runs.

A few edge cases worth a look — left as separate comments below.
allgood

Comment thread cmd/minikube/cmd/start.go
Comment thread cmd/minikube/cmd/start.go
Comment thread cmd/minikube/cmd/start.go
@kubernetes-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bobsira, nirs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

nirs added 9 commits August 26, 2026 22:37
getContainerRuntime used "" as one sentinel for "user did not set a
runtime" and for a stored profile value. After copying from the old
config, a second compare still saw "" and called defaultRuntime(), so
pre-2022 profiles that persisted "" for docker would switch to
containerd.

The actual fix is to return docker for that stored empty value. Rewrite
the function around the three sources so they stay distinct: user value,
existing profile, then the product default.

Add a table test for flag, existing profile, and legacy empty values.
updateExistingConfigFromFlags wrote KubernetesConfig.ContainerRuntime
twice when the flag was passed. The generic updateStringFromFlag copied
viper.GetString(containerRuntime), then later in the same function
getContainerRuntime overwrote that field.

The first write was wrong: an empty or unnormalized flag must not be
stored. The resolved name from getContainerRuntime is the only value
that belongs in the cluster config. Keep only that write.
When --container-runtime is omitted, integration tests return docker
the same way defaultRuntime does. Those two values must change together
when the product default switches to containerd.

Point each function at the other so that commit cannot update only one.
New clusters used docker when --container-runtime was unset. Starting
v1.39.0 that default is containerd.

Return containerd from defaultRuntime and from the integration
ContainerRuntime helper so tests that omit the flag match. Drop the
v1.39 change warning, its constant, and the error_spam allowlist
entry. Existing profiles keep the runtime stored in the cluster
config.
The handbook still described docker as the default, which is no longer
true for new clusters.

State that the default is containerd, and show --container-runtime=docker
as the way to opt in. Drop the redundant flag from the rootless docker
example; that flow only works with containerd.
These starts built their own argv and never appended StartArgs(), so
CI's --container-runtime never reached minikube start. After the
default became containerd they would run containerd even on *-docker-*
jobs.

Append StartArgs() to the custom network, existing network, subnet,
and static-ip starts.
Jobs named *-docker-* set cruntime: docker but never put it on the
start command, so they used the product default. After the default
became containerd those jobs would have run containerd.

Add extra-start-args: --container-runtime=docker to:

- docker-docker-ubuntu24.04-x86
- docker-docker-ubuntu24.04-arm
- baremetal-docker-ubuntu-24.04-x86
- baremetal-docker-ubuntu-24.04-arm

Run the podman job with cri-o, the documented rootful runtime, instead
of docker. Rename podman-docker-ubuntu-24.04-x86 to
podman-crio-ubuntu-24.04-x86 and pass --container-runtime=cri-o.

Containerd and rootless jobs already passed --container-runtime=containerd.
The Windows Hyper-V TestFunctional run only set --driver=hyperv, so it
used the product default. After the default became containerd that job
would no longer test docker.

Add --container-runtime=docker to --minikube-start-args.
The smoke jobs did not pass a runtime flag, so they used the product
default. Smoke has one start per driver and platform, so that run
should exercise containerd, not docker.

Pass --container-runtime=containerd and rename the jobs from
*-docker-* to *-containerd-*. Keep the flag explicit so the job
name and start args stay in sync.
@nirs
nirs force-pushed the default-container-runtime branch from b1c09e2 to db730d4 Compare August 26, 2026 20:04
@nirs
nirs requested a review from bobsira August 26, 2026 20:05
@nirs

nirs commented Aug 26, 2026

Copy link
Copy Markdown
Member Author

@bobsira The issue in getContainerRuntime is resolved in 03314a7.

@nirs

nirs commented Aug 26, 2026

Copy link
Copy Markdown
Member Author

/retest

@nirs

nirs commented Aug 26, 2026

Copy link
Copy Markdown
Member Author

/retest-required

@kubernetes-prow

kubernetes-prow Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

@nirs: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-minikube-docker-crio-linux-x86 db730d4 link false /test pull-minikube-docker-crio-linux-x86

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@Roslaan001 Roslaan001 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nirs , This is well thought out PR, I left some comments. and here are some too

  • Renaming docker-docker-ubuntu-24.04-x86docker-containerd-… and podman-docker-… → podman-crio-… will strand any of those names configured as required checks in test-infra branch protection.

  • site/content/en/docs/contrib/roadmap.en.md:25 still has - [ ] Change the default container runtime from "docker" to "containerd" unchecked (dropping this here because the file is not in the diff)

  • PR title typo: "continer" instead of "container".

Comment thread cmd/minikube/cmd/start.go
driver: podman
cruntime: docker
cruntime: crio
extra-start-args: --container-runtime=cri-o

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the only place in the repo that uses the cri-o spelling; hack/prow/integration_docker_crio_linux_x86.sh:23 and integration_kvm_crio_linux_x86.sh:23 both set CONTAINER_RUNTIME=crio. test/integration/main_test.go:194 returns the raw flag value, so ContainerRuntime() here is cri-o, which matches none of the crio keys the tests use. Inside TestFunctional, which is what this job runs:

  • functional_test.go:1276 (CacheCmd/cache/cache_reload): case containerd, crio doesn't match, binary stays "*, and the command becomes ssh sudo *" rmi → subtest error.
  • functional_test.go:477 (ImageCommands/ImageSaveDaemon): the crio localhost/ prefix isn't applied → docker image inspect fails → t.Fatalf.
  • functional_test.go:2154 (NotActiveRuntimeDisabled): disableMap[cri-o] is nil, so the loop never runs; the check silently asserts nothing.
  • functional_test.go:1355 (checkSaneLogs): drops the crio assertion.

net_test.go:238 is the only spot that normalizes the two spellings. crio is a valid flag value and matches this job's own cruntime: crio on line 120 so I suggest --container-runtime=crio.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea

Comment thread .github/workflows/functional_test.yml
Comment thread cmd/minikube/cmd/start.go
Comment thread cmd/minikube/cmd/start_flags.go
Comment thread cmd/minikube/cmd/start_flags.go

This works only with the default container runtime (containerd).

Unlike Podman driver, it is not necessary to set the `rootless` property of minikube (`minikube config set rootless true`).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this overstates it because cruntime/crio.go:184 enableRootless is explicitly "for running CRI-O in Rootless Docker," and cruntime/docker.go:136 has a rootless path too. The previous "recommended" wording was accurate. Also, per this PR's own CI rationale ("keep the flag explicit"), dropping --container-runtime=containerd from the example on line 47 breaks it for anyone with minikube config set container-runtime docker.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can replace with recommends like it was before, and add back the --container-runtime flag.

| [image build command](/docs/handbook/pushing/#8-building-images-to-in-cluster-container-runtime) | all | ok | no | yes |

* Note 1: The default container-runtime on minikube is `docker`.
* Note 1: The default container-runtime on minikube is `containerd`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's worth adding that method 1 (docker-env) now needs --container-runtime=docker. on a default cluster minikube docker-env takes the nerdctld branch and prints "highly experimental feature", and method 1 is still what this page leads with.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right we need a note about this.

Comment thread cmd/minikube/cmd/start.go
{"nvidia", "docker", "docker", ""},
{"all", "docker", "", ""},
{"nvidia", "docker", "", ""},
{"all", "docker", "", "The gpus flag can only be used with the docker driver and docker container-runtime"},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The {"nvidia", "docker", "", ""} and {"amd", "docker", "", ""} rows were deleted rather than flipped to the error expectation, only all keeps the guard. I think it's cheap to keep all three.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll check this

@nirs nirs changed the title Change default continer runtime to containerd Change default container runtime to containerd Aug 26, 2026
@nirs

nirs commented Aug 27, 2026

Copy link
Copy Markdown
Member Author

@nirs , This is well thought out PR, I left some comments. and here are some too

  • Renaming docker-docker-ubuntu-24.04-x86docker-containerd-… and podman-docker-… → podman-crio-… will strand any of those names configured as required checks in test-infra branch protection.

These jobs are not required - but even if they are we cannot keep wrong name - the settings must be updated.

  • site/content/en/docs/contrib/roadmap.en.md:25 still has - [ ] Change the default container runtime from "docker" to "containerd" unchecked (dropping this here because the file is not in the diff)

The roadmap is stale for years, fixing it is a no-goal for this change.

  • PR title typo: "continer" instead of "container".

Fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Change default container runtime to containerd

3 participants