Driver probing is too slow, waiting 5 seconds for stopped podman vm

[nirs]

Starting minikube without a driver spends 5 seconds checking for stopped podman vm.

Starting with vfkit driver

% timestamp minikube start -d vfkit
[    0.000] minikube start -d vfkit
[    0.097] * minikube v1.38.1 on Darwin 26.6 (arm64)
[    0.113] * Using the vfkit driver based on user configuration
[    0.121] ! Starting v1.39.0, minikube will default to "containerd" container runtime. See #21973 for more info.
[    0.141] * Automatically selected the vmnet-shared network
[    0.150] * Starting "minikube" primary control-plane node in "minikube" cluster
[   14.600] * Configuring bridge CNI (Container Networking Interface) ...
[   14.690] * Verifying Kubernetes components...
[   14.699]   - Using image gcr.io/k8s-minikube/storage-provisioner:v5
[   15.092] * Enabled addons: default-storageclass, storage-provisioner
[   15.567] * Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Starting without --driver

% timestamp minikube start         
[    0.000] minikube start
[    0.097] * minikube v1.38.1 on Darwin 26.6 (arm64)
[    5.219] * Automatically selected the vfkit driver. Other choices: qemu2, ssh, krunkit (experimental)
[    5.223] ! Starting v1.39.0, minikube will default to "containerd" container runtime. See #21973 for more info.
[    5.233] * Automatically selected the vmnet-shared network
[    5.251] * Starting "minikube" primary control-plane node in "minikube" cluster
[   20.743] * Configuring bridge CNI (Container Networking Interface) ...
[   20.847] * Verifying Kubernetes components...
[   20.854]   - Using image gcr.io/k8s-minikube/storage-provisioner:v5
[   21.265] * Enabled addons: default-storageclass, storage-provisioner
[   21.785] * Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Logs

I0813 22:29:05.604329   17752 global.go:135] hyperkit default: true priority: 5, state: {Installed:false Healthy:false Running:false NeedsImprovement:false Error:exec: "hyperkit": executable file not found in $PATH Reason: Fix:Run 'brew install hyperkit' Doc:https://minikube.sigs.k8s.io/docs/reference/drivers/hyperkit/ Version:}
I0813 22:29:05.604492   17752 global.go:135] qemu2 default: true priority: 7, state: {Installed:true Healthy:true Running:true NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0813 22:29:05.604574   17752 global.go:135] vfkit default: true priority: 8, state: {Installed:true Healthy:true Running:true NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0813 22:29:05.604681   17752 global.go:135] virtualbox default: true priority: 6, state: {Installed:false Healthy:false Running:false NeedsImprovement:false Error:unable to find VBoxManage in $PATH Reason: Fix:Install VirtualBox Doc:https://minikube.sigs.k8s.io/docs/reference/drivers/virtualbox/ Version:}
I0813 22:29:05.604743   17752 global.go:135] docker default: true priority: 9, state: {Installed:false Healthy:false Running:false NeedsImprovement:false Error:exec: "docker": executable file not found in $PATH Reason: Fix:Install Docker Doc:https://minikube.sigs.k8s.io/docs/drivers/docker/ Version:}
I0813 22:29:05.604766   17752 global.go:135] ssh default: false priority: 4, state: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0813 22:29:05.624143   17752 main.go:144] libmachine: Validated vmnet-helper (path="/opt/homebrew/opt/vmnet-helper/libexec/vmnet-helper", version="v0.13.0", commit="222c121ba31e49856c9ec6c3a14426b49d77c8fe", needsSudo=false)
I0813 22:29:05.624159   17752 global.go:135] krunkit default: true priority: 3, state: {Installed:true Healthy:true Running:true NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0813 22:29:05.624259   17752 global.go:135] parallels default: true priority: 5, state: {Installed:false Healthy:false Running:false NeedsImprovement:false Error:exec: "prlctl": executable file not found in $PATH Reason: Fix:Install Parallels Desktop for Mac Doc:https://minikube.sigs.k8s.io/docs/drivers/parallels/ Version:}
I0813 22:29:05.624309   17752 global.go:135] vmware default: false priority: 5, state: {Installed:false Healthy:false Running:false NeedsImprovement:false Error:exec: "vmrun": executable file not found in $PATH Reason: Fix:Install vmrun Doc:https://minikube.sigs.k8s.io/docs/reference/drivers/vmware/ Version:}
W0813 22:29:10.711718   17752 podman.go:140] podman returned error: exit status 125
I0813 22:29:10.711803   17752 global.go:135] podman default: true priority: 3, state: {Installed:true Healthy:false Running:false NeedsImprovement:false Error:"podman version --format {{.Server.Version}}" exit status 125: Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to connect: dial tcp: lookup podman-vfkit.local: no such host Reason: Fix: Doc:https://minikube.sigs.k8s.io/docs/drivers/podman/ Version:}
I0813 22:29:10.711830   17752 driver.go:340] not recommending "ssh" due to default: false
I0813 22:29:10.711835   17752 driver.go:345] not recommending "krunkit" due to priority: 3
I0813 22:29:10.711839   17752 driver.go:335] not recommending "podman" due to health: "podman version --format {{.Server.Version}}" exit status 125: Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to connect: dial tcp: lookup podman-vfkit.local: no such host
I0813 22:29:10.711877   17752 driver.go:375] Picked: vfkit
I0813 22:29:10.711884   17752 driver.go:376] Alternatives: [qemu2 ssh krunkit (experimental)]
I0813 22:29:10.711901   17752 driver.go:377] Rejects: [hyperkit virtualbox docker parallels vmware podman]
I0813 22:29:10.716605   17752 out.go:180] * Automatically selected the vfkit driver. Other choices: qemu2, ssh, krunkit (experimental)

Timing analysis

Driver	Timestamp	Duration	Result
hyperkit	22:29:05.604329	~0.1ms	Not installed (PATH lookup)
qemu2	22:29:05.604492	~0.2ms	Healthy
vfkit	22:29:05.604574	~0.1ms	Healthy
virtualbox	22:29:05.604681	~0.1ms	Not installed (PATH lookup)
docker	22:29:05.604743	~0.1ms	Not installed (PATH lookup)
ssh	22:29:05.604766	~0.0ms	Installed, not default
krunkit	22:29:05.624159	~19ms	Healthy (includes vmnet-helper validation)
parallels	22:29:05.624259	~0.1ms	Not installed (PATH lookup)
vmware	22:29:05.624309	~0.1ms	Not installed (PATH lookup)
podman	22:29:10.711803	~5087ms	Unhealthy (connection timeout)

Observations

• All lookups are serial — each driver is probed one after the other.
• Most drivers complete in under 1ms (simple $PATH lookups).
• podman accounts for 99% of the total time (~5.1s out of ~5.1s), due
  to a DNS lookup timeout:

  dial tcp: lookup podman-vfkit.local: no such host
  

• krunkit takes ~19ms due to vmnet-helper binary validation, which is
  negligible.

Why it takes 5 seconds?

The podmnan vm is using mDNS. When the vm is not running, connect blocks 5 seconds resolving the address.

% time nc podman-krunkit.local 22
nc: getaddrinfo: nodename nor servname provided, or not known
nc podman-krunkit.local 22  0.01s user 0.01s system 0% cpu 5.036 total

Can we improve this?

Accessing healthy podman vm takes 0.189s-0.720s on old MacBook Pro M1 Pro (2021):

% time podman version --format '{{.Server.Version}}'
5.8.4
podman version --format '{{.Server.Version}}'  0.05s user 0.02s system 9% cpu 0.720 total

% time podman version --format '{{.Server.Version}}'
5.8.4
podman version --format '{{.Server.Version}}'  0.05s user 0.01s system 31% cpu 0.189 total

% time podman version --format '{{.Server.Version}}'
5.8.4
podman version --format '{{.Server.Version}}'  0.05s user 0.01s system 33% cpu 0.190 total

It seems that there some caching effect - first time takes much longer, and then access is much faster.

Minikube uses timeout of 6 seconds:

	ctx, cancel := context.WithTimeout(context.Background(), 6*time.Second)
	defer cancel()

This is too much for local machine - about 10x times of the maximum time seen. We cannot eliminate the delay, but we can make it shorter. Shorter delay may be too short for very old machines or when podman is overloaded, but overloaded podman is not healthy so we should not select it.

The second issue is checking drivers serially. We don't want to block for 6 seconds for every driver. The search should change to run in parallel, returning results after shorter timeout.

The most annoying effect of this bug is waiting for podman when the user never want to use the podmna driver. The user may use podman to build images, but never use the driver. It can be useful if we lets users disable some drivers that they never want to use, or enable only drivers that they want to use. For example I would enable only vfkit and krunkit drivers on macOS and only kvm and docker on Linux.

It can be something like:

minikube config set enbale-drivers vfkit,krunkit

Or:

minikube config set disable-drivers docker,podman

History of the driver detection timeout

• Jan 2020 — #6228:
  Docker auto-detection hangs when Docker Desktop is installed but stopped.
  tstromberg: "We need to put context timeouts for the auto-detection
  commands: none of them should take >1 second, for instance." Fixed by
  PR #6422 with a
  2-second timeout.

• Feb 2020 — #6579:
  The 2s timeout broke HyperV detection on Windows. The PowerShell command
  Get-WindowsOptionalFeature -FeatureName Microsoft-Hyper-V-All -Online
  takes 3–8 seconds on some machines. Users reported times of 2.009s, 3.18s,
  3.77s, and 8s. Fixed by
  PR #6701 which
  increased the timeout to 8 seconds for HyperV.

• Apr 2020 — Podman driver added (commit 95c9559) with a 6-second
  timeout, presumably influenced by the HyperV experience.

The HyperV issue was specific to a slow PowerShell command on Windows, not
SSH connections. The podman case on macOS is different — the delay comes
from mDNS resolution timing out when the VM is stopped, not from a
legitimately slow health check.

Managing timeout per driver

Design

Different drivers need different timeouts on different platforms. HyperV
needs 8 seconds on Windows because PowerShell is slow. Podman on macOS
connects over SSH, where a healthy response takes under 1 second.

Proposed design:

• Add a ProbeTimeout field to DriverDef, letting each driver declare
  its own timeout (which can vary by platform).
• When probing, build a list of candidate drivers for the current OS.
• Run probes in parallel.
• Use the largest ProbeTimeout among the candidates as the overall
  deadline.

This way each driver controls its own timeout, slow probes don't block
fast ones, and the total wait is bounded by the slowest candidate rather
than the sum of all candidates.

[OchukoWH]

/assign

[nirs]

@OchukoWH This is low priority issue requiring lot of effort. It is better to find issue with higher priority and less effort, but if this is something that you really want to improve feel free to work on it.

[OchukoWH]

Yeah, I picked this issue because it is aligned with this issue which I was already working on
#23349
I would love to work on this issue anyways
Thanks for the heads-up