`image load` silently fails when the image is in use by a running container

[monkey-mas]

Problem

minikube image load <image:tag> silently fails to update an image when a container using that image is already running inside minikube.

This is a common scenario during development: rebuild a Docker image with the same tag (e.g., myapp:0.1.0-SNAPSHOT), then reload it into minikube while the previous deployment is still running. The command exits successfully with no error, but the image inside minikube is not updated.

Related to #11276 (originally fixed in #11366).

Root cause

transferAndLoadImage calls removeExistingImage before loading the new image (ref: v1.38.1 source). removeExistingImage runs the runtime's remove command (e.g., docker rmi <image:tag>) inside minikube, which fails when a running container references the image:

conflict: unable to remove repository reference "myapp:0.1.0" (must force)
- container abc123 is using its referenced image def456

The error-handling in removeExistingImage uses ad-hoc string matching against a hard-coded list of "acceptable" error messages (ref: v1.38.1 source):

• "no such image"
• "unable to remove the image" (added in fix: fix empty tarball when generating image save #19312)

Any error that does not match is treated as fatal. This includes Docker's "unable to remove repository reference" error shown above, as well as any new error messages that container runtimes may introduce in the future. When the error is treated as fatal, LoadImage is never called. Additionally, DoLoadImages swallows the error (returns nil), so the user sees no indication of failure.

Note: with older versions of crictl rmi (cri-tools ≤ v1.31), the error message for any removal failure was a catch-all "unable to remove the image(s)" (ref: cri-tools v1.29.0 source), which happened to match the
hard-coded pattern. Since cri-tools v1.32.0 (ref: cri-tools PR #1639), image removal was parallelized and the catch-all was replaced with per-image error messages, meaning the pattern match no longer works for newer CRI runtimes either. This further demonstrates that ad-hoc string matching is not a sustainable approach.

More fundamentally, removal of the old image cannot be a precondition for loading the new one. A running container holds a reference to the image it was started from, and that reference is not released until the container is stopped - not even after a new image is loaded with the same tag. This means a "remove-then-load" sequence will always fail when a container is running with the target image.

Proposed fix

Two approaches are possible:

Option 1: Remove the removeExistingImage step entirely (preferred)

Remove the removeExistingImage call from transferAndLoadImage altogether, as suggested in the review of #22993 (ref: review comment). This makes minikube image load semantics consistent with the underlying container runtime, i.e., users get the same behavior they would expect from docker load / ctr images import / podman load directly.

As shown in the verification below, all three container runtimes can overwrite an existing tag without prior removal, even when a container is using the old image. Removing this step eliminates minikube-specific logic that has been a recurring source of bugs (#11276, #11992, #19312).

Option 2: Make removeExistingImage best-effort

Keep the removal attempt but treat all errors as non-fatal (log and continue to LoadImage). It preserves the "try to clean up first" behavior while ensuring that LoadImage is always reached.

It looks like this approach was originally suggested during the review in #11366 (ref: review comment):

or alternatively we could do

klog.Warniningf("failed to remove image (which might be okay if it doesn't exists")

The initial commit of that PR (32a91d6) implemented this as a warning, but a follow-up commit within the same PR (16dbac1f) changed it to a fatal error to address performance concerns around ListImages. See the full commit history of #11366 for context.

Comparison

	Option 1 (remove step)	Option 2 (best-effort)
Behavior	Same as underlying container runtime	minikube-specific "try remove first"
Code change	Delete removeExistingImage and its call	Change error handling to non-fatal
Dangling images	Handled by container runtime natively	Same, but may remove some before load
Future maintenance	None	None (no more pattern matching)

Verification: container runtimes can overwrite tags without prior removal

We verified that all three container runtimes supported by minikube can overwrite an existing image tag while a container is using the old image. This proves that the removeExistingImage step is not a precondition for LoadImage.

Note: minikube uses podman load for the CRI-O runtime (ref: v1.38.1 source).

Test environment

Runtime	Version
Docker	29.5.1
containerd	2.2.3
Podman	4.9.3

Test setup

# Dockerfile.v1
FROM busybox:latest
RUN echo "version1" > /version.txt

# Dockerfile.v2
FROM busybox:latest
RUN echo "version2" > /version.txt
RUN echo "sample content" > /sample.txt

# Build and save tar files
docker buildx build -f Dockerfile.v1 -t repro-test:0.1.0 .
docker save repro-test:0.1.0 -o /tmp/v1.tar
docker buildx build -f Dockerfile.v2 -t repro-test:0.1.0 .
docker save repro-test:0.1.0 -o /tmp/v2.tar

Test procedure

1. Load v1 image
2. Verify v1 image is loaded
3. Start a container using v1 (simulates a running deployment)
4. Verify the container is running
5. Load v2 image with the same tag while the container is running
6. Verify v2 image is loaded
7. Verify original container is still running
8. Check old image status

Docker (docker load)

# 1. Load v1
docker load -i /tmp/v1.tar
# Output: Loaded image: repro-test:0.1.0

docker images --no-trunc --format '{{.ID}}' repro-test:0.1.0
# Output: sha256:27e2f051ea8985616e8f9291c09fd35727122ab0155cfd890923b6e0106a2400

# 2. Verify v1
docker run --rm repro-test:0.1.0 cat /version.txt
# Output: version1

# 3. Start container using v1
docker run -d --name repro-running repro-test:0.1.0 sleep 3600

# 4. Verify container is running
docker ps --filter name=repro-running --format '{{.Names}} {{.Status}}'
# Output: repro-running Up Less than a second

# 5. Load v2 (same tag, while container is running)
docker load -i /tmp/v2.tar
# Output:
#   The image repro-test:0.1.0 already exists, renaming the old one with
#   ID sha256:27e2... to empty string
#   Loaded image: repro-test:0.1.0

# 6. Verify v2 is loaded
docker run --rm repro-test:0.1.0 cat /version.txt
# Output: version2
docker run --rm repro-test:0.1.0 cat /sample.txt
# Output: sample content

# 7. Verify original container is still running
docker ps --filter name=repro-running --format '{{.Names}} {{.Status}}'
# Output: repro-running Up Less than a second

# 8. Check old image status
docker images --filter dangling=true --no-trunc --format '{{.ID}}'
# Output includes: sha256:27e2f051ea89... (v1 is now dangling)

# Cleanup
docker rm -f repro-running
docker rmi repro-test:0.1.0

Docker explicitly reports the tag reassignment in step 5: "renaming the old one ... to empty string". Old image becomes dangling (<none>:<none>) and can be cleaned up with docker image prune.

containerd (ctr images import)

NS="test"
IMAGE="docker.io/library/repro-test:0.1.0"

# 1. Load v1
sudo ctr -n="$NS" images import /tmp/v1.tar
# Output: docker.io/library/repro-test:0.1.0 saved
#   sha256:0ba0043d1df615956e2528c841713ea3f3bec0a38d8ce0ece26982b93ef01793

sudo ctr -n="$NS" images ls "name==$IMAGE" | grep -o 'sha256:[a-f0-9]*'
# Output: sha256:0ba0043d1df615956e2528c841713ea3f3bec0a38d8ce0ece26982b93ef01793

# 2. Verify v1
sudo ctr -n="$NS" run --rm "$IMAGE" test-v1 cat /version.txt
# Output: version1

# 3. Start container using v1
sudo ctr -n="$NS" run -d "$IMAGE" repro-running /bin/sh -c 'sleep 3600'

# 4. Verify container is running
sudo ctr -n="$NS" tasks ls | grep repro-running
# Output: repro-running    2257496    RUNNING

# 4a. Record v1 snapshots (before v2 load)
sudo ctr -n="$NS" snapshots ls
# Output:
#   repro-running                  sha256:10dd19a4...  Active
#   sha256:10dd19a4...             sha256:7f74ca72...  Committed
#   sha256:7f74ca72...                                 Committed

# 5. Load v2 (same tag, while container is running)
sudo ctr -n="$NS" images import /tmp/v2.tar
# Output: docker.io/library/repro-test:0.1.0 saved
#   sha256:66fe5e60b722e9596232a64a99a79b8feb10bd95eb29b14ba1094d03339b2a4e

# 6. Verify v2 is loaded
sudo ctr -n="$NS" run --rm "$IMAGE" test-v2 cat /version.txt
# Output: version2
sudo ctr -n="$NS" run --rm "$IMAGE" test-v2b cat /sample.txt
# Output: sample content

# 7. Verify original container is still running
sudo ctr -n="$NS" tasks ls | grep repro-running
# Output: repro-running    2257496    RUNNING

# 8. Check old image status
sudo ctr -n="$NS" images ls "name==$IMAGE" | grep -o 'sha256:[a-f0-9]*'
# Output: sha256:66fe5e60b722... (tag now points to v2)

# v1 manifest may or may not be in content store (GC timing varies)
sudo ctr -n="$NS" content ls | grep sha256:0ba0043d1df6
# Output: present or absent depending on GC timing

# v1 container snapshot is retained (filesystem layers protected)
sudo ctr -n="$NS" containers info repro-running | grep SnapshotKey
# Output: "SnapshotKey": "repro-running"
sudo ctr -n="$NS" snapshots info repro-running
# Output: confirms snapshot exists

# All v1 snapshots from step 4a are retained while container is running
sudo ctr -n="$NS" snapshots info sha256:10dd19a4547510a810302be61ab179dc42af8565f8b3519688e21b2d5a217ed1
# Output: retained
sudo ctr -n="$NS" snapshots info sha256:7f74ca72855668453fc4207c1e5c50bb5653e72bb9939eb1d5512c001624e927
# Output: retained

# Cleanup
sudo ctr -n="$NS" tasks kill -s SIGKILL repro-running
sudo ctr -n="$NS" tasks rm repro-running
sudo ctr -n="$NS" containers rm repro-running
sudo ctr -n="$NS" images rm "$IMAGE"

containerd replaces the tag reference with the new digest. The v1 manifest (metadata) may be garbage-collected, but all v1 filesystem layers (snapshots) recorded in step 4a are confirmed retained after v2 load - the running container's snapshot chain is protected. Unlike Docker, containerd does not keep dangling images as <none>:<none> - old content is handled automatically by GC, while layers referenced by running containers are preserved.

Podman (podman load - used by minikube for CRI-O runtime)

# 1. Load v1
podman load -i /tmp/v1.tar
# Output: Loaded image: docker.io/library/repro-test:0.1.0

podman images --no-trunc --format '{{.ID}}' repro-test:0.1.0
# Output: sha256:27e2f051ea8985616e8f9291c09fd35727122ab0155cfd890923b6e0106a2400

# 2. Verify v1
podman run --rm repro-test:0.1.0 cat /version.txt
# Output: version1

# 3. Start container using v1
podman run -d --name repro-running repro-test:0.1.0 sleep 3600

# 4. Verify container is running
podman ps --filter name=repro-running --format '{{.Names}} {{.Status}}'
# Output: repro-running Up Less than a second

# 5. Load v2 (same tag, while container is running)
podman load -i /tmp/v2.tar
# Output:
#   Copying blob 7f74ca728556 skipped: already exists
#   Copying blob 419389ad29ef done
#   Loaded image: docker.io/library/repro-test:0.1.0

# 6. Verify v2 is loaded
podman run --rm repro-test:0.1.0 cat /version.txt
# Output: version2
podman run --rm repro-test:0.1.0 cat /sample.txt
# Output: sample content

# 7. Verify original container is still running
podman ps --filter name=repro-running --format '{{.Names}} {{.Status}}'
# Output: repro-running Up 1 second

# 8. Check old image status
podman images --filter dangling=true --no-trunc --format '{{.ID}}'
# Output includes: sha256:27e2f051ea89... (v1 is now dangling)

# Cleanup
podman rm -f repro-running
podman rmi repro-test:0.1.0

Podman reuses existing layers ("skipped: already exists") and reassigns the tag. Old image becomes dangling, same as Docker.

Note: minikube uses podman load for the CRI-O runtime (ref: v1.38.1 source).

Summary

	Docker	containerd	Podman (CRI-O)
Tag overwrite	Yes	Yes	Yes
Running container affected	No	No	No
Old image layers (while container runs)	Retained as dangling (<none>)	Snapshots retained; manifest may be GC'd	Retained as dangling (<none>)
Prior removal required	No	No	No

Reproduction steps (minikube)

# Build v1 and v2
docker buildx build -f Dockerfile.v1 -t repro-test:0.1.0 .
docker buildx build -f Dockerfile.v2 -t repro-test:0.1.0 .

# Load v1 into minikube
minikube image load repro-test:0.1.0

# Start a container using v1
minikube ssh -- docker run -d --name repro-running repro-test:0.1.0 sleep 3600

# Rebuild as v2 and try to reload
docker buildx build -f Dockerfile.v2 -t repro-test:0.1.0 .
minikube image load --overwrite repro-test:0.1.0

# Verify: still v1 (silent failure)
minikube ssh -- docker run --rm repro-test:0.1.0 cat /version.txt
# Output: "version1" (expected: "version2")

[monkey-mas]

/cc @nirs Please let me know if there's something missing that you'd like me to describe more in detail.

[nirs]

So if we load an image when the image is used the only issue is dangling images while the image is used. Are these cleaned up when the container using is removed?

For containerd you wrote:

Snapshots retained; manifest may be GC'd

When the container using the old images is removed, what happens to the snapshot and the manifest?

Regardless, not removing before loading looks like the right way, unless there is some advantage in trying to remove. Does it help in any way?

[nirs]

The initial commit of that PR (32a91d6) implemented this as a warning, but a follow-up commit within the same PR (16dbac1f) changed it to a fatal error to address performance concerns around ListImages. See the full commit history of #11366 for context.

Did you find the comment about the performance issue with ListImages? Is this a real issue or just a concern without evidence?

I guess this is #11366 (comment) ?

[monkey-mas]

So if we load an image when the image is used the only issue is dangling images while the image is used. Are these cleaned up when the container using is removed?

For Docker and Podman, dangling images are not automatically cleaned up when a container is removed. Users need to explicitly run something like docker image prune to reclaim space.

When the container using the old images is removed, what happens to the snapshot and the manifest?

For containerd, once the container is removed, the snapshots and manifests become eligible for GC. GC runs automatically. (ie no manual cleanup is needed)

I verified this by extending the containerd test from the issue description with a step 9 below. Steps 1 to 8 are the same as described in the issue description. After step 8 confirms that v1 snapshots are retained while the container is running:

Step 9: Remove container and verify GC cleanup

# 9. Remove container and verify GC cleanup
sudo ctr -n="$NS" tasks kill -s SIGKILL repro-running
sudo ctr -n="$NS" tasks rm repro-running
sudo ctr -n="$NS" containers rm repro-running
sleep 3  # wait for GC

# Check v1 snapshots
sudo ctr -n="$NS" snapshots info repro-running
# Output: not found (cleaned up by GC)
sudo ctr -n="$NS" snapshots info sha256:10dd19a4...
# Output: not found (cleaned up by GC - v1-only layer)

# Check v1 manifest
sudo ctr -n="$NS" content ls | grep sha256:0ba0043d1df6
# Output: (empty - cleaned up by GC)

# Check shared base layer (busybox used by both v1 and v2)
sudo ctr -n="$NS" snapshots info sha256:7f74ca72...
# Output: exists (retained - still referenced by v2)

# Check v2 snapshots and manifest are retained
sudo ctr -n="$NS" snapshots info sha256:bf1f0047...
# Output: exists (retained)
sudo ctr -n="$NS" snapshots info sha256:dcbc6952...
# Output: exists (retained)
sudo ctr -n="$NS" content ls | grep sha256:66fe5e60b722
# Output: exists (retained)

v1-only resources are cleaned up by GC after container removal. The shared base layer (sha256:7f74ca72...) is retained because v2 still references it. v2 snapshots and manifest are all retained since the image tag (repro-test:0.1.0) still points to v2.

Regardless, not removing before loading looks like the right way, unless there is some advantage in trying to remove. Does it help in any way?

I don't think removing-before-loading provides any meaningful advantage. Deleting the removal step simplifies the code, aligns minikube with the runtime semantics, and eliminates a recurring source of issues.

[monkey-mas]

Did you find the comment about the performance issue with ListImages? Is this a real issue or just a concern without evidence?

I guess this is #11366 (comment)?

Yes, that's the comment I referenced, and it was just a concern without evidence. In any case, Option 1 (removing the step entirely) does not reintroduce ListImages.

[nirs]

@monkey-mas Thanks for this detailed issue.

I think there is no reason to do the best effort remove before replacing an image.

The issue of leftover dangling images requiring prune is known for podman and docker. I think minikube should add a prune command to complete image support. We can open a new issue for this.

You can create a new PR for this change and keep the previous PR if someone has a good reason why it is better.

/cc @medyagh
/cc @afbjorklund

[monkey-mas]

@nirs
Thanks for the review!
I'll submit a new PR to remove the removal logic entirely (Option 1), and keep the previous PR (#22993) as an alternative. I'll also open a separate issue for adding a prune command as you suggested once I complete the new PR to delete the removal logic.