Feature Request: Support `docker save` compatible archive format for k3s/containerd air-gap installations

[garysassano]

Summary

When using skopeo copy with docker-archive: destination to pre-cache container images for k3s air-gap installations, the resulting tarball cannot be:

1. Auto-imported by k3s from /var/lib/rancher/k3s/agent/images/
2. Directly imported with ctr images import (fails with overlayfs whiteout errors)

The workaround requires loading into Docker daemon first (docker load), then re-saving (docker save | ctr import), which defeats some of skopeo's benefits.

Use Case

Building Docker images that need to pre-cache container images for air-gapped Kubernetes (k3s) environments:

1. During Docker build (has internet): Download images as tarballs
2. At runtime (no internet): k3s auto-imports from /var/lib/rancher/k3s/agent/images/

Why skopeo? During docker build, there's no Docker daemon available - docker pull/docker save don't work. Skopeo is designed for daemon-less operation, making it ideal for this use case.

Current Behavior

# Download image without Docker daemon (works)
RUN skopeo copy --insecure-policy \
    docker://quay.io/jetstack/cert-manager-controller:v1.19.2 \
    docker-archive:/var/lib/rancher/k3s/agent/images/cert-manager.tar:quay.io/jetstack/cert-manager-controller:v1.19.2

Problem 1: k3s does NOT auto-import these tarballs at startup.

Problem 2: Manual import also fails:

$ ctr -a /run/k3s/containerd/containerd.sock -n k8s.io images import cert-manager.tar

ctr: failed to extract layer ... failed to convert whiteout file "app/.wh..wh..opq": operation not supported

Required Workaround

Must load into Docker daemon first, then pipe to ctr:

# In setup.sh (runtime, Docker daemon available)
docker load -i /opt/cached-images/cert-manager.tar
docker save quay.io/jetstack/cert-manager-controller:v1.19.2 | \
    ctr -a /run/k3s/containerd/containerd.sock -n k8s.io images import -

This works, but:

• Requires Docker daemon at runtime (which may not always be available)
• Extra step/complexity
• Can't use k3s's built-in auto-import feature

Expected Behavior

skopeo copy docker://... docker-archive:... should produce tarballs that:

1. k3s can auto-import from /var/lib/rancher/k3s/agent/images/
2. ctr images import can directly consume
3. Are equivalent to docker save output

Technical Analysis

The issue appears to be in how OCI whiteout files (.wh. prefixed files used for layer deletions) are stored in the archive. Docker's format uses a slightly different representation that containerd's ctr import expects.

Proposed Solutions

1. New format/flag: Add --docker-v2-archive or --containerd-compat flag that produces containerd-compatible output
2. Fix docker-archive format: Ensure the existing docker-archive: output is fully compatible with containerd
3. New destination type: containerd-archive: specifically for containerd/k3s use cases

Environment

• skopeo version: 1.9.3 (Ubuntu 22.04 apt)
• k3s version: v1.31.x
• containerd version: 1.7.x (bundled with k3s)
• Docker version: 24.x (for workaround)

Why Not Use Docker Pull/Save?

You might ask: "Why not just use docker pull + docker save in the Dockerfile?"

Answer: There's no Docker daemon available during docker build. Docker buildkit runs containers to execute RUN commands, but the Docker daemon itself isn't accessible. This is why:

# THIS FAILS - no daemon during build
RUN docker pull quay.io/jetstack/cert-manager-controller:v1.19.2
# Error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock

Skopeo is designed for exactly this use case - it can pull and manipulate images without requiring a daemon.

Workaround Code

The current working pattern requires a two-step process:

Step 1: Dockerfile (no daemon - MUST use skopeo):

RUN skopeo copy --insecure-policy \
    docker://quay.io/jetstack/cert-manager-controller:v1.19.2 \
    docker-archive:/opt/cached-images/cert-manager.tar:quay.io/jetstack/cert-manager-controller:v1.19.2

Step 2: setup.sh (runtime - has daemon - roundtrip through Docker):

# Load skopeo's tarball into Docker
docker load -i /opt/cached-images/cert-manager.tar

# Re-save via Docker and import to containerd
docker save quay.io/jetstack/cert-manager-controller:v1.19.2 | \
    ctr -a /run/k3s/containerd/containerd.sock -n k8s.io images import -

This roundtrip is necessary because skopeo's docker-archive format isn't containerd-compatible.

Impact

This affects anyone trying to build air-gapped k3s/containerd images using Dockerfiles, which is a common CI/CD pattern for edge and restricted environments.

[mtrmac]

Thanks for your report.

I think this needs a bit deeper analysis to understand what is actually going on: per https://github.com/opencontainers/image-spec/blob/main/layer.md#opaque-whiteout , .wh..wh..opq is valid in OCI images. app/.wh..wh..opq and licenses/.wh..wh..opq exist in that on-registry image; so assuming that image can generally be pulled and run, this is something specific to the containerd import code path.

(There’s also the ~scope aspect, in that Skopeo does not, generally, edit contents of layers in that way. The whiteout file exists in the original image, so it is preserved, including its metadata, in the output. Skopeo is not generally a tool for per-file edits; that’s in Buildah’s realm. That is an implementation decision, already sort of violated in zstd:chunked.)

[marratj]

Hm, this actually works just fine for me. I use *.ociarchive files as input and use the docker-archive:/var/lib/rancher/k3s/agent/images/${IMAGE_FILENAME}.tar destination. Those images are then imported just fine on K3s service startup.

But I use the --additional-tag flag instead of specifying the image tag as additional reference in the docker-archive: destination like so:

skopeo copy --additional-tag "registry.com/server/${IMAGE_NAME}:${IMAGE_TAG}" oci-archive:${OCIARCHIVE}.source docker-archive:/var/lib/rancher/k3s/agent/images/${IMAGE_FILENAME}.tar

EDIT: I actually tried your example above and this also works just fine for me:

skopeo copy --insecure-policy \
    docker://quay.io/jetstack/cert-manager-controller:v1.19.2 \
    docker-archive:/var/lib/rancher/k3s/agent/images/cert-manager.tar:quay.io/jetstack/cert-manager-controller:v1.19.2

Leads to the following on k3s startup:

INFO[0002] Importing images from /var/lib/rancher/k3s/agent/images/cert-manager.tar
INFO[0004] Imported quay.io/jetstack/cert-manager-controller:v1.19.2
INFO[0004] Imported images from /var/lib/rancher/k3s/agent/images/cert-manager.tar in 1.708956923s

And a ctr -n k8s.io image ls also shows the imported image just fine.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.