Skip to content

feat: pre-bundle control-plane images for air-gap first boot (ADR-16)#22

Merged
wrkode merged 1 commit into
mainfrom
feat/prebundle-cp-images
Jun 30, 2026
Merged

feat: pre-bundle control-plane images for air-gap first boot (ADR-16)#22
wrkode merged 1 commit into
mainfrom
feat/prebundle-cp-images

Conversation

@wrkode

@wrkode wrkode commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

What

Pre-bundle the kubeadm control-plane images into the OS image and import them into
containerd at boot, so a provider-kubernetes node converges on first boot with
no registry access
(air-gap). Implements ADR-16; resolves the cold-init image
strategy and de-risks the never-hang budget (kubeadm init no longer pulls).

Closes kairos-io/kairos#4205. Addresses kairos-io/kairos#4204 (never-hang budget).

How

  • Build (Dockerfile): a new image-bundler stage installs a
    checksum-verified static crane, resolves the control-plane image set from the
    bundled kubeadm config images list (the same invocation as the pause pin, so
    the two can never drift), and crane pulls each ref to a ctr-importable tarball
    under /opt/provider-kubernetes/images/. Static ctr is added to the containerd
    build so the import works on the musl Hadron base. Build-time assertion: >=5
    tarballs and the pause image present.
  • Import (internal/imageimport + import-images subcommand): imports every
    *.tar into containerd's k8s.io namespace via argv ctr (no shell), bounded
    by context, idempotent, a no-op when nothing is bundled, attempts every tarball
    and reports the first error.
  • Ordering (internal/provider): the import is emitted as a yip step ordered
    before the reconcile step in the same network.after stage, so images are in
    containerd before kubeadm init/join runs. A systemd oneshot alone races the
    kairos-agent reconcile stage. A systemd/provider-kubernetes-image-import.service
    oneshot is kept (ordered before kubelet) for the kubelet/reboot path.
  • Budget (internal/reconcile/budget_test.go): regression test locking the
    #4204 never-hang invariant — Total is the binding ceiling, below
    PerAttempt*MaxAttempts.

Validation

go build / go test ./... / vet / golangci-lint all green (imageimport 95%
coverage).

End-to-end on an isolated (no-NAT) libvirt network on the Hadron v0.4.0 musl
base:

  • Air-gap confirmed: numeric egress to 1.1.1.1 fails (registry unreachable).
  • Node reaches phase=Converged / outcome=success; v1.34.0,
    containerd://2.1.4, Hadron Linux, kernel 7.1.0-hadron.
  • All 5 control-plane static pods + kube-proxy + etcd Running; all 7
    provider-kubernetes.kairos.io/* annotations present.
  • Zero containerd image-pull events — every image came from the local import.
  • Reboot-persistence: images present and node re-converges after reboot.

The first air-gap boot caught a real ordering bug (the systemd oneshot raced the
reconcile yip stage; kubeadm init ran with 1/7 images) — fixed by the yip in-stage
ordering above and re-validated.

Notes / follow-ups (not in this PR)

  • P5 (deferred, security-gated): digest-pin the build-time crane pull
    (content-addressed, reproducible) — needs security-architect review.
  • Consider defaulting localAPIEndpoint.advertiseAddress to control_plane_host
    for routeless air-gap nodes (kubeadm rejects a 0.0.0.0 bind with no default
    route).
  • CNI is intentionally not bundled (operator's choice).

…4205, #4204)

Bake the kubeadm control-plane images into the OS image and import them into
containerd at boot, so a first boot converges with no registry access (ADR-16).
This resolves the cold-init image strategy and de-risks the #4204 never-hang
budget (kubeadm init no longer pulls).

- Dockerfile: new image-bundler stage installs a checksum-verified static crane,
  resolves the image set from the bundled kubeadm (the same invocation as the
  pause pin, so the two cannot drift), and crane-pulls each ref to a
  ctr-importable tarball under /opt/provider-kubernetes/images/. Static ctr is
  added to the containerd build so the import works on the musl Hadron base.
  Build-time assertion: >=5 tarballs and the pause image present.
- internal/imageimport + 'import-images' subcommand: import every *.tar into
  containerd's k8s.io namespace via argv ctr (no shell), bounded, idempotent,
  no-op when nothing is bundled, attempt-all-and-report.
- provider: emit the import as a yip step ordered BEFORE the reconcile step in
  the same network.after stage, so images are in containerd before kubeadm init
  runs. A systemd oneshot alone races the kairos-agent reconcile stage (proven in
  the air-gap VM: kubeadm init ran with 1/7 images and failed); the yip in-stage
  ordering is deterministic. The oneshot is kept for the kubelet/reboot path.
- reconcile: budget regression test locking the #4204 never-hang invariant
  (Total is the binding ceiling, below PerAttempt*MaxAttempts).

Validated end to end on an isolated (no-NAT) libvirt network on the Hadron
v0.4.0 musl base: node reaches Converged with zero containerd image pulls, all
control-plane pods Running, and re-converges after reboot.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: William Rizzo <william.rizzo@gmail.com>

@wrkode wrkode left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wrkode
wrkode merged commit 90abd7a into main Jun 30, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

provider-kubernetes: assess cold-init image pulls vs the reconcile budget

1 participant