Skip to content

feat(build): cosign-verify + digest-pin pre-bundled control-plane images (ADR-16 P5)#25

Merged
wrkode merged 1 commit into
mainfrom
feat/p5-cosign-verify-bundled-images
Jun 30, 2026
Merged

feat(build): cosign-verify + digest-pin pre-bundled control-plane images (ADR-16 P5)#25
wrkode merged 1 commit into
mainfrom
feat/p5-cosign-verify-bundled-images

Conversation

@wrkode

@wrkode wrkode commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

What

Harden the air-gap image bundle (ADR-16 P5, the deferred security-gated slice of
kairos-io/kairos#4205). The pre-bundled control-plane images land in the immutable OS
and are ctr imported at boot, bypassing any runtime CRI image policy — so provenance
is established at build time via cosign signature verification + digest pinning.

How

The image-bundler stage, for each image kubeadm requires:

  1. resolves the floating tag to an immutable digest once (crane digest),
  2. cosign-verifies that digest against the Kubernetes krel keyless identity
    (krel-trust@k8s-releng-prod.iam.gserviceaccount.com / https://accounts.google.com),
  3. pulls the same digest (crane pull ref@digest) — verify and pull are bound to
    one digest (TOCTOU-safe).

cosign and crane are version-pinned and checksum-verified at install.

Best-effort verification with a hard floor

registry.k8s.io signature coverage is empirically incomplete: kube-apiserver,
kube-controller-manager, and kube-proxy carry no krel signature (confirmed by
identical digest + absent .sig/referrers at the canonical us-central1-docker.pkg.dev
backend — a known upstream condition, k8s#129199), while kube-scheduler, coredns,
pause, etcd do verify. So:

  • signature verifies → recorded verified: true.
  • cosign reports exactly no signatures found → recorded verified: false / no-upstream-signature (digest-pinned only; still content-addressed).
  • any other cosign failure (present-but-invalid signature, network/Fulcio/Rekor/TUF
    error) → hard fail.
  • floor (build-blocking): pause + etcd + coredns AND ≥1 kube-* component MUST
    verify — each a positive pass (a timeout can't satisfy the floor).

Auditability

A baked /opt/provider-kubernetes/images/images.lock records ref → digest plus the
per-image verified flag/reason. The release pipeline converts it to a CycloneDX SBOM
(build/gen-bundled-sbom.py) and attests it, so each released image attests exactly
which control-plane images it baked and at what trust level (ADR-15).

Validation

  • Full make image green; the bundler stage exercised cosign against registry.k8s.io
    for real → 7 images, 4 signature-verified, 3 recorded no-upstream-signature.
  • images.lock + 7 tarballs confirmed in the final image; sh -n + shellcheck clean;
    release YAML valid; Go gates green.

Security review

security-architect gate: PASS — B1a/B1b/B1c (with R1: precise no-signature match,
R2: positive-pass floor), B2–B7 all verified in code. Residual risk recorded
(R-resid-1: the kube-* trio is digest-pinned-only and upstream-unsigned; B1a
auto-tightens to require their signatures if/when upstream starts signing them.
R-resid-2: backlog CI canary to detect that change). Two should-fix nits actioned
inline (anchored floor matches; documented closed-enum verifyReason).

…ges (ADR-16 P5)

Harden the air-gap image bundle (kairos#4205 P5): the pre-bundled control-plane
images land in the immutable OS and are ctr-imported at boot, bypassing any
runtime CRI image policy, so provenance is established at build time.

The image-bundler now, per image kubeadm requires:
  1. resolves the floating tag to an immutable digest once (crane digest),
  2. cosign-verifies that digest against the Kubernetes krel keyless identity
     (krel-trust@k8s-releng-prod.iam.gserviceaccount.com / accounts.google.com),
  3. pulls the SAME digest (TOCTOU-safe: verify and pull bound to one digest).
cosign and crane are version-pinned and checksum-verified at install.

registry.k8s.io signature coverage is empirically incomplete: kube-apiserver,
kube-controller-manager and kube-proxy carry no krel signature (confirmed by
identical digest + absent .sig/referrers at the canonical us-central1 backend;
k8s#129199), while kube-scheduler/coredns/pause/etcd do. So verification is
best-effort with a hard floor:
  - signature verifies            -> recorded verified=true.
  - cosign "no signatures found"  -> recorded verified=false / no-upstream-signature
                                     (digest-pinned only; still content-addressed).
  - any other cosign failure      -> hard fail (present-but-invalid sig or error).
  - floor (build-blocking): pause+etcd+coredns AND >=1 kube-* MUST verify, each a
    positive pass (a timeout cannot satisfy the floor).

A baked /opt/provider-kubernetes/images/images.lock records ref->digest plus the
per-image verified flag/reason; the release pipeline converts it to a CycloneDX
SBOM (build/gen-bundled-sbom.py) and attests it, so each released image attests
exactly which control-plane images it baked and at what trust level (ADR-15).

.gitignore/.dockerignore un-ignore the two build/ helper scripts (vmtest
artifacts stay ignored). security-architect gate: PASS (B1a/B1b/B1c with R1/R2,
B2-B7 verified in code); residual risk recorded (R-resid-1/-2).

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: William Rizzo <william.rizzo@gmail.com>

@wrkode wrkode left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wrkode
wrkode merged commit 9f6073a into main Jun 30, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant