|
| 1 | +# Testing |
| 2 | + |
| 3 | +How provider-kubernetes is tested, what each layer proves, how to run the suites |
| 4 | +locally, and -- just as important -- the explicit boundary of what is **not** |
| 5 | +covered automatically and must be validated by hand. |
| 6 | + |
| 7 | +## Philosophy |
| 8 | + |
| 9 | +Two design principles drive the test strategy: |
| 10 | + |
| 11 | +- **Hardware-free testability (principle 6).** Config generation is pure functions |
| 12 | + over typed inputs; the reconcile is a pure `Plan` plus a bounded executor. We |
| 13 | + test the *behavior* (the produced config/objects, the planned actions, the |
| 14 | + on-disk status), never generated strings. |
| 15 | +- **Fail fast, never hang (principle 4 / #4099-1).** This applies to the tests too: |
| 16 | + every wait is bounded, and the e2e harness always tears its containers down. |
| 17 | + |
| 18 | +The result is a layered pyramid: fast pure tests on every change, real-kubeadm |
| 19 | +end-to-end tests on every PR, and a small set of full-VM smokes that remain |
| 20 | +manual by necessity. |
| 21 | + |
| 22 | +## The layers at a glance |
| 23 | + |
| 24 | +| Layer | Command | Runs | What it proves | Hardware | |
| 25 | +|-------|---------|------|----------------|----------| |
| 26 | +| Unit / behavior | `make test` | every PR (gates job) | config generation, pure `Plan`, version skew, prober parsing, status `BuildStatus`, credential/PKI math | none | |
| 27 | +| End-to-end (e2e) | `make e2e` | every PR (e2e matrix, 1.34/1.35/1.36) | real `kubeadm`/`kubelet`/`containerd` accept our config and converge: init, join, reset, the refuse-guards, status, externally-managed join | Docker + privileged container | |
| 28 | +| Manual VM smokes | by hand (libvirt) | before notable releases / risky changes | full Kairos image boot, A/B reboot upgrade, HA failover, kube-vip, TPM2/kcrypt | KVM/libvirt host | |
| 29 | + |
| 30 | +## Unit and behavior tests |
| 31 | + |
| 32 | +```sh |
| 33 | +make test # go test ./... with coverage |
| 34 | +make vet |
| 35 | +make lint # golangci-lint (v2 config) |
| 36 | +make fmt-check |
| 37 | +``` |
| 38 | + |
| 39 | +These cover the bulk of the logic and run in seconds. The e2e files are behind a |
| 40 | +`//go:build e2e` tag, so `make test` / `go test ./...` never compile or run them -- |
| 41 | +the fast gate stays fast. |
| 42 | + |
| 43 | +## End-to-end tests |
| 44 | + |
| 45 | +The e2e suite (`test/e2e/`, behind `//go:build e2e`) proves the layer unit tests |
| 46 | +cannot: that a real `kubeadm` accepts our generated v1beta4 config and that a real |
| 47 | +node actually converges, joins, and resets. |
| 48 | + |
| 49 | +### Mechanism |
| 50 | + |
| 51 | +Each scenario runs the provider against a **real** kubeadm/kubelet/containerd |
| 52 | +inside a kind-style **privileged systemd "node container"**: |
| 53 | + |
| 54 | +- The node image (`Dockerfile.e2e-node`) is `FROM`-derived from a built |
| 55 | + `kairos-kubeadm` image, so it reuses the exact checksum-verified toolchain we |
| 56 | + ship -- nothing is re-downloaded. It only adds the kind-blessed tweaks needed to |
| 57 | + run systemd as PID 1 with the systemd cgroup driver on cgroup v2 (masking |
| 58 | + container-hostile units, `--fail-swap-on=false`, stateful-dir volumes). |
| 59 | +- The harness (`test/e2e/nodecontainer.go`) starts the container `--privileged |
| 60 | + --cgroupns=private` with tmpfs `/run` + `/tmp`, anonymous volumes for |
| 61 | + `/var/lib/{containerd,kubelet,etcd}`, and read-only `/lib/modules` + `/boot` |
| 62 | + (the latter lets kubeadm's `SystemVerification` preflight read the kernel config |
| 63 | + **without** weakening the provider's preflight). |
| 64 | +- It drives the **real** entrypoints, exactly as production does: it serializes a |
| 65 | + `clusterplugin.Cluster` to the same `0600` tmpfs path the yip stage uses and runs |
| 66 | + `agent-provider-kubernetes reconcile --cluster-file=...` (and `reset`, |
| 67 | + `mint-join`) via `docker exec`. There is **no parallel code path** -- the test |
| 68 | + exercises the shipped binary and the real on-disk contract. |
| 69 | +- Assertions read back the typed `status.yaml`, the Node annotations, and |
| 70 | + `kubectl get nodes` -- behavior, not strings. Teardown (`docker rm -fv`) is |
| 71 | + guaranteed via `t.Cleanup`, reaping the anonymous volumes. |
| 72 | + |
| 73 | +No VM and no `/dev/kvm` are required -- only Docker with privileged containers. |
| 74 | + |
| 75 | +### Scenarios (Tier 1, per-PR) |
| 76 | + |
| 77 | +| Test | Sets up | Proves | |
| 78 | +|------|---------|--------| |
| 79 | +| `TestSingleNodeInitConverges` | one node, `role: init` | `kubeadm init` converges: apiserver healthy, node registered, `status.yaml` `phase=Converged` (mode `0640`), all seven `provider-kubernetes.kairos.io/*` annotations present | |
| 80 | +| `TestWorkerJoin` | CP + worker (2 containers) | a real CA-pinned worker join: `mint-join` mints material on the CP, the worker joins with the SPKI pin; both nodes register; `UnsafeSkipCAVerification` never set | |
| 81 | +| `TestExternallyManagedControlPlaneJoin` | external CP via plain `kubeadm init` + worker | joining a CP the provider did **not** bootstrap (#4099-5): worker joins with only the operator's CA **PEM** (`ca_certs`); the provider derives the pin itself | |
| 82 | +| `TestResetCleansArtifacts` | init, then `reset` | the reset path removes the authoritative artifacts (PKI, `admin.conf`, etcd member data, kubelet join marker) | |
| 83 | +| `TestInitClobberRefusal` | live CP + a second `role: init` node | the init-clobber guard (#4099-5) fires: `refuse-init`, terminal, `status.reason=InitRefused`, the existing cluster untouched | |
| 84 | +| `TestUpgradeSkewRefusal` | converged node, skip-level pin | the upgrade skew guard fires: `refuse-upgrade`, terminal, `status.reason=UpgradeRefused`, nothing destructive runs | |
| 85 | + |
| 86 | +### Running locally |
| 87 | + |
| 88 | +Prerequisites: |
| 89 | + |
| 90 | +- Docker with the ability to run `--privileged` containers (cgroup v2 host). |
| 91 | +- `/boot/config-$(uname -r)` and `/lib/modules` present (standard on Ubuntu) for |
| 92 | + the kubeadm preflight. |
| 93 | +- **No** KVM / nested virtualization. |
| 94 | + |
| 95 | +The e2e node image is `FROM`-derived from a `kairos-kubeadm` base image, so build |
| 96 | +that base once for the version you want, then run the suite: |
| 97 | + |
| 98 | +```sh |
| 99 | +# 1. Build the base image (a few minutes; bundles the toolchain). |
| 100 | +make image KUBERNETES_VERSION=v1.34.0 VERSION=v1.34.0 IMAGE=kairos-kubeadm:v1.34.0 |
| 101 | + |
| 102 | +# 2. Build the node image and run the suite (make e2e builds the node image for you). |
| 103 | +make e2e KUBERNETES_VERSION=v1.34.0 |
| 104 | +``` |
| 105 | + |
| 106 | +Notes: |
| 107 | + |
| 108 | +- `make e2e` depends on `e2e-node-image`; if the base image is missing it prints |
| 109 | + the exact `make image` command to run first. |
| 110 | +- Each scenario creates its own container(s) and cleans them up. To run one: |
| 111 | + `E2E_NODE_IMAGE=kairos-kubeadm-e2e-node:v1.34.0 E2E_KUBERNETES_VERSION=v1.34.0 |
| 112 | + go test -tags e2e -count=1 -run TestWorkerJoin -v ./test/e2e/...`. |
| 113 | +- Each scenario pre-pulls the control-plane images before reconcile so a cold |
| 114 | + containerd pull does not race the reconcile budget (a runtime nuance, not a |
| 115 | + production change). |
| 116 | +- A converged node shows `NotReady` because the provider installs no CNI by |
| 117 | + design; the scenarios assert on convergence + registration, not `Ready`. |
| 118 | +- Watch disk: the base + node images are ~3 GB each, and anonymous volumes |
| 119 | + accumulate if a run is killed mid-flight. `docker volume prune` and |
| 120 | + `docker image prune` clean up. |
| 121 | + |
| 122 | +### In CI |
| 123 | + |
| 124 | +The `e2e` job in `.github/workflows/ci.yml` runs on every PR, as a matrix over the |
| 125 | +full supported window (1.34 / 1.35 / 1.36) in parallel. For each minor it resolves |
| 126 | +the latest patch, builds the base + node image, and runs the whole suite. A |
| 127 | +`timeout-minutes` backstop guarantees it never hangs. |
| 128 | + |
| 129 | +Security posture (this job runs a `--privileged` container on |
| 130 | +attacker-influenceable PR code, so the controls matter): |
| 131 | + |
| 132 | +- **GitHub-hosted ephemeral runners only** -- never self-hosted. |
| 133 | +- **`pull_request` trigger** (not `pull_request_target`), so fork PRs get no |
| 134 | + secrets. |
| 135 | +- **Explicit read-only token** (`permissions: contents: read`); no secrets, no |
| 136 | + `packages: write`. |
| 137 | + |
| 138 | +These conditions -- not the container flags -- are what contain a hostile PR. |
| 139 | + |
| 140 | +## Nightly tier (planned) |
| 141 | + |
| 142 | +A heavier `nightly` workflow for the multi-step scenarios that are too slow for |
| 143 | +per-PR (multi-control-plane stacked-etcd HA bring-up, the pre-membership |
| 144 | +failure-status path under a real unreachable endpoint, and a kubeadm-layer |
| 145 | +1.34 -> 1.35 in-place upgrade) is designed but **not yet wired**. Until it lands, |
| 146 | +those paths are covered by unit tests plus the manual VM smokes below. |
| 147 | + |
| 148 | +## Coverage boundary -- what e2e does NOT cover |
| 149 | + |
| 150 | +The container-based e2e cannot reproduce events that need a real machine, a reboot, |
| 151 | +or hardware. These are validated **by hand** on a KVM/libvirt host and are an |
| 152 | +explicit, documented boundary -- not a gap to be silently closed by CI: |
| 153 | + |
| 154 | +- **Full Kairos image boot.** The e2e runs the provider binary in a node |
| 155 | + container; it does not boot a Kairos ISO, install to disk, or exercise the |
| 156 | + `network.after` yip stage end to end. |
| 157 | +- **A/B image upgrade across a real reboot.** The kubeadm-layer upgrade is |
| 158 | + testable in a container; the Kairos A/B swap + reboot (and the kubelet-config |
| 159 | + repair that the swap-then-reboot ordering triggers) is not. |
| 160 | +- **HA failover.** Powering off a control plane and proving the cluster survives on |
| 161 | + remaining quorum needs real VMs. |
| 162 | +- **kube-vip / cross-host networking.** A real VIP and a multi-host LAN. |
| 163 | +- **TPM2 / kcrypt at-rest encryption.** Real hardware/firmware. |
| 164 | + |
| 165 | +The manual smokes use libvirt + auroraboot-built ISOs from this repo's image; the |
| 166 | +maintainer keeps the run logs and results locally (under `build/vmtest/`, which is |
| 167 | +git-ignored). When you change anything in those areas, run the relevant smoke |
| 168 | +before release and record the outcome. |
| 169 | + |
| 170 | +## See also |
| 171 | + |
| 172 | +- [Lifecycle and reset](./lifecycle.md) -- the reconcile model the e2e exercises. |
| 173 | +- [Configuration reference](./configuration.md) -- the `cluster` contract the e2e |
| 174 | + serializes, including externally-managed control planes. |
| 175 | +- [Node status](./status.md) -- the `status.yaml` + annotations the e2e asserts on. |
0 commit comments