Skip to content

Commit f22acdf

Browse files
wrkodeclaude
wrkode
and
claude
committed
test(e2e): drop in-container kubeadm upgrade scenario; VM-only boundary (ADR-13-B1)
The nightly run showed the in-container in-place upgrade scenario is the only one that requires the harness to be a genuinely functional pod-scheduling cluster: `kubeadm upgrade apply` preflight needs a Ready node (CNI) AND an upgrade-health-check Job whose pod must schedule + complete. That property is the least provider-specific (the provider just invokes kubeadm) and is already validated on the libvirt VM upgrade smoke; forcing the minimal container to be a functional data plane (hand-rolled CNI or a kindnet dependency) adds the most fragility for the least signal. Per ADR-13-B1 (staff-architect ratified), extend the VM-only upgrade coverage boundary to include the kubeadm-layer in-place upgrade and drop the in-container scenario. #4202's acceptance is met by (i) pre-membership failure-status and (ii) multi-CP stacked-etcd HA in-container (both proven green), and (iii) the kubeadm upgrade on the VM boundary. - Remove test/e2e/nightly_upgrade_test.go and its exclusive helpers CopyInto + extractBinariesFromImage from nodecontainer.go. - nightly.yml: drop the higher-minor resolve + the two higher-minor image builds and the E2E_UPGRADE_TO_* env; one base build per leg; timeout 150 -> 120. - Makefile: drop the E2E_UPGRADE_TO_* knobs from e2e-nightly. Re-verified after removal: HA scenario still PASSES end-to-end (2 CP nodes, 2 etcd members); go vet -tags "e2e nightly" clean; failure-status + HA remain green. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: William Rizzo <william.rizzo@gmail.com>
1 parent b5b22c4 commit f22acdf

4 files changed

Lines changed: 18 additions & 368 deletions

File tree

‎.github/workflows/nightly.yml‎

Lines changed: 12 additions & 82 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11
name: Nightly E2E
22

33
# ADR-13 Tier-2 (slice E4), kairos-io/kairos#4202: the heavier e2e scenarios that
4-
# exceed the per-PR budget -- multi-control-plane stacked-etcd HA bring-up, the
5-
# pre-membership failure-status path, and a kubeadm-layer in-place minor upgrade.
6-
# These run on a nightly schedule and on demand (workflow_dispatch), NOT per-PR.
4+
# exceed the per-PR budget -- multi-control-plane stacked-etcd HA bring-up and the
5+
# pre-membership failure-status path. (The kubeadm-layer in-place upgrade is
6+
# validated on the libvirt VM boundary, not in-container -- ADR-13-B1.) These run
7+
# on a nightly schedule and on demand (workflow_dispatch), NOT per-PR.
78
#
89
# Gating: the nightly-only scenarios are behind `//go:build e2e && nightly`, so the
910
# per-PR `e2e` job (ci.yml, `-tags e2e`) never compiles them. This workflow runs
@@ -67,17 +68,13 @@ jobs:
6768
echo "minors=$minors" >> "$GITHUB_OUTPUT"
6869
6970
# Tier-2 nightly e2e: real kubeadm in privileged systemd node containers. Per
70-
# supported minor leg, this job:
71-
# - builds the kairos-kubeadm base + e2e node image at the LOWER minor;
72-
# - if a higher adjacent minor exists in the supported window, ALSO builds a
73-
# second e2e node image at that HIGHER minor and exposes it to the test via
74-
# E2E_UPGRADE_TO_NODE_IMAGE / E2E_UPGRADE_TO_VERSION so the in-place upgrade
75-
# scenario runs N -> N+1 (otherwise that one scenario self-skips);
76-
# - runs the full `go test -tags "e2e nightly"` suite.
71+
# supported minor leg, this job builds the kairos-kubeadm base + e2e node image
72+
# and runs the full `go test -tags "e2e nightly"` suite (multi-CP stacked-etcd HA
73+
# and pre-membership failure-status, plus the Tier-1 scenarios). The kubeadm-layer
74+
# upgrade is validated on the libvirt VM boundary, not here (ADR-13-B1).
7775
#
7876
# The from-source containerd+kubelet build of the Hadron base dominates the
79-
# wall-clock; building TWO bases on the upgrade legs is why timeout-minutes is
80-
# generous. Legs are independent (fail-fast: false).
77+
# wall-clock. Legs are independent (fail-fast: false).
8178
nightly-e2e:
8279
name: nightly e2e (k8s ${{ matrix.minor }})
8380
needs: discover
@@ -88,10 +85,9 @@ jobs:
8885
permissions:
8986
contents: read
9087
runs-on: ubuntu-latest
91-
# Two from-source base builds (lower + higher) on the upgrade legs plus the
92-
# heavier multi-container HA/upgrade scenarios. Generous ceiling; the test
93-
# harness itself is bounded (#4099-1) well under this.
94-
timeout-minutes: 150
88+
# One from-source base build plus the heavier multi-container HA scenario.
89+
# Generous ceiling; the test harness itself is bounded (#4099-1) well under this.
90+
timeout-minutes: 120
9591
strategy:
9692
fail-fast: false
9793
matrix:
@@ -135,47 +131,6 @@ jobs:
135131
echo "kcommit=${kcommit}" >> "$GITHUB_OUTPUT"
136132
echo "crictl=${crictl}" >> "$GITHUB_OUTPUT"
137133
138-
- name: Resolve higher adjacent minor (for the in-place upgrade scenario)
139-
id: higher
140-
run: |
141-
set -euo pipefail
142-
# The next adjacent minor IS the upgrade target. It is only valid if it is
143-
# inside the supported window (kubeadm only supports +1 minor per upgrade,
144-
# and ADR-3 refuses out-of-window targets). If stable-<higher>.txt does not
145-
# resolve (e.g. this leg is the newest minor), leave the outputs empty so
146-
# the upgrade scenario self-skips rather than fakes a no-op.
147-
major="$(echo "${{ matrix.minor }}" | cut -d. -f1)"
148-
minor="$(echo "${{ matrix.minor }}" | cut -d. -f2)"
149-
hi_mm="${major}.$((minor + 1))"
150-
window='${{ needs.discover.outputs.minors }}'
151-
if ! echo "$window" | jq -e --arg m "$hi_mm" 'index($m)' >/dev/null; then
152-
echo "no higher minor ${hi_mm} in supported window ${window}; upgrade scenario will self-skip on this leg"
153-
echo "k8s=" >> "$GITHUB_OUTPUT"
154-
exit 0
155-
fi
156-
if ! k8s="$(curl -fsSL "https://dl.k8s.io/release/stable-${hi_mm}.txt")"; then
157-
echo "could not resolve stable patch for higher minor ${hi_mm}; upgrade scenario will self-skip"
158-
echo "k8s=" >> "$GITHUB_OUTPUT"
159-
exit 0
160-
fi
161-
case "$k8s" in
162-
v${hi_mm}.*) : ;;
163-
*) echo "unexpected patch ${k8s} for higher minor ${hi_mm}; upgrade scenario will self-skip"; echo "k8s=" >> "$GITHUB_OUTPUT"; exit 0 ;;
164-
esac
165-
kcommit="$(git ls-remote https://github.com/kubernetes/kubernetes "refs/tags/${k8s}^{}" | cut -f1)"
166-
if [ -z "$kcommit" ]; then
167-
kcommit="$(git ls-remote https://github.com/kubernetes/kubernetes "refs/tags/${k8s}" | cut -f1)"
168-
fi
169-
case "$kcommit" in
170-
?*) : ;;
171-
*) echo "could not resolve commit SHA for higher ${k8s}; upgrade scenario will self-skip"; echo "k8s=" >> "$GITHUB_OUTPUT"; exit 0 ;;
172-
esac
173-
crictl="v${hi_mm}.0"
174-
echo "higher minor: k8s=${k8s} kcommit=${kcommit} crictl=${crictl}"
175-
echo "k8s=${k8s}" >> "$GITHUB_OUTPUT"
176-
echo "kcommit=${kcommit}" >> "$GITHUB_OUTPUT"
177-
echo "crictl=${crictl}" >> "$GITHUB_OUTPUT"
178-
179134
- name: Build kairos-kubeadm base image (lower minor)
180135
run: |
181136
make image \
@@ -192,30 +147,10 @@ jobs:
192147
BASE_IMAGE=kairos-kubeadm:${{ steps.lower.outputs.k8s }} \
193148
E2E_NODE_IMAGE=kairos-kubeadm-e2e-node:${{ steps.lower.outputs.k8s }}
194149
195-
- name: Build kairos-kubeadm base image (higher minor, upgrade target)
196-
if: steps.higher.outputs.k8s != ''
197-
run: |
198-
make image \
199-
VERSION=ci-upgrade \
200-
KUBERNETES_VERSION=${{ steps.higher.outputs.k8s }} \
201-
KUBERNETES_COMMIT=${{ steps.higher.outputs.kcommit }} \
202-
CRICTL_VERSION=${{ steps.higher.outputs.crictl }} \
203-
IMAGE=kairos-kubeadm:${{ steps.higher.outputs.k8s }}
204-
205-
- name: Build e2e node image (higher minor, upgrade target)
206-
if: steps.higher.outputs.k8s != ''
207-
run: |
208-
make e2e-node-image \
209-
KUBERNETES_VERSION=${{ steps.higher.outputs.k8s }} \
210-
BASE_IMAGE=kairos-kubeadm:${{ steps.higher.outputs.k8s }} \
211-
E2E_NODE_IMAGE=kairos-kubeadm-e2e-node:${{ steps.higher.outputs.k8s }}
212-
213150
- name: Run nightly e2e suite (e2e + nightly)
214151
run: |
215152
E2E_NODE_IMAGE=kairos-kubeadm-e2e-node:${{ steps.lower.outputs.k8s }} \
216153
E2E_KUBERNETES_VERSION=${{ steps.lower.outputs.k8s }} \
217-
E2E_UPGRADE_TO_NODE_IMAGE=${{ steps.higher.outputs.k8s != '' && format('kairos-kubeadm-e2e-node:{0}', steps.higher.outputs.k8s) || '' }} \
218-
E2E_UPGRADE_TO_VERSION=${{ steps.higher.outputs.k8s }} \
219154
go test -tags "e2e nightly" -count=1 -timeout 90m -v ./test/e2e/...
220155
221156
- name: Prune e2e containers, volumes, and images on failure
@@ -226,9 +161,4 @@ jobs:
226161
docker image rm -f \
227162
"kairos-kubeadm-e2e-node:${{ steps.lower.outputs.k8s }}" \
228163
"kairos-kubeadm:${{ steps.lower.outputs.k8s }}" || true
229-
if [ -n "${{ steps.higher.outputs.k8s }}" ]; then
230-
docker image rm -f \
231-
"kairos-kubeadm-e2e-node:${{ steps.higher.outputs.k8s }}" \
232-
"kairos-kubeadm:${{ steps.higher.outputs.k8s }}" || true
233-
fi
234164
docker volume prune -f || true

‎Makefile‎

Lines changed: 6 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -40,13 +40,10 @@ IMAGE ?= kairos-kubeadm:$(VERSION)
4040
BASE_IMAGE ?= kairos-kubeadm:$(KUBERNETES_VERSION)
4141
E2E_NODE_IMAGE ?= kairos-kubeadm-e2e-node:$(KUBERNETES_VERSION)
4242
E2E_TIMEOUT ?= 40m
43-
# Tier-2 nightly suite runs heavier multi-container scenarios (HA, upgrade), so it
44-
# carries a longer wall-clock ceiling than the per-PR Tier-1 suite.
43+
# Tier-2 nightly suite runs heavier multi-container scenarios (HA, failure-status),
44+
# so it carries a longer wall-clock ceiling than the per-PR Tier-1 suite. (The
45+
# kubeadm-layer upgrade is validated on the libvirt VM boundary, not here -- ADR-13-B1.)
4546
E2E_NIGHTLY_TIMEOUT ?= 90m
46-
# Higher-minor node image + version for the in-place kubeadm upgrade scenario; set
47-
# by the nightly workflow. Empty -> the upgrade scenario self-skips (never faked).
48-
E2E_UPGRADE_TO_NODE_IMAGE ?=
49-
E2E_UPGRADE_TO_VERSION ?=
5047

5148
.PHONY: all build test vet fmt fmt-check lint tidy verify-pins image clean e2e-node-image e2e e2e-nightly
5249

@@ -137,16 +134,13 @@ e2e: e2e-node-image
137134
$(GO) test -tags e2e -timeout $(E2E_TIMEOUT) -v ./test/e2e/...
138135

139136
## e2e-nightly: run the heavier Tier-2 (ADR-13 E4) scenarios that are too slow for
140-
## per-PR (CP-join/HA, pre-membership failure-status, kubeadm-layer upgrade). It
141-
## mirrors `e2e` but compiles the nightly-gated files too (-tags "e2e nightly", so
137+
## per-PR (multi-CP stacked-etcd HA, pre-membership failure-status). It mirrors
138+
## `e2e` but compiles the nightly-gated files too (-tags "e2e nightly", so
142139
## //go:build e2e && nightly files are included) and allows a longer timeout. The
143-
## upgrade scenario also consumes E2E_UPGRADE_TO_NODE_IMAGE / E2E_UPGRADE_TO_VERSION
144-
## (the higher-minor node image); when unset that one scenario self-skips.
140+
## kubeadm-layer upgrade is validated on the libvirt VM boundary, not here (ADR-13-B1).
145141
e2e-nightly: e2e-node-image
146142
E2E_NODE_IMAGE=$(E2E_NODE_IMAGE) \
147143
E2E_KUBERNETES_VERSION=$(KUBERNETES_VERSION) \
148-
E2E_UPGRADE_TO_NODE_IMAGE=$(E2E_UPGRADE_TO_NODE_IMAGE) \
149-
E2E_UPGRADE_TO_VERSION=$(E2E_UPGRADE_TO_VERSION) \
150144
$(GO) test -tags "e2e nightly" -count=1 -timeout $(E2E_NIGHTLY_TIMEOUT) -v ./test/e2e/...
151145

152146
## clean: remove build artifacts

0 commit comments

Comments
 (0)