Skip to content

Commit 7563975

Browse files
wrkodeclaude
wrkode
and
claude
authored
test: prove externally-managed control-plane join end to end (#4099-5) (#16)
Design principle 7 / issue #4099-5 (joining a control plane the provider did not bootstrap) was solid in code and unit tests but had no end-to-end proof: every prior join test/e2e joined a PROVIDER-bootstrapped CP via mint-join. Add test/e2e/external_cp_test.go: bring up a control plane with plain kubeadm init (the provider never runs on it, asserted by the absence of a provider status file), then join a worker THROUGH the provider supplying only the operator-delivered CA certificate in PEM form (cluster.ca_certs) plus a bootstrap token -- no caCertHashes. The provider derives the SPKI pin from the PEM (resolveCACertHashes, OQ-9) and joins CA-pinned; UnsafeSkipCAVerification is never set. Asserts the worker converges/joined and both nodes register on the external control plane. Runs in the existing per-PR e2e job (go test picks up the new _test.go; no ci.yml change). docs/configuration.md expands the externally-managed section to document the ca_certs -> derived-pin path and note the e2e coverage. Signed-off-by: William Rizzo <william.rizzo@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
1 parent bf43565 commit 7563975

2 files changed

Lines changed: 178 additions & 3 deletions

File tree

‎docs/configuration.md‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -111,9 +111,23 @@ Notes:
111111
## Externally-managed control planes
112112

113113
You can join a control plane the provider did not bootstrap. Supply the trust
114-
anchor explicitly: a `discovery.bootstrapToken` with `caCertHashes`, or a
115-
CA-embedded discovery file. If you supply both `CACerts` (via the cloud-config)
116-
and explicit `caCertHashes`, they are cross-validated and a mismatch fails loud.
114+
anchor explicitly, in any of these ways:
115+
116+
- `ca_certs` (the CA certificate in PEM form) with a `discovery.bootstrapToken`
117+
token: the provider derives the SPKI pin from the PEM for you. You do not need
118+
to precompute a hash.
119+
- a `discovery.bootstrapToken` with explicit `caCertHashes` (precomputed
120+
`sha256:...` SPKI pins).
121+
- a CA-embedded discovery file (`discovery.file.kubeConfigPath`).
122+
123+
If you supply both `ca_certs` and explicit `caCertHashes`, they are
124+
cross-validated and a mismatch fails loud. CA pinning is mandatory in every case;
125+
`unsafeSkipCAVerification` is never set.
126+
127+
This path is exercised end-to-end in CI: an e2e scenario stands up a control plane
128+
with plain `kubeadm init` (the provider never touches it), then joins a worker
129+
through the provider using only the operator-supplied CA PEM, asserting the
130+
provider derives the pin and the worker registers on the external control plane.
117131

118132
## Proxy environment
119133

‎test/e2e/external_cp_test.go‎

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
//go:build e2e
2+
3+
package e2e
4+
5+
// external_cp_test.go proves design principle 7 / issue #4099-5: a node joins a
6+
// control plane the provider did NOT bootstrap, using operator-supplied trust
7+
// anchors.
8+
//
9+
// This is distinct from TestWorkerJoin (scenarios_test.go), where the CP is
10+
// provider-init'd and the join material comes from `mint-join`. Here the CP is
11+
// brought up by PLAIN `kubeadm init` -- the provider never runs on it, so it is a
12+
// genuinely externally-managed control plane. The operator then hands the worker
13+
// only:
14+
// - the CP's CA certificate in PEM form (cluster.ca_certs), and
15+
// - a bootstrap token (kubeadm token create), and the endpoint.
16+
//
17+
// Crucially the worker config carries NO caCertHashes. The provider must DERIVE
18+
// the SPKI pin from the operator-supplied CA PEM (resolveCACertHashes, OQ-9/ADR-10)
19+
// and join with CA pinning -- never UnsafeSkipCAVerification. A successful join
20+
// with only the PEM supplied is the proof that the derive-and-pin path works
21+
// end-to-end against a real CA, not just in unit tests.
22+
23+
import (
24+
"strings"
25+
"testing"
26+
"time"
27+
28+
"github.com/kairos-io/kairos-sdk/clusterplugin"
29+
)
30+
31+
// externalKubeadmInit brings up a single-node control plane WITHOUT the provider:
32+
// it runs `kubeadm init` directly in the container with plain flags. The provider
33+
// binary is never invoked here, so the resulting CP is "externally managed" from
34+
// the provider's point of view. Waits for the apiserver to be healthy.
35+
func externalKubeadmInit(t *testing.T, nc *nodeContainer, ip, k8sVer string) {
36+
t.Helper()
37+
out, err := nc.ExecTimeout(reconcileTimeout,
38+
"kubeadm", "init",
39+
"--apiserver-advertise-address", ip,
40+
"--pod-network-cidr", "10.244.0.0/16",
41+
"--kubernetes-version", k8sVer,
42+
"--cri-socket", "unix:///run/containerd/containerd.sock",
43+
)
44+
if err != nil {
45+
t.Fatalf("plain kubeadm init (external CP): %v\n--- output ---\n%s", err, out)
46+
}
47+
nc.waitFor(t, "external CP apiserver /healthz ok", 120*time.Second, func() bool {
48+
o, e := nc.execErr("kubectl", "--kubeconfig", adminConf, "get", "--raw", "/healthz")
49+
return e == nil && strings.TrimSpace(o) == "ok"
50+
})
51+
}
52+
53+
// TestExternallyManagedControlPlaneJoin (design principle 7 / #4099-5): a worker
54+
// joins a control plane the provider did not create, with the provider deriving
55+
// the CA pin from an operator-supplied CA PEM.
56+
//
57+
// Flow:
58+
// 1. Start the CP container; bring up the CP with PLAIN kubeadm init (no provider).
59+
// 2. Operator extracts trust anchors out-of-band: the CA cert PEM
60+
// (/etc/kubernetes/pki/ca.crt) and a fresh bootstrap token (kubeadm token
61+
// create). NO SPKI hash is computed by the operator.
62+
// 3. Build a role=worker Cluster supplying ONLY ca_certs (the PEM) + the token +
63+
// endpoint -- no caCertHashes. The provider must derive the pin from the PEM.
64+
// 4. Start a second container (worker), reconcile via the provider.
65+
// 5. Assert: worker converges/joined; the externally-managed CP shows 2 nodes;
66+
// and the CP never had the provider run on it (no provider status file) --
67+
// confirming it is genuinely externally managed.
68+
func TestExternallyManagedControlPlaneJoin(t *testing.T) {
69+
k8sVer := kubernetesVersion()
70+
71+
// 1. Externally-managed CP: plain kubeadm init, provider never involved.
72+
cpNC := startNode(t, uniqueName("ext-cp"))
73+
cpIP := cpNC.IP(t)
74+
prepullControlPlaneImages(t, cpNC, k8sVer)
75+
externalKubeadmInit(t, cpNC, cpIP, k8sVer)
76+
77+
// Sanity: the CP is externally managed -- the provider never ran on it, so it
78+
// has no provider status file. This distinguishes the scenario from TestWorkerJoin.
79+
if _, statErr := cpNC.execErr("test", "-e", statusRunPath); statErr == nil {
80+
t.Fatalf("external CP unexpectedly has a provider status file at %s; it should be externally managed (provider never ran on it)", statusRunPath)
81+
}
82+
83+
// 2. Operator extracts trust anchors out-of-band.
84+
caPEM, err := cpNC.ReadFile("/etc/kubernetes/pki/ca.crt")
85+
if err != nil {
86+
t.Fatalf("read external CP ca.crt: %v\n%s", err, caPEM)
87+
}
88+
if !strings.Contains(caPEM, "BEGIN CERTIFICATE") {
89+
t.Fatalf("external CP ca.crt does not look like a PEM cert:\n%s", caPEM)
90+
}
91+
tokenOut, err := cpNC.ExecTimeout(60*time.Second, "kubeadm", "token", "create")
92+
if err != nil {
93+
t.Fatalf("kubeadm token create on external CP: %v\n%s", err, tokenOut)
94+
}
95+
token := strings.TrimSpace(tokenOut)
96+
if token == "" {
97+
t.Fatal("kubeadm token create returned an empty token")
98+
}
99+
cpEndpoint := cpIP + ":6443"
100+
t.Logf("external CP up at %s; operator extracted CA PEM (%d bytes) + bootstrap token", cpEndpoint, len(caPEM))
101+
102+
// 3. Worker Cluster: supply ONLY the CA PEM (ca_certs) + token + endpoint.
103+
// No caCertHashes -- the provider derives the SPKI pin from the PEM
104+
// (resolveCACertHashes). CA pinning is mandatory; UnsafeSkipCAVerification is
105+
// never set.
106+
workerCluster := clusterplugin.Cluster{
107+
Role: clusterplugin.Role(clusterplugin.RoleWorker),
108+
ClusterToken: randomToken(t),
109+
ControlPlaneHost: cpEndpoint,
110+
CACerts: []string{caPEM},
111+
ProviderOptions: map[string]string{"cluster_root_path": "/"},
112+
Options: strings.Join([]string{
113+
"joinConfiguration:",
114+
" discovery:",
115+
" bootstrapToken:",
116+
" token: " + token,
117+
" apiServerEndpoint: " + cpEndpoint,
118+
// Deliberately NO caCertHashes: the provider must derive the pin from
119+
// ca_certs (the operator-supplied PEM above).
120+
}, "\n") + "\n",
121+
}
122+
123+
// 4. Worker container joins the external CP via the provider.
124+
workerNC := startNode(t, uniqueName("ext-worker"))
125+
prepullControlPlaneImages(t, workerNC, k8sVer)
126+
127+
workerOut, workerErr := writeClusterAndReconcile(t, workerNC, workerCluster)
128+
if workerErr != nil {
129+
t.Fatalf("worker reconcile joining external CP failed: %v\n--- output ---\n%s", workerErr, workerOut)
130+
}
131+
t.Logf("worker reconcile output:\n%s", workerOut)
132+
133+
// 5a. Worker status: converged + joined.
134+
ws := readStatus(t, workerNC)
135+
if ws.Phase != "Converged" {
136+
t.Errorf("worker status phase = %q, want Converged (full: %+v)", ws.Phase, ws)
137+
}
138+
if ws.Outcome != "success" {
139+
t.Errorf("worker status outcome = %q, want success", ws.Outcome)
140+
}
141+
if ws.Membership != "joined" {
142+
t.Errorf("worker status membership = %q, want joined", ws.Membership)
143+
}
144+
145+
// 5b. The externally-managed CP now sees 2 registered nodes.
146+
var nodeCount int
147+
cpNC.waitFor(t, "2 nodes registered on external CP", 90*time.Second, func() bool {
148+
o, e := cpNC.execErr("kubectl", "--kubeconfig", adminConf,
149+
"get", "nodes", "-o", "jsonpath={.items[*].metadata.name}")
150+
if e != nil {
151+
return false
152+
}
153+
names := strings.Fields(strings.TrimSpace(o))
154+
nodeCount = len(names)
155+
t.Logf("nodes registered on external CP: %v", names)
156+
return nodeCount >= 2
157+
})
158+
if nodeCount < 2 {
159+
t.Errorf("expected 2 registered nodes on the external CP after join, got %d", nodeCount)
160+
}
161+
}

0 commit comments

Comments
 (0)