fix(hetzner): unbreak bare-metal re-provisioning on Ubuntu 26.04

[nihaldivyam]

Two independent bugs make bare-metal nodes unusable after re-provisioning. We hit it while upgrading to KubeAid 32.4.0 / Kubernetes v1.35 / Ubuntu 26.04, and both are fixed here. (Supersedes #210, which carried the first commit alone.)

1. Node never joins — empty link: in the vSwitch netplan

cloud-init's runcmd can start before networking is up, so the interface-detection one-liner fails:

+ ip route get 8.8.8.8
+ awk {print $5; exit}
RTNETLINK answers: Network is unreachable

NETWORK_INTERFACE ends up empty, the generated netplan gets a bare link:, and netplan apply rejects the whole file:

/etc/netplan/60-vlan.yaml:7:12: Error in network definition: vswitch.4000: interface '' is not defined

No vSwitch VLAN → no address on the private network → kubeadm join times out against the private API endpoint → runcmd fails → cloud-init status = error → CAPH parks the host in ensure-provisioned forever.

Ubuntu 24.04 generally won this race; 26.04 brings networking up later and loses it consistently.

Fix: replace the three copies of the one-liner with a shared hetzner.detectNetworkInterfaceScript helper that polls for the default route (60 × 2s) and aborts loudly rather than writing an invalid netplan. Call sites: KubeadmConfig.yaml (workers), KubeadmControlPlane.yaml ×2 (bare-metal CP vSwitch VLAN, HCloud floating IPs).

Note the kubeaid-storagectl block already carried a sleep 10s for this same race — the mitigation just sat after the code that needed it.

2. Node joins but kubelet has no serving cert — CAPH denies the vSwitch IP

With serverTLSBootstrap: true, kubelet requests a serving cert whose SANs include its --node-ip vSwitch address. CAPH denies it:

Denied: CSRValidationFailed - Validation by cluster-api-provider-hetzner failed:
the IP address "10.0.1.2" is not allowed

CAPH's allow-list (pkg/csr/csr.go) is built from HetznerBareMetalMachine.status.addresses, derived from HetznerBareMetalHost.spec.status.hardwareDetails.nics — a NIC snapshot taken in the Hetzner rescue system, before cloud-init creates the vSwitch VLAN. The private IP structurally cannot be there. We saw 84 consecutive denials on one node.

Result: no serving cert, so every apiserver-initiated call — logs, exec, port-forward, metrics — fails with remote error: tls: internal error, on a node that is otherwise Ready.

Upstream: syself/cluster-api-provider-hetzner#2095 (open; the maintainer rejected the proposed fix as circular, so nothing is coming soon). The validation code is unchanged between CAPH v1.1.7 and v1.1.8 — not a provider regression; it simply only bites on re-provision.

Fix: pass --disable-csr-approval and let kubelet-csr-approver — which KubeAid already ships — handle these CSRs. It validates the same requests against providerIpPrefixes, plus node-name and expiry checks, so security posture is preserved.

Worth knowing: CAPH's controller skips a CSR only if len(Status.Conditions) > 0, i.e. once someone has decided it. With both controllers running it is a pure race, and CAPH won 84/84 on our cluster — so the approver alone is not sufficient, and leaving both enabled would at best give intermittent denials.

Rejected alternatives: serverTLSBootstrap: false (drops to unverified self-signed kubelet certs) and hand-patching hardwareDetails (CAPH v1.1.8 removed the if HardwareDetails == nil guard, so it is overwritten on every re-registration).

--disable-csr-approval is global to the management cluster, which is fine for KubeAid: each cluster is self-managed, and KubeAid ships the replacement approver as a standard component.

Testing

• Generated script verified POSIX-sh clean (sh -n) — cloud-init runs runcmd under /bin/sh.
• Validated end to end on hardware. Worker 2836489 (before these fixes) needed manual SSH rescue to join and never got a serving cert. Worker 2836490, re-provisioned with this branch, went image-installing → provisioned → Ready on Ubuntu 26.04 / v1.35.0 in ~19 minutes fully unattended, with both CSRs Approved,Issued automatically. Ceph returned to 3 mons and 6/6 OSDs with all PGs active+clean.

Documentation

Adds docs/operations/upgrade-hetzner-hybrid-cluster.md — a node-by-node runbook for upgrading a mode: hybrid cluster (HCloud control plane + bare-metal workers on a Robot vSwitch), written from this run.

It documents the parts that are not discoverable from the charts: which values field drives which resource and which of those are immutable (so a plain sync reports partial failure by design); recreating HCloudMachineTemplate and forcing a control-plane roll — including that CAPI v1beta2 renamed the field to spec.rollout.after and patching the old spec.rolloutAfter silently no-ops; and that bare-metal workers are standalone Machines with no MachineDeployment, so nothing rolls them and they upgrade only by delete + re-provision, strictly one at a time because each holds a Ceph mon.

It also carries a troubleshooting section for the failures actually hit during this upgrade — stuck deletions (orphaned VolumeAttachments, pods in deleted namespaces), cloud-init recovery including that status.json is persistent rather than tmpfs, CAPH's ~16 minute reconcile backoff, both flavours of denied kubelet serving CSR, and ArgoCD apps silently serving cached manifests.