Skip to content

fix(hetzner): wait for the default route before detecting the primary NIC - #210

Closed
nihaldivyam wants to merge 1 commit into
masterfrom
fix/hetzner-network-interface-detection
Closed

fix(hetzner): wait for the default route before detecting the primary NIC#210
nihaldivyam wants to merge 1 commit into
masterfrom
fix/hetzner-network-interface-detection

Conversation

@nihaldivyam

Copy link
Copy Markdown
Member

Problem

Bare-metal nodes re-imaged to Ubuntu 26.04 never join the cluster.

cloud-init's runcmd can start before networking is up. The interface detection one-liner then fails:

+ ip route get 8.8.8.8
+ awk {print $5; exit}
RTNETLINK answers: Network is unreachable

NETWORK_INTERFACE ends up empty, so the netplan file written from it gets a bare link:, and netplan apply rejects the whole config:

/etc/netplan/60-vlan.yaml:7:12: Error in network definition: vswitch.4000: interface '' is not defined

The vSwitch VLAN is never created → the node has no address on 10.0.1.0/24 and no route to the private network → kubeadm join times out against the private API endpoint → runcmd fails → cloud-init status = error → CAPH parks the host in ensure-provisioned and retries forever.

Ubuntu 24.04 generally won this race; 26.04 brings networking up later and loses it consistently. Hit on qa-htz1-kilroy-eu worker 2836489 during a 32.4.0 + Ubuntu 26.04 upgrade; recovery needed manual SSH intervention on the node.

Note the kubeaid-storagectl block a few lines further down already carries a sleep 10s "to wait for networking to fully setup during fresh server setups" — the hazard was known, but the mitigation sits after the code that needs it.

Fix

Replace the three copies of the one-liner with a shared hetzner.detectNetworkInterfaceScript helper that:

  • polls for the default route (60 attempts x 2s) instead of assuming it exists, and
  • aborts loudly with a clear error rather than writing an invalid netplan.

Call sites updated: KubeadmConfig.yaml (workers), and KubeadmControlPlane.yaml twice (bare-metal CP vSwitch VLAN, HCloud floating IPs).

Testing

  • helm lint passes.
  • Rendered against real cluster values for qa-htz1-kilroy-eu and production-htz1-kilroy-eu: renders clean, guard present at all call sites, no old-style occurrences remain.
  • Generated script verified as POSIX sh syntax-clean (sh -n) — cloud-init runs runcmd under /bin/sh.
  • The equivalent logic was applied by hand on the failed node: interface detected correctly, netplan applied, VLAN came up, kubeadm join succeeded, node Ready on Ubuntu 26.04 / v1.35.0 with Ceph OSDs reattached.

… NIC

cloud-init's runcmd can start before networking is up. `ip route get
8.8.8.8` then fails with "Network is unreachable", NETWORK_INTERFACE ends
up empty, and the netplan file written from it gets a bare `link:`.
netplan apply rejects the whole config with

  vswitch.4000: interface '' is not defined

so the vSwitch VLAN is never created, the node has no address on the
private network, and kubeadm join times out against the private API
endpoint. runcmd fails, cloud-init reports status=error, and CAPH parks
the host in "ensure-provisioned" indefinitely.

Ubuntu 24.04 generally won this race; 26.04 brings networking up later
and loses it every time, so every bare-metal node re-imaged to 26.04
fails to join. Hit on qa-htz1-kilroy-eu worker 2836489.

Replace the three copies of the one-liner with a shared helper that
polls for the default route (60 x 2s) and aborts loudly if it never
appears, rather than writing an invalid netplan. The kubeaid-storagectl
block already carries a `sleep 10s` for this same race, but it runs
after the code that needs it.
@nihaldivyam

Copy link
Copy Markdown
Member Author

Superseded by #213, which now targets master and carries both commits: this netplan interface-detection fix plus the CAPH CSR-approval fix. Both are needed for a bare-metal node to re-provision to Ubuntu 26.04 and come back fully usable, so they are easier to review and release together. Branch fix/hetzner-network-interface-detection left in place for now — the qa-htz1-kilroy-eu ArgoCD app pins to the combined branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant