fix(hetzner): wait for the default route before detecting the primary NIC - #210
Closed
nihaldivyam wants to merge 1 commit into
Closed
fix(hetzner): wait for the default route before detecting the primary NIC#210nihaldivyam wants to merge 1 commit into
nihaldivyam wants to merge 1 commit into
Conversation
… NIC cloud-init's runcmd can start before networking is up. `ip route get 8.8.8.8` then fails with "Network is unreachable", NETWORK_INTERFACE ends up empty, and the netplan file written from it gets a bare `link:`. netplan apply rejects the whole config with vswitch.4000: interface '' is not defined so the vSwitch VLAN is never created, the node has no address on the private network, and kubeadm join times out against the private API endpoint. runcmd fails, cloud-init reports status=error, and CAPH parks the host in "ensure-provisioned" indefinitely. Ubuntu 24.04 generally won this race; 26.04 brings networking up later and loses it every time, so every bare-metal node re-imaged to 26.04 fails to join. Hit on qa-htz1-kilroy-eu worker 2836489. Replace the three copies of the one-liner with a shared helper that polls for the default route (60 x 2s) and aborts loudly if it never appears, rather than writing an invalid netplan. The kubeaid-storagectl block already carries a `sleep 10s` for this same race, but it runs after the code that needs it.
Member
Author
|
Superseded by #213, which now targets |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Bare-metal nodes re-imaged to Ubuntu 26.04 never join the cluster.
cloud-init'sruncmdcan start before networking is up. The interface detection one-liner then fails:NETWORK_INTERFACEends up empty, so the netplan file written from it gets a barelink:, andnetplan applyrejects the whole config:The vSwitch VLAN is never created → the node has no address on
10.0.1.0/24and no route to the private network →kubeadm jointimes out against the private API endpoint →runcmdfails →cloud-init status=error→ CAPH parks the host inensure-provisionedand retries forever.Ubuntu 24.04 generally won this race; 26.04 brings networking up later and loses it consistently. Hit on
qa-htz1-kilroy-euworker2836489during a 32.4.0 + Ubuntu 26.04 upgrade; recovery needed manual SSH intervention on the node.Note the
kubeaid-storagectlblock a few lines further down already carries asleep 10s"to wait for networking to fully setup during fresh server setups" — the hazard was known, but the mitigation sits after the code that needs it.Fix
Replace the three copies of the one-liner with a shared
hetzner.detectNetworkInterfaceScripthelper that:Call sites updated:
KubeadmConfig.yaml(workers), andKubeadmControlPlane.yamltwice (bare-metal CP vSwitch VLAN, HCloud floating IPs).Testing
helm lintpasses.qa-htz1-kilroy-euandproduction-htz1-kilroy-eu: renders clean, guard present at all call sites, no old-style occurrences remain.shsyntax-clean (sh -n) — cloud-init runsruncmdunder/bin/sh.kubeadm joinsucceeded, node Ready on Ubuntu 26.04 / v1.35.0 with Ceph OSDs reattached.