fix: apply k3s version bumps declaratively through site.yml#544
Conversation
| run: | | ||
| sed -i "s/k3s_version: .*/k3s_version: ${UPGRADE_K3S_VERSION}/" tests/${{ matrix.inventory }}.yml | ||
|
|
||
| - name: Run Upgrade Playbook |
There was a problem hiding this comment.
We still want to test the older upgrade playbook. People are still using it. Keep this here and test the second upgrade with the site.yml
There was a problem hiding this comment.
Restored the upgrade.yml test and added the site.yml bump as a second upgrade on top of it (start → upgrade.yml → site.yml, across three real k3s versions). The HA job already ran them in that order.
| ansible.builtin.service: | ||
| name: k3s-agent | ||
| state: "{{ 'restarted' if _agent_config_result.changed else 'started' }}" | ||
| state: "{{ 'restarted' if (k3s_agent_binary_changed or _agent_config_result.changed) else 'started' }}" |
There was a problem hiding this comment.
Lets just simplify this and always make it restarted because that's guaranteed to bounce the service and handle both cases, then we don't need to do a bunch of logic around determining if the binary was replaced.
There was a problem hiding this comment.
Simplified to always state: restarted and dropped the binary-change detection. Applied the same simplification to the server role for consistency: the first server already always restarted, and joined servers now do too — still one at a time behind serial: 1 + the health check, so the etcd quorum is preserved.
f601d9a to
3c5e332
Compare
| hosts: server | ||
| gather_facts: false | ||
| become: true | ||
| serial: 1 |
There was a problem hiding this comment.
We are not going to reduce the initial install of k3s to serial just to support upgrades. If you want this, use the ansible-playbook site.yml --forks=1 system to force serial. Include that in the documentation.
There was a problem hiding this comment.
Done — dropped serial: 1 from the server play, so a fresh install runs in parallel as before. The joined-server restart is now unconditional, which is the actual version-bump fix (the first server and the agents already always restart). The README documents running a multi-server upgrade with --forks=1, and the HA integration test now runs the site.yml upgrade with --forks=1 so that path is covered.
Re-running site.yml with a raised k3s_version replaced the on-disk binary but left agents and joined servers running the old runtime, because the install script skips the service start and the roles only restarted the service on a config change. A version-only bump was therefore not applied until the services were restarted by hand. Always restart the k3s service in the server and agent roles on a site.yml run, so the cluster reliably picks up a new config or runtime without any logic to detect whether the binary changed. On a multi-server cluster, run the playbook with --forks=1 so the servers restart one at a time and the etcd quorum is preserved; document this in the README. Keep the dedicated upgrade.yml integration test and add a second upgrade through site.yml, asserting the running version the kubelet reports rather than the on-disk binary, so the tests fail if the restart regresses. Co-authored-by: Dominik Wombacher <dominik@wombacher.cc> Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
3c5e332 to
8457b46
Compare
What
Re-running
site.ymlwith a raisedk3s_versionnow upgrades the running cluster instead of only replacing the on-disk binary. Previously the install script replaced the binary but never restarted the service (INSTALL_K3S_SKIP_START), and the roles only restarted on a config change — so a version-only bump left agents and joined servers running the old runtime until they were restarted by hand.Why
This makes
site.ymlsafely declarative for the cluster lifecycle, which is the ask in #485. It rebuilds the direction started in #505 on top of the now template-free roles: that earlier PR was parked pending the jinja-to-config.yaml refactor, which has since landed, so this is a fresh implementation rather than a rebase of it.How
serial: 1(matchingupgrade.yml) and each restarted server — including the first — waits for its API to report healthy before the play moves on, so a rolling change never drops the etcd quorum.site.yml(single server + agent, and the three-server HA path) and assert the running version the kubelet reports, not the on-disk binary, so the tests fail if the restart ever regresses.Scope note
upgrade.ymland thek3s_upgraderole are intentionally left in place and unchanged. #485 also suggested removing them, but that is a larger, breaking change (a v2.0.0 concern) and is deliberately out of scope here —site.ymlsimply gains the declarative-upgrade capability alongside the existing playbook.Testing
The full integration suite — including the HA job's rolling
site.ymlversion bump across three etcd servers — was run green on a fork before opening this PR.Addresses #485. Builds on the approach in #505 (credited via
Co-authored-byon the commit).