Skip to content

fix: apply k3s version bumps declaratively through site.yml#544

Merged
dereknola merged 1 commit into
k3s-io:mainfrom
lexfrei:fix/site-declarative-restart-485
Jul 14, 2026
Merged

fix: apply k3s version bumps declaratively through site.yml#544
dereknola merged 1 commit into
k3s-io:mainfrom
lexfrei:fix/site-declarative-restart-485

Conversation

@lexfrei

@lexfrei lexfrei commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

What

Re-running site.yml with a raised k3s_version now upgrades the running cluster instead of only replacing the on-disk binary. Previously the install script replaced the binary but never restarted the service (INSTALL_K3S_SKIP_START), and the roles only restarted on a config change — so a version-only bump left agents and joined servers running the old runtime until they were restarted by hand.

Why

This makes site.yml safely declarative for the cluster lifecycle, which is the ask in #485. It rebuilds the direction started in #505 on top of the now template-free roles: that earlier PR was parked pending the jinja-to-config.yaml refactor, which has since landed, so this is a fresh implementation rather than a rebase of it.

How

  • The server and agent roles track whether the k3s binary was actually replaced and restart when it was, in addition to the existing config-change trigger.
  • The server play runs with serial: 1 (matching upgrade.yml) and each restarted server — including the first — waits for its API to report healthy before the play moves on, so a rolling change never drops the etcd quorum.
  • The joined-server readiness check waits for the servers handled so far rather than the full set, which keeps it correct under both serial and parallel runs.
  • Integration tests upgrade through site.yml (single server + agent, and the three-server HA path) and assert the running version the kubelet reports, not the on-disk binary, so the tests fail if the restart ever regresses.

Scope note

upgrade.yml and the k3s_upgrade role are intentionally left in place and unchanged. #485 also suggested removing them, but that is a larger, breaking change (a v2.0.0 concern) and is deliberately out of scope here — site.yml simply gains the declarative-upgrade capability alongside the existing playbook.

Testing

The full integration suite — including the HA job's rolling site.yml version bump across three etcd servers — was run green on a fork before opening this PR.

Addresses #485. Builds on the approach in #505 (credited via Co-authored-by on the commit).

run: |
sed -i "s/k3s_version: .*/k3s_version: ${UPGRADE_K3S_VERSION}/" tests/${{ matrix.inventory }}.yml

- name: Run Upgrade Playbook

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We still want to test the older upgrade playbook. People are still using it. Keep this here and test the second upgrade with the site.yml

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restored the upgrade.yml test and added the site.yml bump as a second upgrade on top of it (start → upgrade.ymlsite.yml, across three real k3s versions). The HA job already ran them in that order.

Comment thread roles/k3s_agent/tasks/main.yml Outdated
ansible.builtin.service:
name: k3s-agent
state: "{{ 'restarted' if _agent_config_result.changed else 'started' }}"
state: "{{ 'restarted' if (k3s_agent_binary_changed or _agent_config_result.changed) else 'started' }}"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets just simplify this and always make it restarted because that's guaranteed to bounce the service and handle both cases, then we don't need to do a bunch of logic around determining if the binary was replaced.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplified to always state: restarted and dropped the binary-change detection. Applied the same simplification to the server role for consistency: the first server already always restarted, and joined servers now do too — still one at a time behind serial: 1 + the health check, so the etcd quorum is preserved.

@lexfrei
lexfrei force-pushed the fix/site-declarative-restart-485 branch from f601d9a to 3c5e332 Compare July 6, 2026 17:30
Comment thread playbooks/site.yml Outdated
hosts: server
gather_facts: false
become: true
serial: 1

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are not going to reduce the initial install of k3s to serial just to support upgrades. If you want this, use the ansible-playbook site.yml --forks=1 system to force serial. Include that in the documentation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — dropped serial: 1 from the server play, so a fresh install runs in parallel as before. The joined-server restart is now unconditional, which is the actual version-bump fix (the first server and the agents already always restart). The README documents running a multi-server upgrade with --forks=1, and the HA integration test now runs the site.yml upgrade with --forks=1 so that path is covered.

Re-running site.yml with a raised k3s_version replaced the on-disk
binary but left agents and joined servers running the old runtime,
because the install script skips the service start and the roles only
restarted the service on a config change. A version-only bump was
therefore not applied until the services were restarted by hand.

Always restart the k3s service in the server and agent roles on a
site.yml run, so the cluster reliably picks up a new config or runtime
without any logic to detect whether the binary changed. On a
multi-server cluster, run the playbook with --forks=1 so the servers
restart one at a time and the etcd quorum is preserved; document this
in the README.

Keep the dedicated upgrade.yml integration test and add a second
upgrade through site.yml, asserting the running version the kubelet
reports rather than the on-disk binary, so the tests fail if the
restart regresses.

Co-authored-by: Dominik Wombacher <dominik@wombacher.cc>

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
@lexfrei
lexfrei force-pushed the fix/site-declarative-restart-485 branch from 3c5e332 to 8457b46 Compare July 14, 2026 20:20
@dereknola
dereknola merged commit e5ec2f0 into k3s-io:main Jul 14, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants