docs: reflect cluster upgrades as implemented (#10)

PR #9 landed the upgrade lifecycle, so update the docs/README to match:
- New docs/upgrades.md: operator runbook (model + explicit-pin trigger, supported
  edges, single-node and one-at-a-time HA upgrade, the automatic kubelet-config
  repair, best-effort encrypted etcd snapshots, rollback, common failures).
- docs/lifecycle.md: Upgrades section now describes the implemented flow and links
  the runbook (was 'not yet implemented').
- docs index, configuration (kubernetesVersion bump triggers an upgrade) and
  troubleshooting (upgrade-didn't-run / refuse-upgrade / snapshot-skipped) updated.
- README: 'What works today' adds the upgrade bullet; Status table flips Cluster
  upgrades to implemented/validated; full init/join/upgrade/reset lifecycle noted;
  Documentation table links the new page.

Docs only; no code change.

Signed-off-by: William Rizzo <william.rizzo@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

Author: wrkode

README.md

@@ -56,6 +56,13 @@ built and verified in CI):
   bounded-TTL join material (for control planes, re-uploading the cluster certs
   under a fresh certificate key), computes the cluster CA SPKI pin, derives the
   API endpoint from `admin.conf`, and prints a ready-to-paste join cloud-config.
+- **Cluster upgrades (`kubeadm upgrade`).** Pin a newer `kubernetesVersion` and
+  boot the matching image; the provider converges the cluster one minor at a time
+  (control plane via `upgrade apply`, followers/workers via `upgrade node`),
+  refusing downgrades / skip-level / out-of-window. It auto-repairs the kubelet
+  config when an image swap leaves the new kubelet unable to start, and takes a
+  best-effort etcd snapshot (only onto encrypted storage). Validated live for
+  1.34 -> 1.35. See [`docs/upgrades.md`](./docs/upgrades.md).
 - **CNI is the operator's choice.** The provider installs no CNI;
   [`samples/cni-calico/`](./samples/cni-calico/) shows two ways to add Calico
   (apply after the cluster is up, or bundle it in the control-plane
@@ -64,10 +71,9 @@ built and verified in CI):
   image build across the supported Kubernetes window; tagged releases publish
   per-minor images to ghcr (see "Released images").
 
-Cluster bootstrap, credential handling, multi-control-plane HA, the reset
-handler, the image build, CI, and release publishing are implemented. The next
-major capability is cluster upgrades (`kubeadm upgrade`). See the roadmap
-discussion in the issue above.
+The full lifecycle - bootstrap, join, **upgrade**, and reset - plus credential
+handling, multi-control-plane HA, the image build, CI, and release publishing are
+implemented and validated on VMs. See the roadmap discussion in the issue above.
 
 ## Status
 
@@ -81,7 +87,7 @@ discussion in the issue above.
 | Kairos image build (pinned, checksum-verified) | implemented |
 | CI (build/vet/test/lint + image build across 1.34/1.35/1.36) | implemented |
 | Release automation (per-minor images to ghcr + binary) | implemented |
-| Cluster upgrades (`kubeadm upgrade`) | not yet (next) |
+| Cluster upgrades (`kubeadm upgrade`) | implemented, validated on libvirt (1.34 -> 1.35) |
 | Field readiness | not ready |
 
 ## Building
@@ -171,6 +177,7 @@ Usage documentation lives in [`docs/`](./docs/):
 | [Creating a cluster](./docs/creating-a-cluster.md) | Single control plane plus workers. |
 | [High availability](./docs/high-availability.md) | Multi-control-plane (stacked etcd). |
 | [mint-join](./docs/mint-join.md) | The join-material CLI. |
+| [Upgrades](./docs/upgrades.md) | Upgrading between Kubernetes minors with `kubeadm upgrade`. |
 | [CNI](./docs/cni.md) | Installing a CNI. |
 | [Security model](./docs/security.md) | Tokens, the cert-key blast radius, CA pinning, at-rest encryption. |
 | [Lifecycle and reset](./docs/lifecycle.md) | Reconcile, reset, the version window, upgrades. |

docs/README.md

@@ -21,6 +21,7 @@ Usage documentation for **provider-kubernetes**, the Go-native
 | [Creating a cluster](./creating-a-cluster.md) | Single control plane, adding workers, the join-material flow. |
 | [High availability](./high-availability.md) | Multi-control-plane (stacked etcd): stable endpoint, one-at-a-time bring-up, failover. |
 | [mint-join reference](./mint-join.md) | The `agent-provider-kubernetes mint-join` subcommand. |
+| [Upgrades](./upgrades.md) | Upgrading between Kubernetes minors (`kubeadm upgrade`): pin + image, single-node and HA, kubelet-config repair, etcd snapshots, rollback. |
 | [CNI](./cni.md) | Installing a CNI (the provider installs none). |
 | [Security model](./security.md) | `cluster_token`, certificate-key blast radius, CA pinning, at-rest encryption, the trust boundary. |
 | [Lifecycle and reset](./lifecycle.md) | Reboot idempotency, reset / `EventClusterReset`, the supported version window, upgrades. |

docs/configuration.md

@@ -64,7 +64,7 @@ Common fields:
 ```yaml
 config: |
   clusterConfiguration:
-    kubernetesVersion: v1.34.0          # must be within the supported window
+    kubernetesVersion: v1.34.0          # within the window; must match the image; bumping it triggers an upgrade (see docs/upgrades.md)
     controlPlaneEndpoint: "vip:6443"    # stable endpoint; REQUIRED for HA
     imageRepository: registry.k8s.io
     networking:

docs/lifecycle.md

@@ -55,6 +55,11 @@ Each released image bundles one Kubernetes minor; pick the matching image tag (s
 
 ## Upgrades
 
-Cluster **upgrades** (`kubeadm upgrade`) are **not yet implemented** - this is the
-next planned capability. For now the lifecycle covers bootstrap, join, and reset.
-Track the roadmap in the project's issues.
+Cluster **upgrades** (`kubeadm upgrade`) are **implemented**: pin a newer
+`kubernetesVersion`, boot the matching image, and the provider converges the
+cluster on the next reconcile (control plane via `kubeadm upgrade apply`, followers
+and workers via `kubeadm upgrade node`, one minor at a time). A newer image without
+a pin bump does not auto-upgrade, and downgrade / skip-level / out-of-window
+targets are refused. See [Upgrades](./upgrades.md) for the full operator runbook
+(single-node and HA), the automatic kubelet-config repair, etcd snapshots, and
+rollback.

docs/troubleshooting.md

@@ -51,6 +51,18 @@ match the `kubeadm` binary in the image. A mismatch fails fast by design. Use th
 image tag for the minor you want, or adjust the pin. See
 [Lifecycle](./lifecycle.md#supported-version-window).
 
+### Upgrade didn't run, or was refused
+
+- **Nothing happened after booting a newer image.** An upgrade runs only when you
+  bump `clusterConfiguration.kubernetesVersion` to the new minor; a newer binary
+  alone is a no-op by design.
+- **Reconcile logs `refuse-upgrade`.** The pin is a downgrade, skip-level
+  (e.g. 1.34 -> 1.36), or out-of-window. Upgrade one minor at a time within the
+  window.
+- **Snapshot skipped warning.** The provider refuses to write a plaintext etcd
+  snapshot when it can't confirm the persistent partition is encrypted - take a
+  manual etcd backup before upgrading. See [Upgrades](./upgrades.md).
+
 ### A reset left a stale etcd member (HA)
 
 If a control plane was reset while the cluster was unreachable, its etcd member is

docs/upgrades.md

@@ -0,0 +1,103 @@
+# Upgrades
+
+The provider upgrades a running cluster between supported Kubernetes minors by
+driving `kubeadm upgrade`. Upgrades are **image-driven and explicitly pinned**:
+you boot a newer provider-kubernetes image and bump the version pin; on the next
+boot the provider detects the delta and converges the cluster.
+
+> Validated live for 1.34 -> 1.35 on a single control plane. Multi-node/HA upgrade
+> ordering is implemented (see below) but exercise it carefully. As always, this
+> provider is under active development and not field-ready.
+
+## The model
+
+- The **target version is the bundled `kubeadm` binary** in the image you boot.
+  By ADR-3 an operator `kubernetesVersion` pin must equal the bundled minor, so the
+  pin and the binary always agree.
+- An upgrade runs **only when you pin** `clusterConfiguration.kubernetesVersion` to
+  the new minor. A newer image **without** a matching pin bump does **not**
+  auto-upgrade - it is a no-op (so a routine image refresh never silently
+  re-versions your cluster).
+- Only **+1 minor** at a time (kubeadm's rule). Skipping a minor (1.34 -> 1.36),
+  downgrades, and out-of-window targets are **refused loudly** before anything
+  destructive runs.
+
+## Supported edges
+
+Within the current window: **1.34 -> 1.35** and **1.35 -> 1.36**. To go 1.34 ->
+1.36, step through 1.35 (boot the 1.35 image first, let it converge, then the 1.36
+image).
+
+## Upgrading a single control plane
+
+1. Make sure the cluster is healthy (`kubectl get nodes`, control-plane pods
+   Running). Take an **etcd backup** first - see "etcd snapshots" below.
+2. Bump the pin in the node's cloud-config (`/oem` on an installed node) to the new
+   minor, e.g. `kubernetesVersion: v1.35.5`.
+3. Upgrade the OS image to the matching minor and reboot. Either:
+   - `kairos-agent upgrade --source oci:<registry>/provider-kubernetes:<tag>-k8s1.35`
+     (then reboot), or
+   - boot the new image via your provisioning flow.
+4. On reboot the provider's reconcile detects that the bundled binary is a minor
+   ahead of the cluster and runs `kubeadm upgrade apply <version>`. When the new
+   kubelet cannot start against the old node config (a flag removed across the
+   minor), the provider first repairs the kubelet config (see below), brings the
+   control plane back, then applies. The kubelet is restarted onto the new version.
+5. Verify: `kubectl version` (server), `kubectl get nodes` (node at the new
+   version, Ready).
+
+## Upgrading multiple control planes / workers (HA)
+
+Upgrade **one node at a time**, control planes first, then workers:
+
+1. Upgrade the first control plane as above; wait until the cluster version has
+   flipped and it is Ready.
+2. Upgrade each remaining control plane one at a time. A follower runs
+   `kubeadm upgrade node` (the provider detects the cluster already advanced and
+   does not re-apply); it waits, bounded, for the control plane to be healthy first.
+3. Upgrade workers last, one at a time; each runs `kubeadm upgrade node`.
+
+This sequencing is the operator's responsibility (the same one-at-a-time delivery
+contract as HA join); the provider adds bounded health gates but builds no
+cross-node lock.
+
+## The kubelet-config repair (why an upgrade "just works" after an image swap)
+
+Kubernetes removes/renames kubelet flags and config keys across minors. Because a
+Kairos A/B image swap replaces the kubelet binary *before* any upgrade runs, the
+new kubelet can crashloop on a flag the old kubeadm wrote (for example
+`--pod-infra-container-image`, removed in 1.35), which would leave the control
+plane down. The provider handles this automatically: when an upgrade is due and the
+local API is unreachable, it runs `kubeadm init phase kubelet-start` to regenerate
+the kubelet config with the new kubeadm (no API, no secrets), which lets the
+kubelet start and the existing control plane return before `kubeadm upgrade apply`
+runs. You do not need to do anything for this.
+
+## etcd snapshots
+
+`kubeadm upgrade apply` mutates etcd and is largely forward-only. The provider
+takes a **best-effort** etcd snapshot before applying, but **only** onto an
+encrypted persistent partition (an etcd snapshot is a full plaintext dump of every
+cluster secret). If it cannot confirm the partition is encrypted, it **refuses to
+write a plaintext snapshot** and logs a warning - so on an unencrypted node you
+must take your own snapshot before upgrading. The provider never copies the
+snapshot off the node; backup custody is yours.
+
+## Rollback
+
+kubeadm upgrades (especially etcd) are forward-only; there is no automatic
+rollback. To recover, restore your etcd snapshot and boot the previous image. A
+node wedged mid-upgrade can be recovered with the [reset](./lifecycle.md) flow and
+re-joined. The provider never auto-resets a control plane on upgrade failure - it
+fails loud and leaves the node for you to inspect.
+
+## What can go wrong
+
+| Symptom | Cause / fix |
+|---------|-------------|
+| Reconcile logs `refuse-upgrade` | Skip-level, downgrade, or out-of-window pin. Pin only +1 minor within the window. |
+| Upgrade doesn't start | No version pin bump (a newer binary alone won't upgrade). Bump `clusterConfiguration.kubernetesVersion`. |
+| Pin/binary mismatch hard error | The pin minor must equal the bundled image's minor. |
+| Snapshot skipped warning | Persistent partition encryption unconfirmed - take a manual etcd backup. |
+
+See also [Lifecycle and reset](./lifecycle.md) and [Troubleshooting](./troubleshooting.md).