feat: surface reconcile status on the node (ADR-4-S)

The reconcile runs as a fire-and-forget boot stage whose exit code never
flows back to kairos-agent, so a node that failed to converge was a black
box unless you SSH in and read journald. Publish the outcome through two
best-effort, bounded channels instead.

Layer 1 (always): a small, secret-free YAML status doc written atomically
to /run/provider-kubernetes/status.yaml plus a /var/log mirror, mode 0640
root:adm. This is the only channel that works pre-membership - exactly the
failure you most need to debug (bad endpoint, CA-pin mismatch, init
refused). The schema is closed (camelCase keys) and the only free-text
field is sanitized, so it cannot carry a secret.

Layer 2 (post-membership): the same outcome as own-Node annotations under
provider-kubernetes.kairos.io/* via a single bounded kubectl-argv call,
reusing the least-privilege kubelet.conf identity already on the node - no
client-go, no new RBAC. It no-ops when the node is not a member.

Status writing never blocks, never retries forever, and never masks the
real reconcile result (#4099-1). Plan stays pure; surfacing is executor
and run-wiring only. Validated live on libvirt: a converged control plane
publishes phase=Converged plus all annotations, and a worker pointed at an
unreachable endpoint writes phase=Failed/reason=ControlPlaneUnreachable
with no annotation.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: William Rizzo <william.rizzo@gmail.com>

Author: wrkode

README.md

@@ -181,6 +181,7 @@ Usage documentation lives in [`docs/`](./docs/):
 | [CNI](./docs/cni.md) | Installing a CNI. |
 | [Security model](./docs/security.md) | Tokens, the cert-key blast radius, CA pinning, at-rest encryption. |
 | [Lifecycle and reset](./docs/lifecycle.md) | Reconcile, reset, the version window, upgrades. |
+| [Node status](./docs/status.md) | Why a node did or did not converge: status file + Node annotations. |
 | [Troubleshooting](./docs/troubleshooting.md) | Logs, common failures, filing issues. |
 
 ## Contributing

docs/README.md

@@ -25,6 +25,7 @@ Usage documentation for **provider-kubernetes**, the Go-native
 | [CNI](./cni.md) | Installing a CNI (the provider installs none). |
 | [Security model](./security.md) | `cluster_token`, certificate-key blast radius, CA pinning, at-rest encryption, the trust boundary. |
 | [Lifecycle and reset](./lifecycle.md) | Reboot idempotency, reset / `EventClusterReset`, the supported version window, upgrades. |
+| [Node status](./status.md) | Why a node did or did not converge: the local status file and the own-Node annotations. |
 | [Troubleshooting](./troubleshooting.md) | Where logs are, common failures, and what "fail loud, never hang" means in practice. |
 
 ## How it works in one paragraph

docs/status.md

@@ -0,0 +1,111 @@
+# Node status (why a node did or did not converge)
+
+The provider runs one bounded reconcile pass on each boot. Because that pass runs
+as a fire-and-forget boot-time stage, its exit code does not flow back to
+`kairos-agent` - so the provider publishes its outcome through two channels you can
+inspect directly, instead of making you read journald. Both are **best-effort**: a
+status write never blocks the boot, never retries forever, and never changes or
+masks the real reconcile result.
+
+## Layer 1 - the local status file (always written)
+
+Every reconcile pass and every reset writes a small YAML status document to:
+
+- `/run/provider-kubernetes/status.yaml` - current boot, on tmpfs (authoritative).
+- `/var/log/provider-kubernetes/status.yaml` - a persistent mirror that survives
+  reboot, for post-mortem after a failed boot.
+
+Both are written atomically (temp file + rename, so a reader never sees a partial
+doc) with mode **0640, owner root, group adm** (group-readable by an `adm`-group
+monitoring agent; world-unreadable). This is the only channel that works when a
+node never joined the cluster - the very failure you most need to debug.
+
+The document carries **no secrets by construction**: every field except `message`
+is a closed enum, and `message` is sanitized (bootstrap tokens, PEM blocks, and
+long secret-like runs are redacted; it is truncated to one short line).
+
+### Example
+
+```yaml
+apiVersion: provider-kubernetes.kairos.io/v1
+phase: Failed
+role: worker
+membership: uninitialized
+outcome: failure
+reason: ControlPlaneUnreachable
+terminal: false
+lastAction: wait-for-control-plane
+message: "control-plane endpoint not reachable within budget"
+budget:
+  attempts: 3
+  maxAttempts: 3
+updatedAt: "2026-06-03T12:00:00Z"
+bootID: 7c9e6679-7425-40de-944b-e07fc1f90ae7
+version: v0.2.0
+```
+
+### Fields
+
+| Field | Meaning |
+|-------|---------|
+| `phase` | `Reconciling` (in progress), `Converged` (success), `Failed`, or `Reset`. |
+| `role` | The node's declared role: `init`, `controlplane`, or `worker`. |
+| `membership` | `uninitialized`, `initialized`, or `joined` (probed actual state). |
+| `outcome` | `success` or `failure`. |
+| `reason` | Closed enum, empty on success. See table below. |
+| `terminal` | `true` if the failure will not be retried on the next boot (fail-fast); `false` if a later boot may converge. |
+| `lastAction` | The last kubeadm action attempted (e.g. `run-join`, `upgrade-apply`). |
+| `message` | One short, sanitized, operator-facing line. |
+| `budget` | `attempts` used out of `maxAttempts`. |
+| `updatedAt` | RFC3339 timestamp of this status. |
+| `bootID` | The kernel boot ID, so you can tell which boot produced it. |
+| `version` | The provider build version. |
+
+### Reason codes
+
+| `reason` | What happened |
+|----------|---------------|
+| `ControlPlaneUnreachable` | The control-plane endpoint was not reachable within the budget. |
+| `JoinTimeout` | A join did not complete within the budget. |
+| `InitRefused` | A second `role: init` was refused to avoid clobbering an existing cluster (#4099-5). |
+| `UpgradeRefused` | A downgrade, skip-level, or out-of-window upgrade pin was refused. |
+| `BudgetExhausted` | The bounded retry budget ran out. |
+| `KubeadmError` | A kubeadm action failed. |
+| `ConfigInvalid` | The supplied cluster config was invalid (e.g. empty/short `cluster_token`). |
+| `ResetFailed` / `ResetOK` | The outcome of an `EventClusterReset`. |
+
+## Layer 2 - Node annotations (when the node is a cluster member)
+
+When - and only when - the node is already a member and a local kubeconfig exists,
+the provider **also** records the same outcome as annotations on its own Node
+object, so you can see it cluster-wide without SSH:
+
+```sh
+kubectl get node <name> -o jsonpath='{.metadata.annotations}' | tr ',' '\n' | grep provider-kubernetes.kairos.io
+```
+
+Keys published (all under `provider-kubernetes.kairos.io/`):
+`phase`, `outcome`, `reason`, `terminal`, `last-action`, `updated-at`, `version`.
+
+Notes:
+
+- The free-text `message` is **not** published as an annotation - only the closed
+  enum fields are, so an annotation can never carry a secret.
+- These annotations are readable by any principal with `get node`. They are coarse
+  lifecycle metadata and intentionally carry no sensitive data.
+- The write uses the **least-privilege** credential already on the node
+  (`kubelet.conf`, the `system:node:<name>` identity, preferred over `admin.conf`).
+  The provider creates **no** new RBAC, ServiceAccount, or token.
+- If the node is not a member, has no kubeconfig, or the API is unreachable, this
+  layer is silently skipped - Layer 1 still has the truth.
+
+## How to use it
+
+- A node didn't come up? Read `/run/provider-kubernetes/status.yaml` (current boot)
+  or, after a reboot, the `/var/log` mirror. `phase: Failed` + `reason` + `message`
+  tells you what to fix; `terminal: true` means the next boot won't retry on its
+  own.
+- Watching a fleet? Scrape the Node annotations with `kubectl`.
+
+See also [Lifecycle and reset](./lifecycle.md) and
+[Troubleshooting](./troubleshooting.md).

docs/troubleshooting.md

@@ -2,6 +2,12 @@
 
 ## Where to look first
 
+- **Status file (start here):** `/run/provider-kubernetes/status.yaml` (current
+  boot) or `/var/log/provider-kubernetes/status.yaml` (persists across reboot). A
+  one-glance `phase` / `reason` / `message` summary of why the node did or did not
+  converge, plus whether the failure is `terminal`. On a cluster member the same
+  outcome is also on the Node as `provider-kubernetes.kairos.io/*` annotations. See
+  [Node status](./status.md).
 - **Reconcile log:** `/var/log/provider-kubernetes-reconcile.log` on the node. It
   records the role, observed membership, the planned actions, and any bounded
   failure (with kubeadm output, secret-sanitized).

internal/provider/options.go

@@ -17,6 +17,11 @@ const (
 	// ReconcileLogPath is where the boot-time reconcile subcommand writes its
 	// stdout/stderr; surfacing failures to operators (OQ-4 logs-only for v1).
 	ReconcileLogPath = "/var/log/provider-kubernetes-reconcile.log"
+
+	// The Layer-1 structured status paths are defined in internal/status:
+	//   status.StatusRunPath = "/run/provider-kubernetes/status.yaml"  (tmpfs, 0640)
+	//   status.StatusLogPath = "/var/log/provider-kubernetes/status.yaml"  (persistent, 0640)
+	// See ADR-4-S for the full design.
 )
 
 // Context is the parsed, validated provider input derived from a Cluster. It is a

internal/provider/reset_handler.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
+	"time"
 
 	"github.com/kairos-io/kairos-sdk/bus"
 	"github.com/kairos-io/kairos-sdk/clusterplugin"
@@ -12,22 +13,25 @@ import (
 	"gopkg.in/yaml.v3"
 
 	"github.com/kairos-io/provider-kubernetes/internal/kubeadm"
+	"github.com/kairos-io/provider-kubernetes/internal/reconcile/actualstate"
 	"github.com/kairos-io/provider-kubernetes/internal/reset"
+	"github.com/kairos-io/provider-kubernetes/internal/status"
+	"github.com/kairos-io/provider-kubernetes/version"
 )
 
 // HandleClusterReset is the Kairos "cluster.reset" event handler. It parses the
 // payload and performs a bounded, idempotent reset (ADR-4). It deliberately does
 // NOT run cluster_token validation: reset must succeed regardless of token state.
 func HandleClusterReset(event *pluggable.Event) pluggable.EventResponse {
-	return handleClusterReset(event, kubeadm.ExecRunner{})
+	return handleClusterReset(event, kubeadm.ExecRunner{}, status.NewFileSink())
 }
 
 // maxResetPayloadBytes caps the event payload size before unmarshaling, to bound
 // CPU/memory of (in-process but still untrusted-by-shape) YAML/JSON parsing.
 const maxResetPayloadBytes = 1 << 20 // 1 MiB
 
-// handleClusterReset is the testable core (runner injected).
-func handleClusterReset(event *pluggable.Event, runner kubeadm.Runner) pluggable.EventResponse {
+// handleClusterReset is the testable core (runner and sink injected).
+func handleClusterReset(event *pluggable.Event, runner kubeadm.Runner, sink status.StatusSink) pluggable.EventResponse {
 	var resp pluggable.EventResponse
 	if event == nil {
 		return resp
@@ -69,12 +73,35 @@ func handleClusterReset(event *pluggable.Event, runner kubeadm.Runner) pluggable
 	}
 
 	logrus.Infof("provider-kubernetes: handling cluster reset (root=%s)", rootPath)
-	if err := reset.Run(context.Background(), reset.Options{
+	resetErr := reset.Run(context.Background(), reset.Options{
 		Runner:    runner,
 		RootPath:  rootPath,
 		CRISocket: criSocket,
-	}); err != nil {
-		resp.Error = fmt.Sprintf("cluster reset: %s", err)
+	})
+
+	// S4: write terminal reset status. Best-effort, bounded, swallowed on error.
+	writeResetStatus(sink, actualstate.Role(string(config.Cluster.Role)), resetErr)
+
+	if resetErr != nil {
+		resp.Error = fmt.Sprintf("cluster reset: %s", resetErr)
 	}
 	return resp
 }
+
+// writeResetStatus records a terminal Phase=Reset status after a cluster reset.
+// It is best-effort: write errors are swallowed so they never mask the reset result.
+func writeResetStatus(sink status.StatusSink, role actualstate.Role, resetErr error) {
+	if sink == nil {
+		return
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	sink.Record(ctx, status.BuildStatus(status.BuildParams{
+		Role:     role,
+		IsReset:  true,
+		ResetErr: resetErr,
+		Now:      time.Now().UTC().Format(time.RFC3339),
+		BootID:   readBootID(),
+		Version:  version.Version,
+	}))
+}

internal/provider/reset_handler_test.go

@@ -38,7 +38,7 @@ func TestHandleClusterResetRunsReset(t *testing.T) {
 	clusterYAML := "cluster:\n  providerConfig:\n    cluster_root_path: " + root + "\n  role: init\n"
 	fr := &resetFakeRunner{}
 
-	resp := handleClusterReset(resetEvent(t, clusterYAML), fr)
+	resp := handleClusterReset(resetEvent(t, clusterYAML), fr, nil)
 	if resp.Error != "" {
 		t.Fatalf("unexpected error: %s", resp.Error)
 	}
@@ -51,14 +51,14 @@ func TestHandleClusterResetRunsReset(t *testing.T) {
 }
 
 func TestHandleClusterResetNilEventIsSafe(t *testing.T) {
-	if resp := handleClusterReset(nil, &resetFakeRunner{}); resp.Error != "" {
+	if resp := handleClusterReset(nil, &resetFakeRunner{}, nil); resp.Error != "" {
 		t.Fatalf("nil event must be a safe no-op, got %q", resp.Error)
 	}
 }
 
 func TestHandleClusterResetNoClusterIsNoop(t *testing.T) {
 	fr := &resetFakeRunner{}
-	resp := handleClusterReset(resetEvent(t, "other: value\n"), fr)
+	resp := handleClusterReset(resetEvent(t, "other: value\n"), fr, nil)
 	if resp.Error != "" {
 		t.Fatalf("unexpected error: %s", resp.Error)
 	}
@@ -73,7 +73,7 @@ func TestHandleClusterResetRejectsOversizedPayload(t *testing.T) {
 	for i := range big {
 		big[i] = 'a'
 	}
-	resp := handleClusterReset(&pluggable.Event{Data: string(big)}, &resetFakeRunner{})
+	resp := handleClusterReset(&pluggable.Event{Data: string(big)}, &resetFakeRunner{}, nil)
 	if resp.Error == "" {
 		t.Fatal("expected reset to reject oversized payload, got nil error")
 	}
@@ -83,7 +83,7 @@ func TestHandleClusterResetIgnoresTokenValidation(t *testing.T) {
 	// Reset must work even with an empty/invalid cluster_token (no token gate).
 	root := t.TempDir()
 	clusterYAML := "cluster:\n  cluster_token: \"\"\n  providerConfig:\n    cluster_root_path: " + root + "\n"
-	resp := handleClusterReset(resetEvent(t, clusterYAML), &resetFakeRunner{})
+	resp := handleClusterReset(resetEvent(t, clusterYAML), &resetFakeRunner{}, nil)
 	if resp.Error != "" {
 		t.Fatalf("reset must not fail on empty token, got %q", resp.Error)
 	}