feat: K3s cluster recovery tooling and documentation

Recovery tooling:
- cluster_manager.py: Add rejoin_node command for fresh node recovery
- cluster_manager.py: Add SSH key setup with sshpass fallback
- k3s.yaml: Update to 2-node cluster config (pumped-piglet, still-fawn)
- cluster.yaml: Add network interface documentation for nodes

Scripts:
- check-config.sh: Fix HA config check to use correct API endpoint

Documentation:
- Add comprehensive "Mistakes Made" section to prevent future errors
- Document face recognition bootstrap requirement
- Document camera IP verification workflow

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Author: web-flow

docs/runbooks/k3s-cluster-recovery-2026-01.md

@@ -247,3 +247,59 @@ face_recognition:
 ```
 
 Reference: [Frigate Face Recognition Docs](https://docs.frigate.video/configuration/face_recognition/)
+
+## Mistakes Made During Recovery (Learn From These)
+
+This section documents errors made during the recovery session to prevent repeating them.
+
+### 1. Claimed GPU Was Working Without Verification
+**What happened**: Reported "still-fawn is working" after seeing node Ready status, but never verified GPU passthrough was actually configured.
+
+**Result**: User had to discover Frigate had no GPU access. The passthrough job hadn't run because its SSH key secret was missing.
+
+**Lesson**: "Node Ready" ≠ "Everything Works". Always verify hardware passthrough with actual device checks (`ls /dev/dri/`, `vainfo`, etc.).
+
+### 2. Guessed Ollama Model Names Instead of Checking Config
+**What happened**: Assumed HA needed `gemma3:4b` for conversation without checking what model HA actually expected.
+
+**Result**: Pulled wrong model. HA actually needed `qwen2.5:7b` for conversation (gemma3 is for llmvision image analysis).
+
+**Lesson**: Never guess configuration. Always read the actual config:
+```bash
+ssh -p 22222 root@192.168.4.240 "cat /mnt/data/supervisor/homeassistant/.storage/core.config_entries" | jq '.data.entries[] | select(.domain == "ollama")'
+```
+
+### 3. Forgot HAOS SSH Access Exists
+**What happened**: Repeatedly used slow `qm guest exec` method to access HAOS instead of direct SSH.
+
+**Result**: Wasted time and frustrated user who had already documented SSH access.
+
+**Lesson**: HAOS has SSH on port 22222:
+```bash
+ssh -p 22222 root@192.168.4.240 "command"
+```
+Check existing documentation before reinventing access methods.
+
+### 4. Did Not Check for Missing SOPS Secret Before Claiming Job Would Work
+**What happened**: Said the GPU passthrough K8s Job would handle everything, but the job required an SSH key secret that didn't exist.
+
+**Result**: Job couldn't run. Had to manually create the SOPS-encrypted secret and run passthrough commands by hand.
+
+**Lesson**: Before claiming automation will work, verify all dependencies exist:
+```bash
+kubectl get secret <secret-name> -n <namespace>
+```
+
+### 5. Assumed Face Recognition Would Auto-Populate
+**What happened**: Configured face_recognition in Frigate config and expected faces to appear in Train tab automatically.
+
+**Result**: Train tab remained empty. Frigate requires at least one face to be manually uploaded before it starts collecting detected faces.
+
+**Lesson**: Read the docs. Face recognition needs bootstrapping - upload one face first, then detections appear.
+
+### 6. Did Not Update Camera IPs in Config After Network Changes
+**What happened**: Living room camera IP changed from 192.168.1.140 to 192.168.1.183, but initially kept checking old IP.
+
+**Result**: Camera appeared offline until config was updated with correct IP.
+
+**Lesson**: When cameras stop working, verify current IP first (check router DHCP leases or Reolink app), then update Frigate config.

proxmox/homelab/config/cluster.yaml

@@ -26,6 +26,19 @@ nodes:
     fqdn: still-fawn.maas
     role: compute
     enabled: true
+    # Network configuration
+    network:
+      primary:
+        interface: enp3s0
+        bridge: vmbr0
+        ip: 192.168.4.17/24
+        gateway: 192.168.4.1
+        speed: 1000  # 1 Gbps onboard NIC
+      secondary:
+        interface: enx803f5df89165
+        ip: 192.168.4.18/24
+        speed: 2500  # 2.5 Gbps USB adapter (Realtek RTL8156B)
+        # Use for PBS restores and bulk transfers
     # AMD GPU passthrough for Frigate VAAPI hardware encoding
     gpu_passthrough:
       enabled: true
@@ -52,6 +65,18 @@ nodes:
     fqdn: pumped-piglet.maas
     role: compute
     enabled: true
+    # Network configuration
+    network:
+      primary:
+        interface: enx803f5df8d628
+        bridge: vmbr0
+        ip: 192.168.4.175/24
+        gateway: 192.168.4.1
+        speed: 2500  # 2.5 Gbps USB adapter (Realtek RTL8156B)
+      secondary:
+        interface: eno1
+        # Onboard NIC - currently unused/down
+        enabled: false
     # GPU passthrough for RTX 3070 to K3s VM for Ollama/AI workloads
     gpu_passthrough:
       enabled: true

proxmox/homelab/config/k3s.yaml

@@ -36,47 +36,30 @@ kube_vip:
 
 # K3s control plane nodes (VMs running K3s server)
 # These run inside Proxmox VMs, not on bare metal
+# Current cluster: 2 nodes (pumped-piglet, still-fawn) after Jan 2026 recovery
 control_plane_nodes:
-  - name: k3s-vm-pve
-    # VM runs on this Proxmox host
-    proxmox_host: pve
-    vmid: 107
-    ip: 192.168.4.238
-    # Primary node used for cluster operations
-    is_primary: true
-
   - name: k3s-vm-pumped-piglet-gpu
     proxmox_host: pumped-piglet
     vmid: 105
     ip: 192.168.4.210
-    is_primary: false
+    is_primary: true
     # GPU passthrough for Ollama/AI workloads
     gpu_enabled: true
 
-  - name: k3s-vm-fun-bedbug
-    proxmox_host: fun-bedbug
-    vmid: 114
-    ip: 192.168.4.203
-    is_primary: false
-    # Quorum-only node - minimal resources (1 vCPU, 4GB RAM)
-    # AMD A9-9400 thermal limited - reduced from 2 cores due to 92°C temps
-    role: quorum
-
   - name: k3s-vm-still-fawn
     proxmox_host: still-fawn
     vmid: 108
     ip: 192.168.4.212
     is_primary: false
+    # AMD GPU + Coral TPU for Frigate
 
 # TLS-SAN entries for API server certificate
 # These are added to /etc/rancher/k3s/config.yaml on each control plane node
 tls_san:
   # The VIP must be in SAN for kube-vip to work
   - 192.168.4.79
-  # All control plane node IPs (auto-added by K3s, but explicit is safer)
-  - 192.168.4.238
+  # All control plane node IPs
   - 192.168.4.210
-  - 192.168.4.203
   - 192.168.4.212
   # DNS names if using DNS-based access
   - k3s.homelab