Skip to content

Commit e0e98de

Browse files
web-flowclaude
web-flow
and
claude
committed
docs(runbooks): add K3s cluster recovery runbook Jan 2026
Documents complete recovery process including: - TLS-SAN for kube-vip VIP access - AMD GPU + Coral TPU passthrough to VM - MetalLB IP assignments - SOPS secrets created - Ollama models for HA (qwen2.5:7b, gemma3:4b) - SSH access to HAOS - Verification commands Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 4ebe582 commit e0e98de

1 file changed

Lines changed: 214 additions & 0 deletions

File tree

‎docs/runbooks/k3s-cluster-recovery-2026-01.md‎

Lines changed: 214 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
# K3s Cluster Recovery Runbook - January 2026
2+
3+
## Summary
4+
5+
Full cluster recovery after still-fawn disk failure. Documents all steps to restore K3s cluster with GPU passthrough, Frigate, Ollama, and all dependent services.
6+
7+
## Cluster Configuration
8+
9+
### Nodes
10+
| Node | IP | Role | Hardware |
11+
|------|-----|------|----------|
12+
| k3s-vm-pumped-piglet-gpu | 192.168.4.210 | control-plane, primary | RTX 3070 GPU |
13+
| k3s-vm-still-fawn | 192.168.4.212 | control-plane | AMD RX 580 GPU, Coral USB TPU |
14+
15+
### kube-vip VIP
16+
- **VIP**: 192.168.4.79
17+
- **IMPORTANT**: VIP must be in TLS-SAN for kubectl to work via VIP
18+
19+
## Recovery Steps
20+
21+
### 1. TLS-SAN Configuration for VIP Access
22+
23+
The kubeconfig must use the VIP for HA access, but VIP must be in the K3s certificate SANs.
24+
25+
```bash
26+
# Check current TLS-SAN status
27+
cd ~/code/home/proxmox/homelab
28+
poetry run python -m homelab.k3s_manager status
29+
30+
# Add VIP to TLS-SAN and rotate certs
31+
poetry run python -m homelab.k3s_manager prepare-kube-vip --dry-run # Preview
32+
poetry run python -m homelab.k3s_manager prepare-kube-vip # Apply
33+
34+
# Update kubeconfig to use VIP
35+
sed -i '' 's/192.168.4.210/192.168.4.79/' ~/kubeconfig
36+
```
37+
38+
Config file: `proxmox/homelab/config/k3s.yaml`
39+
40+
### 2. GPU Passthrough to K3s VMs
41+
42+
#### NVIDIA GPU (pumped-piglet)
43+
Handled by GPU Operator with time-slicing:
44+
- Config: `gitops/clusters/homelab/infrastructure/gpu-operator/time-slicing-config.yaml`
45+
- 4 virtual GPUs from 1 physical RTX 3070
46+
47+
#### AMD GPU + Coral TPU (still-fawn)
48+
Requires manual passthrough job because Crossplane API lacks USB/PCI permissions.
49+
50+
**Prerequisites:**
51+
1. SSH key secret in SOPS: `gitops/clusters/homelab/infrastructure/crossplane/proxmox-ssh-key.sops.yaml`
52+
2. VM must be STOPPED before passthrough changes
53+
54+
**Process:**
55+
```bash
56+
# 1. Drain the node
57+
KUBECONFIG=~/kubeconfig kubectl drain k3s-vm-still-fawn --ignore-daemonsets --delete-emptydir-data --force
58+
59+
# 2. Stop the VM
60+
ssh root@still-fawn.maas "qm stop 108"
61+
62+
# 3. Add passthrough devices
63+
ssh root@still-fawn.maas "
64+
qm set 108 --usb0 host=1a6e:089a,usb3=1 # Coral TPU mode 1
65+
qm set 108 --usb1 host=18d1:9302,usb3=1 # Coral TPU mode 2
66+
qm set 108 --hostpci0 0000:01:00,pcie=1 # AMD GPU
67+
"
68+
69+
# 4. Start VM
70+
ssh root@still-fawn.maas "qm start 108"
71+
72+
# 5. Wait for boot, then load AMD GPU driver
73+
sleep 60
74+
ssh root@still-fawn.maas "qm guest exec 108 -- bash -c 'apt-get update && apt-get install -y linux-modules-extra-\$(uname -r) && modprobe amdgpu'"
75+
76+
# 6. Verify GPU is available
77+
ssh root@still-fawn.maas "qm guest exec 108 -- ls -la /dev/dri/"
78+
# Should show: card0, renderD128
79+
80+
# 7. Uncordon node
81+
KUBECONFIG=~/kubeconfig kubectl uncordon k3s-vm-still-fawn
82+
83+
# 8. Restart Frigate to pick up GPU
84+
KUBECONFIG=~/kubeconfig kubectl rollout restart deployment/frigate -n frigate
85+
```
86+
87+
**USB3 passthrough note**: The `usb3=1` flag is critical - it uses xHCI controller for USB 3.0 speeds, matching LXC performance.
88+
89+
### 3. MetalLB IP Assignments
90+
91+
**CRITICAL**: Always use explicit IP annotations to prevent race conditions.
92+
93+
| IP | Service | Namespace |
94+
|----|---------|-----------|
95+
| 192.168.4.80 | traefik | kube-system |
96+
| 192.168.4.81 | frigate | frigate |
97+
| 192.168.4.82 | stable-diffusion-webui | stable-diffusion |
98+
| 192.168.4.84 | frigate-webrtc-udp | frigate |
99+
| 192.168.4.85 | ollama-lb | ollama |
100+
| 192.168.4.120 | samba-lb | samba |
101+
102+
Documentation: `gitops/clusters/homelab/infrastructure-config/metallb-config/IP-ASSIGNMENTS.md`
103+
104+
### 4. SOPS Secrets
105+
106+
All secrets use age encryption with key: `age1uwvq3llqjt666t4ckls9wv44wcpxxwlu8svqwx5kc7v76hncj94qg3tsna`
107+
108+
Created secrets:
109+
- `gitops/clusters/homelab/apps/claudecodeui/secrets/ghcr-credentials.sops.yaml`
110+
- `gitops/clusters/homelab/apps/frigate/secrets/frigate-credentials.sops.yaml`
111+
- `gitops/clusters/homelab/apps/frigate/secrets/ghcr-creds.sops.yaml`
112+
- `gitops/clusters/homelab/infrastructure/monitoring/secrets/smtp-credentials.sops.yaml`
113+
- `gitops/clusters/homelab/infrastructure/tailscale/secrets/operator-oauth.sops.yaml`
114+
- `gitops/clusters/homelab/apps/samba/secrets/samba-users.sops.yaml`
115+
- `gitops/clusters/homelab/infrastructure/crossplane/proxmox-ssh-key.sops.yaml`
116+
117+
### 5. Ollama Models for Home Assistant
118+
119+
HA uses two Ollama integrations:
120+
121+
1. **ollama** (conversation): `qwen2.5:7b`
122+
2. **llmvision** (image analysis): `gemma3:4b`
123+
124+
```bash
125+
# Pull required models
126+
KUBECONFIG=~/kubeconfig kubectl exec -n ollama deploy/ollama-gpu -- ollama pull qwen2.5:7b
127+
KUBECONFIG=~/kubeconfig kubectl exec -n ollama deploy/ollama-gpu -- ollama pull gemma3:4b
128+
129+
# Verify
130+
KUBECONFIG=~/kubeconfig kubectl exec -n ollama deploy/ollama-gpu -- ollama list
131+
132+
# Reload HA to pick up models
133+
ssh -p 22222 root@192.168.4.240 "ha core restart"
134+
```
135+
136+
**Finding the model HA expects:**
137+
```bash
138+
ssh -p 22222 root@192.168.4.240 "cat /mnt/data/supervisor/homeassistant/.storage/core.config_entries" | jq '.data.entries[] | select(.domain == "ollama") | .subentries[].data.model'
139+
```
140+
141+
### 6. Frigate Camera Network
142+
143+
Cameras are on ISP network (192.168.1.x), not homelab (192.168.4.x).
144+
145+
| Camera | IP | Connection | Status |
146+
|--------|-----|------------|--------|
147+
| TrendNet | 192.168.1.107 | Wired | Works |
148+
| Reolink Doorbell | 192.168.1.10 | 5GHz WiFi | Depends on AT&T router |
149+
| Living Room (E1 Zoom) | 192.168.1.140 | 5GHz WiFi | Depends on AT&T router |
150+
151+
**Routing**: K3s pods reach 192.168.1.x via OPNsense (192.168.4.1) which routes to ISP network.
152+
153+
## SSH Access to HAOS
154+
155+
HAOS VM 116 on chief-horse.maas has SSH via dropbear on port 22222.
156+
157+
```bash
158+
# Direct SSH (if key configured)
159+
ssh -p 22222 root@192.168.4.240 "command"
160+
161+
# Via qm guest exec (always works)
162+
ssh root@chief-horse.maas "qm guest exec 116 -- command"
163+
164+
# Using scripts
165+
/Users/10381054/code/home/scripts/haos/read-from-ha.sh /path/in/haos
166+
/Users/10381054/code/home/scripts/haos/copy-to-ha.sh local_file /path/in/haos
167+
```
168+
169+
Setup guide: `docs/source/md/homeassistant-os-ssh-access-setup.md`
170+
171+
## Verification Commands
172+
173+
```bash
174+
# Cluster health
175+
KUBECONFIG=~/kubeconfig kubectl get nodes
176+
KUBECONFIG=~/kubeconfig kubectl get pods -A --field-selector=status.phase!=Running
177+
178+
# GPU availability
179+
KUBECONFIG=~/kubeconfig kubectl get nodes -o custom-columns="NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"
180+
181+
# Frigate GPU access
182+
KUBECONFIG=~/kubeconfig kubectl exec -n frigate deploy/frigate -- ls -la /dev/dri/
183+
KUBECONFIG=~/kubeconfig kubectl exec -n frigate deploy/frigate -- vainfo --display drm --device /dev/dri/renderD128
184+
185+
# Coral TPU
186+
KUBECONFIG=~/kubeconfig kubectl logs -n frigate deploy/frigate | grep -i "TPU\|coral"
187+
188+
# Ollama
189+
curl -s http://192.168.4.85/api/tags | jq '.models[].name'
190+
191+
# HA integrations
192+
HA_URL=http://192.168.4.240:8123 /Users/10381054/code/home/scripts/haos/list-integrations.sh | grep -i ollama
193+
```
194+
195+
## Key Files
196+
197+
| Purpose | Path |
198+
|---------|------|
199+
| K3s config | `proxmox/homelab/config/k3s.yaml` |
200+
| K3s manager | `proxmox/homelab/src/homelab/k3s_manager.py` |
201+
| GPU passthrough job | `gitops/clusters/homelab/instances/job-vm108-passthrough.yaml` |
202+
| GPU time-slicing | `gitops/clusters/homelab/infrastructure/gpu-operator/time-slicing-config.yaml` |
203+
| MetalLB IPs | `gitops/clusters/homelab/infrastructure-config/metallb-config/IP-ASSIGNMENTS.md` |
204+
| Frigate config | `gitops/clusters/homelab/apps/frigate/configmap.yaml` |
205+
| HA SSH setup | `docs/source/md/homeassistant-os-ssh-access-setup.md` |
206+
207+
## Lessons Learned
208+
209+
1. **Always verify GPU passthrough** - don't assume "node Ready" means GPU works
210+
2. **USB3 flag matters** - `usb3=1` gives near-native Coral TPU performance in VM
211+
3. **VIP needs TLS-SAN** - kubeconfig pointing to VIP fails without cert update
212+
4. **MetalLB race conditions** - explicit IP annotations prevent IP stealing on cluster rebuild
213+
5. **SOPS for all secrets** - never create secrets manually with kubectl
214+
6. **Check HA config for models** - don't guess Ollama model names, read `.storage/core.config_entries`

0 commit comments

Comments
 (0)