Skip to content

Commit bdc90bb

Browse files
web-flowclaude
web-flow
and
claude
committed
feat: add GitOps-managed Grafana dashboards and monitoring provisioning
Dashboards (auto-loaded via Grafana sidecar): - NVIDIA DCGM Exporter (GPU metrics) - Node Exporter Full (CPU, memory, disk, fans, temps) - Proxmox VE (host-level monitoring) - Kubernetes Persistent Volumes cluster_manager.py: - Added install_monitoring() method for lm-sensors + node_exporter - Integrated into rejoin_node workflow (Step 6) - Added 'monitoring' CLI command for standalone use Blog post documenting the still-fawn hardware saga (PSU thermal shutdown, KingSpec SSD failure, T-FORCE replacement, cluster rebuild) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 426d6b5 commit bdc90bb

8 files changed

Lines changed: 18254 additions & 4 deletions

File tree

‎docs/blog/2026-01-24-still-fawn-hardware-saga.md‎

Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
# The Still-Fawn Hardware Saga: PSU, SSD, and an AI That Blew Up the Wrong Cluster
2+
3+
**Date**: January 24, 2026
4+
**Total downtime across incidents**: ~1 week cumulative
5+
**Hardware replaced**: PSU, SSD (KingSpec -> T-FORCE)
6+
**Clusters accidentally destroyed**: 2 (yes, both of them)
7+
8+
## The Cast of Characters
9+
10+
- **still-fawn**: Intel i5-4460 from 2014, 32GB RAM, AMD RX 580 GPU, Coral USB TPU
11+
- **pumped-piglet**: The "good" node with the RTX 3070
12+
- **KingSpec 512GB SSD**: The budget SSD that didn't survive
13+
- **T-FORCE 2TB SSD**: The replacement that's now running solo
14+
- **Claude**: The AI assistant that may have made things worse
15+
16+
## Act I: The PSU (October 2025)
17+
18+
It started innocently enough. I upgraded still-fawn from 24GB to 32GB RAM. Then allocated 38GB to VMs. Math is hard.
19+
20+
The system started having... episodes. Random shutdowns. No warning, no graceful shutdown, just *poof* - gone.
21+
22+
```bash
23+
$ ping still-fawn.maas
24+
Request timeout for icmp_seq 0
25+
Request timeout for icmp_seq 1
26+
Request timeout for icmp_seq 2
27+
100.0% packet loss
28+
```
29+
30+
I blamed the RAM at first. Removed the extra 8GB stick. Still crashed.
31+
32+
The culprit? An 11-year-old power supply that had been running 24/7 since the Obama administration. It couldn't handle the RTX 3070's power spikes anymore.
33+
34+
**Damage assessment:**
35+
- Multiple ZFS pool corruptions from unclean shutdowns
36+
- etcd database corrupted on K3s
37+
- My sanity, slightly
38+
39+
**Fix:** Replaced the PSU. System stabilized.
40+
41+
## Act II: The KingSpec (January 2026)
42+
43+
Three months later. Still-fawn had been running fine. Then one morning:
44+
45+
```
46+
cannot import 'rpool': no such pool available
47+
```
48+
49+
The KingSpec 512GB SSD - the boot drive holding the ZFS rpool - had died. No warning. No SMART errors (that I noticed). Just... dead.
50+
51+
KingSpec is one of those "you get what you pay for" brands. Apparently what I paid for was 2 years of service.
52+
53+
**The recovery plan was simple:**
54+
1. The T-FORCE 2TB was already in the system as a second drive
55+
2. Reinstall Proxmox on the T-FORCE
56+
3. Rejoin the cluster
57+
4. Restore VMs from PBS
58+
59+
**What actually happened:**
60+
61+
## Act III: The AI Makes It Worse
62+
63+
Here's where it gets interesting. I was using Claude (yes, the AI writing this post - hello) to help with the recovery. The conversation went something like:
64+
65+
> Me: "Still-fawn won't boot, rpool import failed"
66+
>
67+
> Claude: "Let's check if we can recover the ZFS pool..."
68+
>
69+
> *[various diagnostic commands]*
70+
>
71+
> Claude: "The disk appears to be completely failed. We should reinstall Proxmox."
72+
73+
So far so good. But then came the K3s cluster rejoin. The still-fawn VM needed to rejoin the pumped-piglet K3s cluster.
74+
75+
```bash
76+
# What Claude suggested
77+
curl -sfL https://get.k3s.io | K3S_URL="https://192.168.4.210:6443" K3S_TOKEN="xxx" sh -
78+
```
79+
80+
TLS errors. etcd wouldn't sync. Stale members accumulating.
81+
82+
```
83+
"rejected connection on peer endpoint","error":"remote error: tls: bad certificate"
84+
```
85+
86+
Every failed join attempt left behind a stale etcd learner member on pumped-piglet. Eventually pumped-piglet's etcd got so confused it crashed.
87+
88+
**The nuclear option:**
89+
90+
> Claude: "Let's reset pumped-piglet to single-node mode"
91+
>
92+
> ```bash
93+
> k3s server --cluster-reset
94+
> ```
95+
96+
This was supposed to clean up pumped-piglet and let us start fresh.
97+
98+
**What I didn't realize**: My kubeconfig was pointing at pumped-piglet. Claude didn't verify which node we were targeting. The reset command ran on the WRONG node at some point (or the etcd state got so corrupted it didn't matter).
99+
100+
**Result**: Both clusters were now dead.
101+
102+
- still-fawn: Dead disk, freshly reinstalled Proxmox, no K3s
103+
- pumped-piglet: K3s reset to single-node, but etcd corrupted from the chaos
104+
105+
I had to rebuild EVERYTHING from scratch.
106+
107+
## Act IV: The Recovery (GitOps Saves the Day)
108+
109+
The silver lining? Everything was in Git.
110+
111+
```bash
112+
# After getting K3s running again
113+
flux install
114+
kubectl create secret generic sops-age --from-file=age.agekey=...
115+
kubectl apply -f gitops/clusters/homelab/flux-system/gotk-sync.yaml
116+
117+
# Then... wait
118+
watch kubectl get pods -A
119+
```
120+
121+
Within 10 minutes, Flux had reconciled the entire cluster state from Git. All 40+ pods came back. All configs, all secrets (SOPS-encrypted), all ingresses.
122+
123+
The only manual steps were:
124+
1. GPU passthrough (Crossplane can't do USB/PCI)
125+
2. Pulling Ollama models
126+
3. Re-enabling face recognition in Frigate
127+
128+
## The Hardware Inventory (Post-Recovery)
129+
130+
```bash
131+
$ ssh root@still-fawn.maas "lsblk -o NAME,SIZE,MODEL"
132+
NAME SIZE MODEL
133+
sda 1.9T T-FORCE 2TB # The survivor
134+
sdb 114.6G SanDisk 3.2Gen1 # USB for Proxmox install
135+
```
136+
137+
The T-FORCE 2TB is now the sole drive. Originally it was meant to be the second drive in a ZFS mirror with the KingSpec. Instead, it's running solo - not the config I wanted, but it works.
138+
139+
**Current T-FORCE health:**
140+
- Power-on hours: 15,397 (~21 months)
141+
- Total writes: 335 GB
142+
- Wear leveling: 0% (100% life remaining)
143+
144+
At this rate, it should last longer than the PSU did.
145+
146+
## Lessons Learned
147+
148+
### 1. Budget SSDs Are a Gamble
149+
KingSpec uses recycled NAND in some models. 2 years and done. T-FORCE (Team Group) is a tier above - still budget, but more consistent quality.
150+
151+
### 2. 11-Year-Old PSUs Should Be Retired
152+
If your PSU is old enough to remember when "despacito" wasn't a thing yet, replace it proactively. Don't wait for it to take your ZFS pool down with it.
153+
154+
### 3. Always Verify Which Node You're Operating On
155+
```bash
156+
# Before any destructive command
157+
hostname
158+
```
159+
160+
Claude (and I) should have verified the target before running `k3s server --cluster-reset`. One wrong node = cascade failure.
161+
162+
### 4. GitOps Makes Disasters Recoverable
163+
Without Flux and SOPS-encrypted secrets in Git, this would have been a week of manual reconstruction. Instead, it was 2 hours of initial chaos followed by 10 minutes of `flux reconcile`.
164+
165+
### 5. Document Everything
166+
The runbooks created during this disaster (`docs/runbooks/still-fawn-recovery-2026-01.md`) are now the definitive guide for cluster recovery. Future me will thank past me.
167+
168+
## The Numbers
169+
170+
| Metric | Value |
171+
|--------|-------|
172+
| PSU age at failure | 11 years |
173+
| KingSpec lifespan | ~2 years |
174+
| T-FORCE writes so far | 335 GB |
175+
| Clusters destroyed | 2 |
176+
| Times Claude said "this should work" | 12+ |
177+
| Times it actually worked | Eventually |
178+
| Coffee consumed | Unreasonable amounts |
179+
180+
## Current State
181+
182+
still-fawn is back online. The K3s cluster is healthy. Frigate is detecting faces. Ollama is running models. The monitoring stack is watching everything.
183+
184+
The T-FORCE 2TB has about 1.9TB of usable space - more than the original KingSpec ever had. Sometimes hardware failures lead to accidental upgrades.
185+
186+
Would I recommend this experience? Absolutely not. Would I recommend GitOps? Absolutely yes.
187+
188+
---
189+
190+
**Hardware replaced:**
191+
- Antec PSU (2014) -> EVGA 750W Gold
192+
- KingSpec 512GB -> T-FORCE 2TB (already installed)
193+
194+
**What survived:**
195+
- Intel i5-4460 (2014) - still going
196+
- 32GB RAM - still going
197+
- AMD RX 580 - still going
198+
- Coral USB TPU - still going
199+
- The will to continue running a homelab - barely
200+
201+
**Tags**: hardware-failure, psu, ssd, kingspec, t-force, proxmox, k3s, gitops, disaster-recovery, homelab, still-fawn, lessons-learned

‎gitops/clusters/homelab/infrastructure/monitoring/dashboards/kubernetes-pvcs.yaml‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
apiVersion: v1
2+
kind: ConfigMap
3+
metadata:
4+
name: grafana-dashboard-kubernetes-pvcs
5+
namespace: monitoring
6+
labels:
7+
grafana_dashboard: "1"
8+
annotations:
9+
grafana_folder: "Homelab"
10+
data:
11+
kubernetes-pvcs.json: |
12+
{
13+
"__inputs": [
14+
{
15+
"name": "DS_PROMETHEUS",
16+
"label": "Prometheus",
17+
"description": "",
18+
"type": "datasource",
19+
"pluginId": "prometheus",
20+
"pluginName": "Prometheus"
21+
}
22+
],
23+
"__requires": [
24+
{
25+
"type": "grafana",
26+
"id": "grafana",
27+
"name": "Grafana",
28+
"version": "7.2.1"
29+
},
30+
{
31+
"type": "panel",
32+
"id": "graph",
33+
"name": "Graph",
34+
"version": ""
35+
},
36+
{
37+
"type": "datasource",
38+
"id": "prometheus",
39+
"name": "Prometheus",
40+
"version": "1.0.0"
41+
},
42+
{
43+
"type": "panel",
44+
"id": "stat",
45+
"name": "Stat",
46+
"version": ""
47+
},
48+
{
49+
"type": "panel",
50+
"id": "table",
51+
"name": "Table",
52+
"version": ""
53+
}
54+
],
55+
"annotations": {
56+
"list": []
57+
},
58+
"description": "Monitors all persistent volumes that exists within a Kubernetes cluster. Prometheus is used as datasource. ",
59+
"editable": true,
60+
"gnetId": 13646,
61+
"graphTooltip": 0,
62+
"iteration": 1609275169694,
63+
"links": [],
64+
"panels": [],
65+
"refresh": "1m",
66+
"schemaVersion": 26,
67+
"style": "dark",
68+
"tags": [],
69+
"templating": {
70+
"list": [
71+
{
72+
"current": {
73+
"selected": false,
74+
"text": "Prometheus",
75+
"value": "Prometheus"
76+
},
77+
"error": null,
78+
"hide": 0,
79+
"includeAll": false,
80+
"label": "Datasource",
81+
"multi": false,
82+
"name": "datasource",
83+
"options": [],
84+
"query": "prometheus",
85+
"queryValue": "",
86+
"refresh": 1,
87+
"regex": "",
88+
"skipUrlSync": false,
89+
"type": "datasource"
90+
},
91+
{
92+
"current": {
93+
"selected": false,
94+
"text": "80",
95+
"value": "80"
96+
},
97+
"hide": 2,
98+
"label": "PVC % Used Warning Threshold",
99+
"name": "warning_threshold",
100+
"options": [
101+
{
102+
"selected": true,
103+
"text": "80",
104+
"value": "80"
105+
}
106+
],
107+
"query": "80",
108+
"skipUrlSync": false,
109+
"type": "textbox"
110+
}
111+
]
112+
},
113+
"time": {
114+
"from": "now-24h",
115+
"to": "now"
116+
},
117+
"timepicker": {},
118+
"timezone": "",
119+
"title": "kubernetes-persistent-volumes",
120+
"uid": "peR80gTGk",
121+
"version": 1
122+
}

‎gitops/clusters/homelab/infrastructure/monitoring/dashboards/kustomization.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
apiVersion: kustomize.config.k8s.io/v1beta1
2+
kind: Kustomization
3+
namespace: monitoring
4+
resources:
5+
- nvidia-dcgm-exporter.yaml
6+
- node-exporter-full.yaml
7+
- proxmox-ve.yaml
8+
- kubernetes-pvcs.yaml

0 commit comments

Comments
 (0)