Skip to content

Commit 9664e96

Browse files
web-flowclaude
andcommitted
docs: add thermal analysis and final decision to k3s-vm-pve blog
- fun-bedbug hit 91°C (9°C from critical) - shut down - pve runs at 57°C with 3GB memory headroom - acceptable - Final cluster: 3 nodes (pumped-piglet, still-fawn, pve) - Added lessons: thermals trump isolation, NotReady nodes aren't inert Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 65a0b26 commit 9664e96

1 file changed

Lines changed: 69 additions & 1 deletion

File tree

docs/source/md/blog-k3s-vm-pve-fourth-node.md

Lines changed: 69 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -210,6 +210,70 @@ ssh root@pve.maas "qm status 107"
210210

211211
A spare tire in the trunk. Not using resources, but ready.
212212

213+
## Update: The Thermal Reality Check
214+
215+
An hour after deployment, Grafana told a different story.
216+
217+
fun-bedbug's temperature graph showed the AMD A9-9400 sitting at **91°C sustained** - just 9 degrees below the critical threshold. Before the K3s VM started, it idled at 45-55°C. The tiny ATOPNUC MA90 with its passive cooling simply couldn't handle the load.
218+
219+
The question became: which node should be the standby?
220+
221+
| Node | Thermal Risk | Resource Risk | Isolation |
222+
|------|-------------|---------------|-----------|
223+
| fun-bedbug | **High** (91°C, passive cooling) | Low | Good - dedicated box |
224+
| pve | Low (57°C, proper cooling) | **Medium** (shares host with OPNsense/MAAS) | Poor - critical services |
225+
226+
The concern with pve was valid: it runs OPNsense (the router) and MAAS (infrastructure provisioning). If the K3s VM misbehaved and starved resources, the entire network could go down.
227+
228+
But the numbers told a clearer story:
229+
230+
```
231+
=== pve host with k3s-vm-pve running ===
232+
CPU Temp: 56-57°C (crit @ 105°C) - 48°C headroom
233+
Load Average: 0.85 (4-core CPU) - barely noticeable
234+
Memory: 12.8/15.7GB used - 3GB headroom
235+
236+
VM allocation:
237+
- OPNsense: 4GB
238+
- MAAS: 5GB
239+
- k3s-vm-pve: 4GB
240+
- Host free: ~3GB
241+
```
242+
243+
Compare to fun-bedbug:
244+
245+
```
246+
=== fun-bedbug with k3s-vm-fun-bedbug running ===
247+
CPU Temp: 91°C (crit @ 100°C) - 9°C headroom
248+
Load Average: 1.5+ (2-core CPU) - saturated
249+
```
250+
251+
**Decision: Shut down fun-bedbug, keep pve running.**
252+
253+
The thermal risk on fun-bedbug was immediate and hardware-threatening. The resource risk on pve was theoretical and manageable. A 3GB memory buffer with stable VMs is acceptable; running 9°C from thermal shutdown is not.
254+
255+
```bash
256+
# Via Crossplane (the right way)
257+
# k3s-vm-fun-bedbug.yaml: started: false, onBoot: false
258+
git commit -m "fix(k3s): disable k3s-vm-fun-bedbug due to thermal issues"
259+
git push
260+
flux reconcile kustomization flux-system --with-source
261+
262+
# Remove the NotReady ghost from cluster
263+
kubectl delete node k3s-vm-fun-bedbug
264+
```
265+
266+
Final cluster state:
267+
268+
```
269+
NAME STATUS ROLES VERSION
270+
k3s-vm-pumped-piglet-gpu Ready control-plane,etcd,master v1.35.0+k3s1
271+
k3s-vm-still-fawn Ready control-plane,etcd,master v1.35.0+k3s1
272+
k3s-vm-pve Ready control-plane,etcd v1.35.0+k3s1
273+
```
274+
275+
Three nodes. Proper etcd quorum. One node can fail without losing the cluster. And no hardware cooking itself to death.
276+
213277
## Lessons Learned
214278

215279
1. **Homogeneity is a myth.** Even in your own infrastructure, hosts have quirks. Document them.
@@ -222,9 +286,13 @@ A spare tire in the trunk. Not using resources, but ready.
222286

223287
5. **Fail-fast checks pay dividends.** A 5-second curl in cloud-init would have made the failure obvious immediately.
224288

289+
6. **Thermals trump isolation.** Running a node 9°C from critical is worse than sharing a host with other VMs. Hardware damage is permanent; resource contention is manageable.
290+
291+
7. **NotReady nodes are not inert.** They hold etcd membership, leases, and can cause leader election issues. If a node is intentionally offline, remove it from the cluster entirely.
292+
225293
---
226294

227-
*k3s-vm-pve sleeps now, powered off, its brief moment of confusion forgotten. The cluster hums along with four nodes registered, three active. And somewhere in a Git commit, a one-line diff tells the whole story: `vmbr0``vmbr25gbe`. The infrastructure remembers what I almost forgot.*
295+
*k3s-vm-pve hums along now at 57°C, sharing its host with OPNsense and MAAS. fun-bedbug sits powered off, its passive cooling finally getting a break. The cluster runs with three nodes - proper quorum, proper thermals, proper GitOps. And somewhere in a Grafana dashboard, a temperature graph tells the real story: 91°C was too close. 57°C is just right.*
228296

229297
---
230298

0 commit comments

Comments
 (0)