Skip to content

Commit 319de66

Browse files
web-flowclaude
web-flow
and
claude
committed
docs: add RCA for fun-bedbug K3s VM re-enable after thermal paste
- Thermal paste replacement dropped temps from 92°C to 54.5°C (-37.5°C) - Documented D-Bus socket activation issue on fun-bedbug - Documented stale etcd data cleanup procedure - Updated thermal runbook with Phase 2 results Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 029bc53 commit 319de66

2 files changed

Lines changed: 172 additions & 3 deletions

File tree

‎docs/rca/RCA-fun-bedbug-K3s-VM-Reenable-2026-01-27.md‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
# RCA: fun-bedbug K3s VM Re-enable After Thermal Paste Replacement
2+
3+
**Date:** 2026-01-27
4+
**Duration:** ~15 minutes (D-Bus fix + etcd cleanup)
5+
**Severity:** Low (no production impact)
6+
**Root Cause Category:** D-Bus service not running after reboot + stale etcd data
7+
8+
---
9+
10+
## Summary
11+
12+
Successfully re-enabled the K3s control plane VM on fun-bedbug (VMID 114) via Crossplane GitOps. The node had been disabled since 2026-01-25 due to thermal issues (92°C+ under load). After cleaning and applying new thermal paste, temperatures dropped dramatically, allowing the node to rejoin the cluster.
13+
14+
**Temperature improvement: 92°C → 54.5°C (37.5°C reduction)**
15+
16+
---
17+
18+
## Timeline
19+
20+
| Time (PST) | Event |
21+
|------------|-------|
22+
| 2026-01-18 | K3s VM on fun-bedbug disabled due to thermal throttling (92°C) |
23+
| 2026-01-25 | VM formally stopped via Crossplane (`started: false`) |
24+
| 2026-01-26 | Physical maintenance: case cleaning + thermal paste replacement |
25+
| 2026-01-27 13:00 | Crossplane GitOps change pushed (`started: true`) |
26+
| 2026-01-27 13:06 | Crossplane failed to start VM - D-Bus socket missing |
27+
| 2026-01-27 13:14 | D-Bus service started manually |
28+
| 2026-01-27 13:14 | VM started, K3s began joining cluster |
29+
| 2026-01-27 13:15 | etcd join failed - stale etcd data from previous attempt |
30+
| 2026-01-27 13:16 | Cleared etcd data, K3s restarted |
31+
| 2026-01-27 13:17 | Node joined cluster successfully |
32+
| 2026-01-27 13:18 | Crossplane reconciled, showing SYNCED: True |
33+
34+
---
35+
36+
## Technical Details
37+
38+
### Issue 1: D-Bus Service Not Running
39+
40+
**Symptom:**
41+
```
42+
org.freedesktop.DBus.Error.FileNotFound: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
43+
```
44+
45+
**Root Cause:**
46+
- D-Bus was stopped during a reboot on Jan 26
47+
- Socket activation (`dbus.socket`) failed to trigger D-Bus on the Jan 26 15:40 boot
48+
- Proxmox services (pvedaemon) started before D-Bus was activated
49+
- `qm start` requires D-Bus for inter-process communication
50+
51+
**Resolution:**
52+
```bash
53+
ssh root@fun-bedbug.maas "systemctl start dbus"
54+
```
55+
56+
**Why D-Bus Wasn't Auto-Started:**
57+
- D-Bus uses socket activation (no `WantedBy=` in unit file)
58+
- Something in the boot sequence bypassed socket activation
59+
- Likely a race condition between pvedaemon and dbus.socket
60+
61+
### Issue 2: Stale etcd Data
62+
63+
**Symptom:**
64+
```
65+
Failed to test etcd connection: failed to get etcd status: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
66+
```
67+
68+
**Root Cause:**
69+
- Previous K3s join attempts (Jan 16-18) left etcd data in `/var/lib/rancher/k3s/server/db/etcd/`
70+
- The stale data contained invalid member IDs and certificates
71+
- K3s couldn't rejoin with mismatched etcd state
72+
73+
**Resolution:**
74+
```bash
75+
ssh root@fun-bedbug.maas "qm guest exec 114 -- bash -c 'systemctl stop k3s && rm -rf /var/lib/rancher/k3s/server/db/etcd/* && systemctl start k3s'"
76+
```
77+
78+
---
79+
80+
## Thermal Improvement Results
81+
82+
### Before (2026-01-18)
83+
| Metric | Value |
84+
|--------|-------|
85+
| Temperature | 92°C (critical threshold: 100°C) |
86+
| vCPUs | 2 |
87+
| Load Average | 2.90 |
88+
| Status | Thermal throttling, VM disabled |
89+
90+
### After Thermal Paste Replacement (2026-01-27)
91+
| Metric | Value |
92+
|--------|-------|
93+
| Temperature | **54.5°C** (normal threshold: 70°C) |
94+
| vCPUs | 1 |
95+
| Load Average | 0.18 |
96+
| Status | Running, joined cluster |
97+
98+
**Temperature delta: -37.5°C**
99+
100+
The ATOPNUC MA90's AMD A9-9400 APU (2016 Bristol Ridge, 15W TDP) was likely suffering from dried-out original thermal paste after years of operation.
101+
102+
---
103+
104+
## Current Cluster State
105+
106+
```
107+
NAME STATUS ROLES AGE INTERNAL-IP
108+
k3s-vm-fun-bedbug Ready control-plane,etcd 27s 192.168.4.192
109+
k3s-vm-pumped-piglet-gpu Ready control-plane,etcd,master 3d14h 192.168.4.210
110+
k3s-vm-pve Ready control-plane,etcd 46h 192.168.4.193
111+
k3s-vm-still-fawn Ready control-plane,etcd,master 3d14h 192.168.4.212
112+
```
113+
114+
4-node K3s cluster with all nodes as control-plane + etcd members.
115+
116+
---
117+
118+
## Lessons Learned
119+
120+
### 1. D-Bus Socket Activation is Not Reliable on fun-bedbug
121+
122+
**Action:** Consider adding explicit D-Bus startup to Proxmox boot sequence or monitoring for missing D-Bus socket.
123+
124+
### 2. Clear etcd Data Before Rejoining After Long Outage
125+
126+
**Action:** When a K3s control-plane node has been offline for days, clear etcd state before rejoining:
127+
```bash
128+
rm -rf /var/lib/rancher/k3s/server/db/etcd/*
129+
```
130+
131+
### 3. Thermal Paste Replacement is Worth It for Old Hardware
132+
133+
**Action:** The dramatic 37.5°C improvement justifies the effort. Consider thermal paste replacement for other aging homelab hardware.
134+
135+
---
136+
137+
## Prevention
138+
139+
| Item | Action |
140+
|------|--------|
141+
| D-Bus monitoring | Add check for `/run/dbus/system_bus_socket` existence in host health checks |
142+
| etcd cleanup script | Create snippet that clears etcd before K3s start on cold boot |
143+
| Thermal monitoring | Alert if fun-bedbug exceeds 70°C |
144+
145+
---
146+
147+
## Related
148+
149+
- Crossplane VM config: `gitops/clusters/homelab/instances/k3s-vm-fun-bedbug.yaml`
150+
- Thermal runbook: `docs/runbooks/fun-bedbug-thermal-management.md`
151+
- Cloud-init snippet: `scripts/k3s/snippets/k3s-server-fun-bedbug.yaml`
152+
153+
**Tags:** fun-bedbug, k3s, crossplane, dbus, etcd, thermal, thermal-paste, proxmox, atopnuc, amd-a9

‎docs/runbooks/fun-bedbug-thermal-management.md‎

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -78,18 +78,34 @@ Add a 120mm USB fan pointed at the case vents. Expected improvement: 5-10°C.
7878
7979
## Results
8080
81+
### Phase 1: vCPU Reduction (2026-01-18)
82+
8183
| Metric | Before | After |
8284
|--------|--------|-------|
8385
| vCPUs | 2 | 1 |
8486
| Load Average | 2.90 | 1.04 |
8587
| CPU Idle | 27% | 50% |
8688
| Temperature | 92°C | 89-90°C |
8789
90+
### Phase 2: Thermal Paste Replacement (2026-01-26)
91+
92+
| Metric | Before | After |
93+
|--------|--------|-------|
94+
| Temperature (idle) | 89-90°C | ~50°C |
95+
| Temperature (VM running) | 92°C+ | **54.5°C** |
96+
| Improvement | - | **-37.5°C** |
97+
98+
The original thermal paste was dried out after years of operation. Replacement with fresh paste dramatically improved thermal performance.
99+
100+
## Completed Improvements
101+
102+
1. ~~External USB fan~~ - Not needed after thermal paste replacement
103+
2. **Dust cleaning** - Done 2026-01-26
104+
3. **Thermal paste** - Replaced 2026-01-26, dramatic improvement
105+
88106
## Future Improvements
89107
90-
1. **External USB fan** - Point 120mm fan at exhaust vents
91-
2. **Dust cleaning** - Open case and blow out accumulated dust
92-
3. **Thermal paste** - Repaste with Arctic MX-4 or Noctua NT-H1 if still hot after above steps
108+
1. **D-Bus monitoring** - Add check for socket existence after reboot (see RCA)
93109
94110
## Re-enabling DaemonSets
95111

0 commit comments

Comments
 (0)