Skip to content

Commit f915f2f

Browse files
web-flowclaude
web-flow
and
claude
committed
docs: add secrets backup blog post and crucible scripts README
- Blog post explaining using Crucible for secrets backup - Comprehensive README indexing all crucible scripts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent f798a74 commit f915f2f

2 files changed

Lines changed: 478 additions & 0 deletions

File tree

‎docs/blog/2026-01-26-crucible-secrets-backup.md‎

Lines changed: 278 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,278 @@
1+
# Using Oxide Crucible as a Secrets Backup for Your Homelab
2+
3+
**Date**: January 26, 2026
4+
**Author**: Claude + Human collaboration
5+
**Tags**: oxide, crucible, secrets, backup, homelab, sops, ssh, configuration
6+
7+
---
8+
9+
## The Problem: Where Do You Put the Stuff That Can't Go in Git?
10+
11+
Every homelab has secrets that can't be checked into version control:
12+
13+
- SOPS age keys (the key that decrypts your other keys)
14+
- SSH private keys
15+
- `.env` files with API tokens and passwords
16+
- Home Assistant `secrets.yaml`
17+
- Database credentials
18+
- Certificate private keys
19+
20+
These files are critical. Lose them and you're locked out of your own infrastructure. But where do you store them safely?
21+
22+
Cloud storage? Now your secrets depend on someone else's infrastructure. Local backup drive? Single point of failure. Password manager? Doesn't handle files well.
23+
24+
## The Solution: Distributed Storage You Control
25+
26+
I already had [Oxide Crucible running on a $30 mini PC](/docs/blog/2026-01-25-budget-oxide-storage-sled.md) for VM storage experiments. Then it hit me: Crucible provides 12.5GB of distributed block storage to each of my 5 Proxmox hosts. That's plenty of space for configs and secrets.
27+
28+
```
29+
┌─────────────────────────────────────────────────────────────────┐
30+
│ Secrets Backup Architecture │
31+
├─────────────────────────────────────────────────────────────────┤
32+
│ │
33+
│ Mac (source of truth) │
34+
│ ~/.config/sops/age/keys.txt ─────┐ │
35+
│ ~/.ssh/id_ed25519_pve ───────────┼──► sync-secrets-to-crucible│
36+
│ ~/code/home/.env ────────────────┘ │ │
37+
│ │ │
38+
│ ▼ │
39+
│ ┌─────────────────────────────────────────────────────────┐ │
40+
│ │ pve:/mnt/crucible-storage/ │ │
41+
│ │ secrets/ │ │
42+
│ │ ├── sops/age.key │ │
43+
│ │ ├── ssh/id_ed25519_pve │ │
44+
│ │ └── env/homelab.env │ │
45+
│ │ services/ │ │
46+
│ │ └── haos/20260126-231325/ │ │
47+
│ │ ├── configuration.yaml │ │
48+
│ │ ├── automations.yaml │ │
49+
│ │ └── secrets.yaml │ │
50+
│ └─────────────────────────────────────────────────────────┘ │
51+
│ │ │
52+
│ │ replicate-crucible-storage │
53+
│ ▼ │
54+
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
55+
│ │still-fawn│ │pumped- │ │chief- │ │fun- │ │
56+
│ │ /mnt/ │ │piglet │ │horse │ │bedbug │ │
57+
│ │crucible- │ │ /mnt/ │ │ /mnt/ │ │ /mnt/ │ │
58+
│ │storage/ │ │crucible- │ │crucible- │ │crucible- │ │
59+
│ │ │ │storage/ │ │storage/ │ │storage/ │ │
60+
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
61+
│ │ │ │ │ │
62+
│ └────────────┴─────┬──────┴────────────┘ │
63+
│ │ │
64+
│ ▼ │
65+
│ ┌─────────────────────────┐ │
66+
│ │ proper-raptor │ │
67+
│ │ (Crucible storage) │ │
68+
│ │ 192.168.4.189 │ │
69+
│ └─────────────────────────┘ │
70+
│ │
71+
└─────────────────────────────────────────────────────────────────┘
72+
```
73+
74+
## What Gets Backed Up
75+
76+
### Local Secrets (from Mac)
77+
78+
```bash
79+
./scripts/crucible/sync-secrets-to-crucible.sh
80+
```
81+
82+
| File | Purpose |
83+
|------|---------|
84+
| `~/.config/sops/age/keys.txt` | SOPS decryption key |
85+
| `~/.ssh/id_ed25519_pve` | Proxmox SSH key |
86+
| `~/code/home/.env` | Homelab credentials |
87+
88+
### Service Configs (from VMs/containers)
89+
90+
```bash
91+
./scripts/crucible/backup-services-to-crucible.sh
92+
```
93+
94+
| Service | Files Backed Up |
95+
|---------|-----------------|
96+
| Home Assistant | `configuration.yaml`, `automations.yaml`, `secrets.yaml`, `core.config_entries` |
97+
| Frigate | `config.yml`, model list |
98+
| Ollama | Model list, model details |
99+
100+
### Proxmox Host Configs
101+
102+
```bash
103+
./scripts/crucible/backup-configs-to-crucible.sh
104+
```
105+
106+
| Config | Purpose |
107+
|--------|---------|
108+
| `/etc/pve/*` | Cluster config, VM/LXC definitions, storage.cfg |
109+
| `/etc/network/interfaces` | Network configuration |
110+
| `/etc/systemd/system/crucible-*` | Crucible service definitions |
111+
| ZFS pool/dataset lists | Storage topology |
112+
113+
## Replication: 5 Copies of Everything
114+
115+
The magic happens with replication. After backing up to pve, one command copies everything to all other hosts:
116+
117+
```bash
118+
./scripts/crucible/replicate-crucible-storage.sh
119+
```
120+
121+
```
122+
[23:16:40] Starting Crucible storage replication
123+
[23:16:40] Primary: pve
124+
[23:16:40] Replicas: still-fawn.maas pumped-piglet.maas chief-horse.maas fun-bedbug.maas
125+
126+
=== still-fawn.maas ===
127+
[23:16:41] Syncing secrets to still-fawn.maas...
128+
[23:16:42] Syncing services to still-fawn.maas...
129+
130+
=== pumped-piglet.maas ===
131+
[23:16:43] Syncing secrets to pumped-piglet.maas...
132+
...
133+
134+
[23:16:48] Storage contents:
135+
--- pve ---
136+
44K /mnt/crucible-storage/secrets/
137+
104K /mnt/crucible-storage/services/
138+
--- still-fawn.maas ---
139+
44K /mnt/crucible-storage/secrets/
140+
104K /mnt/crucible-storage/services/
141+
...
142+
```
143+
144+
Now the same secrets exist on all 5 Proxmox hosts. If one host dies, four others have copies.
145+
146+
## The Scripts
147+
148+
All scripts are idempotent - safe to run repeatedly.
149+
150+
### sync-secrets-to-crucible.sh
151+
152+
```bash
153+
#!/bin/bash
154+
# Syncs local secrets from Mac to Crucible storage
155+
156+
LOCAL_SECRETS=(
157+
"sops/age.key:$HOME/.config/sops/age/keys.txt"
158+
"ssh/id_ed25519_pve:$HOME/.ssh/id_ed25519_pve"
159+
"env/homelab.env:$REPO_ROOT/proxmox/homelab/.env"
160+
)
161+
162+
for pair in "${LOCAL_SECRETS[@]}"; do
163+
dest_path="${pair%%:*}"
164+
src_path="${pair#*:}"
165+
scp "$src_path" "root@pve:/mnt/crucible-storage/secrets/$dest_path"
166+
done
167+
```
168+
169+
### backup-services-to-crucible.sh
170+
171+
Uses `qm guest exec` to pull configs from HAOS VM:
172+
173+
```bash
174+
# HAOS config lives at /mnt/data/supervisor/homeassistant/
175+
ssh "root@chief-horse.maas" "
176+
qm guest exec 116 -- cat '$ha_config/configuration.yaml' | \
177+
jq -r '.[\"out-data\"]' > '$dest_dir/configuration.yaml'
178+
qm guest exec 116 -- cat '$ha_config/secrets.yaml' | \
179+
jq -r '.[\"out-data\"]' > '$dest_dir/secrets.yaml'
180+
"
181+
```
182+
183+
### replicate-crucible-storage.sh
184+
185+
Uses tar pipe through local machine (avoids SSH host key issues between Proxmox hosts):
186+
187+
```bash
188+
for replica in "${REPLICA_HOSTS[@]}"; do
189+
ssh "root@$PRIMARY_HOST" "tar -C '$src' -cf - ." | \
190+
ssh "root@$replica" "tar -C '$dest' -xf -"
191+
done
192+
```
193+
194+
## Why This Works
195+
196+
1. **No cloud dependency** - Everything stays on your network
197+
2. **Distributed** - 5 copies across 5 hosts
198+
3. **Fast recovery** - Secrets are already on every Proxmox host
199+
4. **Git-friendly** - Scripts are version controlled, secrets aren't
200+
5. **Cheap** - Uses existing Crucible storage (costs $0 extra)
201+
202+
## Recovery Scenarios
203+
204+
### Lost my Mac
205+
206+
SSH to any Proxmox host, secrets are at `/mnt/crucible-storage/secrets/`:
207+
208+
```bash
209+
ssh root@pve
210+
cat /mnt/crucible-storage/secrets/sops/age.key
211+
cat /mnt/crucible-storage/secrets/ssh/id_ed25519_pve
212+
```
213+
214+
### Proxmox host died
215+
216+
Four other hosts have identical copies. Pick one.
217+
218+
### proper-raptor (Crucible storage) died
219+
220+
This is the single point of failure today. The `/mnt/crucible-storage` would become unavailable. But! The last-synced data still exists on each host's local mount until you disconnect NBD.
221+
222+
**Future improvement**: Add 2 more MA90 sleds (~$60) for true 3-way Crucible replication. Then even the storage backend survives hardware failure.
223+
224+
### Home Assistant config corrupted
225+
226+
Restore from timestamped backup:
227+
228+
```bash
229+
ls /mnt/crucible-storage/services/haos/
230+
# 20260125-231325/ 20260126-080000/ ...
231+
232+
cat /mnt/crucible-storage/services/haos/20260125-231325/configuration.yaml
233+
```
234+
235+
## Automation Ideas
236+
237+
Add to cron for automatic backups:
238+
239+
```bash
240+
# Daily secrets sync (2 AM)
241+
0 2 * * * /home/user/code/home/scripts/crucible/sync-secrets-to-crucible.sh
242+
243+
# Daily service backup (3 AM)
244+
0 3 * * * /home/user/code/home/scripts/crucible/backup-services-to-crucible.sh
245+
246+
# Daily replication (4 AM)
247+
0 4 * * * /home/user/code/home/scripts/crucible/replicate-crucible-storage.sh
248+
```
249+
250+
Or trigger on git push with a hook.
251+
252+
## Total Cost
253+
254+
| Item | Cost |
255+
|------|------|
256+
| MA90 mini PC (Crucible storage) | $30 |
257+
| USB 2.5GbE adapters (5x) | $75 |
258+
| 2.5GbE switch | $60 |
259+
| **Total** | **$165** |
260+
261+
For that, you get distributed secrets storage across 5 hosts with 12.5GB per host. Not bad for a homelab.
262+
263+
## Conclusion
264+
265+
Crucible wasn't designed for secrets backup. It's enterprise distributed block storage for VMs. But the primitives it provides - replicated storage accessible from multiple hosts - solve the secrets problem elegantly.
266+
267+
Sometimes the best tool for a job is the one you already have running.
268+
269+
---
270+
271+
**Related posts:**
272+
- [Budget Oxide Storage Sled](/docs/blog/2026-01-25-budget-oxide-storage-sled.md)
273+
274+
**Scripts:**
275+
- [scripts/crucible/sync-secrets-to-crucible.sh](/scripts/crucible/sync-secrets-to-crucible.sh)
276+
- [scripts/crucible/backup-services-to-crucible.sh](/scripts/crucible/backup-services-to-crucible.sh)
277+
- [scripts/crucible/backup-configs-to-crucible.sh](/scripts/crucible/backup-configs-to-crucible.sh)
278+
- [scripts/crucible/replicate-crucible-storage.sh](/scripts/crucible/replicate-crucible-storage.sh)

0 commit comments

Comments
 (0)