Skip to content

Commit 79be92d

Browse files
web-flowclaude
web-flow
and
claude
committed
docs: add RCA and runbook for PBS disk-full VM crash incident
PBS 3TB pool hit 100% → vzdump backup froze VM 105 filesystem → thaw timed out → VM crashed for 28 hours with no alerts. Configured PBS prune/GC/verify schedules, recovered 828GB. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent a683a7b commit 79be92d

2 files changed

Lines changed: 451 additions & 0 deletions

File tree

‎docs/runbooks/pbs-backup-maintenance-runbook.md‎

Lines changed: 216 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,216 @@
1+
# Runbook: PBS Backup Maintenance
2+
3+
## Overview
4+
5+
Proxmox Backup Server (PBS) requires three scheduled maintenance tasks to prevent disk exhaustion: **prune**, **garbage collection (GC)**, and **verify**. Without these, the backup datastore grows unbounded and can crash VMs during backup (see RCA: `pbs-disk-full-vm-crash-rca.md`).
6+
7+
## Architecture
8+
9+
```
10+
┌─────────────────────────────────────────────────────────────────────┐
11+
│ PBS Maintenance Pipeline (Daily) │
12+
├─────────────────────────────────────────────────────────────────────┤
13+
│ │
14+
│ 22:30 vzdump backup runs (PVE cluster job) │
15+
│ │ └─ Backs up all VMs/CTs except 103,108 │
16+
│ ▼ │
17+
│ 04:00 Prune job (PBS) │
18+
│ │ └─ Marks old snapshots for removal (keep-daily=3, weekly=2) │
19+
│ ▼ │
20+
│ 05:00 Garbage collection (PBS) │
21+
│ │ └─ Deletes unreferenced chunks, reclaims disk space │
22+
│ ▼ │
23+
│ sat 03:00 Verify job (PBS, weekly) │
24+
│ └─ Validates backup integrity, checks for bit rot │
25+
│ │
26+
│ PBS Container: LXC 103 on pumped-piglet.maas │
27+
│ Datastore: homelab-backup at /mnt/homelab-backup │
28+
│ Storage: ZFS pool local-3TB-backup │
29+
│ │
30+
└─────────────────────────────────────────────────────────────────────┘
31+
```
32+
33+
## Configuration Reference
34+
35+
| Setting | Value | Location |
36+
|---------|-------|----------|
37+
| PBS container | LXC 103 on pumped-piglet | `qm list` |
38+
| PBS IP | 192.168.4.211 | `/etc/pve/storage.cfg` |
39+
| PBS Web UI | `https://proxmox-backup-server.maas:8007` | Browser |
40+
| Datastore | `homelab-backup` at `/mnt/homelab-backup` | PBS config |
41+
| ZFS pool | `local-3TB-backup` | pumped-piglet host |
42+
| Backup schedule | 2nd and 22nd of month at 22:30 | PVE cluster job |
43+
| Backup exclusions | VM 103 (PBS itself), VM 108 (still-fawn K3s) | PVE cluster job |
44+
| Retention (vzdump) | keep-daily=3, keep-weekly=2 | PVE cluster job |
45+
| Retention (PBS prune) | keep-daily=3, keep-weekly=2 | PBS prune job |
46+
| GC schedule | Daily at 05:00 | PBS datastore config |
47+
| Verify schedule | Saturday at 03:00 | PBS verify job |
48+
| Age key (SOPS) | `~/.config/sops/age/keys.txt` | Local Mac |
49+
50+
## Common Operations
51+
52+
### Check Datastore Health
53+
54+
```bash
55+
# Pool free space (should be >500 GB)
56+
ssh root@pumped-piglet.maas "zfs list -o name,used,avail local-3TB-backup"
57+
58+
# GC status (last run, bytes removed)
59+
ssh root@pumped-piglet.maas "pct exec 103 -- proxmox-backup-manager garbage-collection status homelab-backup"
60+
61+
# List scheduled jobs
62+
ssh root@pumped-piglet.maas "pct exec 103 -- proxmox-backup-manager prune list"
63+
ssh root@pumped-piglet.maas "pct exec 103 -- proxmox-backup-manager verify-job list"
64+
ssh root@pumped-piglet.maas "pct exec 103 -- proxmox-backup-manager datastore show homelab-backup"
65+
66+
# Backup inventory (all snapshots)
67+
ssh root@pumped-piglet.maas "pvesm list homelab-backup"
68+
69+
# Dedup stats
70+
ssh root@pumped-piglet.maas "pct exec 103 -- proxmox-backup-manager garbage-collection status homelab-backup"
71+
# Look for: Deduplication factor, On-Disk usage
72+
```
73+
74+
### Manual Prune
75+
76+
```bash
77+
# Run prune job immediately
78+
ssh root@pumped-piglet.maas "pct exec 103 -- proxmox-backup-manager prune run homelab-prune"
79+
```
80+
81+
### Manual Garbage Collection
82+
83+
```bash
84+
# Run GC immediately (can take 10-30 min on large datastores)
85+
ssh root@pumped-piglet.maas "pct exec 103 -- proxmox-backup-manager garbage-collection start homelab-backup"
86+
87+
# NOTE: GC will fail with ENOSPC if the pool is 100% full
88+
# In that case, free space at the ZFS level first (see Emergency section)
89+
```
90+
91+
### Manual Verify
92+
93+
```bash
94+
# Run verify immediately
95+
ssh root@pumped-piglet.maas "pct exec 103 -- proxmox-backup-manager verify-job run homelab-verify"
96+
```
97+
98+
### Check Backup Job Status
99+
100+
```bash
101+
# PVE cluster backup job config
102+
ssh root@pumped-piglet.maas "pvesh get /cluster/backup --output-format json | jq '.[0]'"
103+
104+
# Recent backup task logs
105+
ssh root@pumped-piglet.maas "journalctl | grep -iE 'vzdump|backup' | tail -20"
106+
```
107+
108+
## Emergency: Pool Full
109+
110+
If `local-3TB-backup` hits 100%, follow this order:
111+
112+
### 1. Free ZFS-Level Space First
113+
114+
GC cannot run on a full pool (needs space to update atimes). Free space at the ZFS level:
115+
116+
```bash
117+
# Check for stale vzdump temp directories
118+
ssh root@pumped-piglet.maas "du -sh /local-3TB-backup/backup-tmpdir/*"
119+
# Delete if stale (no active backup running):
120+
ssh root@pumped-piglet.maas "rm -rf /local-3TB-backup/backup-tmpdir/vzdumptmp*"
121+
122+
# Check for old ZFS snapshots
123+
ssh root@pumped-piglet.maas "zfs list -t snapshot -o name,used,creation -r local-3TB-backup"
124+
# Destroy old snapshots:
125+
ssh root@pumped-piglet.maas "zfs destroy local-3TB-backup/DATASET@SNAPSHOT_NAME"
126+
127+
# Verify space freed
128+
ssh root@pumped-piglet.maas "zfs list -o name,avail local-3TB-backup"
129+
```
130+
131+
### 2. Run Prune Then GC
132+
133+
```bash
134+
# Prune first (marks snapshots for removal)
135+
ssh root@pumped-piglet.maas "pct exec 103 -- proxmox-backup-manager prune run homelab-prune"
136+
137+
# Then GC (actually deletes chunks and reclaims space)
138+
ssh root@pumped-piglet.maas "pct exec 103 -- proxmox-backup-manager garbage-collection start homelab-backup"
139+
```
140+
141+
### 3. If a VM Crashed
142+
143+
If a VM is stopped due to a frozen filesystem from a failed backup:
144+
145+
```bash
146+
# Check VM status
147+
ssh root@pumped-piglet.maas "qm status <VMID>"
148+
149+
# Start the VM (only after freeing disk space)
150+
ssh root@pumped-piglet.maas "qm start <VMID>"
151+
152+
# If start fails, check for lock files
153+
ssh root@pumped-piglet.maas "ls -la /var/lock/qemu-server/lock-<VMID>.conf"
154+
ssh root@pumped-piglet.maas "rm /var/lock/qemu-server/lock-<VMID>.conf"
155+
```
156+
157+
## Modifying Retention Policy
158+
159+
### PBS Prune Job
160+
161+
```bash
162+
# Update retention (e.g., keep more daily backups)
163+
ssh root@pumped-piglet.maas "pct exec 103 -- proxmox-backup-manager prune update homelab-prune \
164+
--keep-daily 7 --keep-weekly 4 --keep-monthly 3"
165+
```
166+
167+
### PVE Cluster Backup Job
168+
169+
```bash
170+
# Update via pvesh (match PBS prune retention)
171+
ssh root@pumped-piglet.maas "pvesh set /cluster/backup/backup-ff3d789f-f52b \
172+
--prune-backups keep-daily=7,keep-weekly=4,keep-monthly=3"
173+
```
174+
175+
### GitOps (pbs-storage.yaml)
176+
177+
The desired state is in `proxmox/homelab/config/pbs-storage.yaml`. Update `prune_backups` there and run `pbs apply` to reconcile.
178+
179+
**Note**: As of 2026-02-28, the YAML config (`keep-daily=7, weekly=4, monthly=3`) is out of sync with the live cluster job (`keep-daily=3, weekly=2`). Reconcile when appropriate.
180+
181+
## Troubleshooting
182+
183+
| Symptom | Cause | Fix |
184+
|---------|-------|-----|
185+
| GC fails with ENOSPC | Pool 100% full | Free ZFS-level space first (see Emergency section) |
186+
| VM stopped after backup | Freeze/thaw timeout on full disk | Free space, start VM |
187+
| Backup fails with "storage inactive" | PBS LXC 103 down or DNS issue | `pct start 103`, check DNS |
188+
| Prune removes nothing | All snapshots within retention window | Expected behavior — check with `pvesm list` |
189+
| GC removes 0 bytes | No dead chunks (nothing pruned recently) | Run prune first, then GC |
190+
| High dedup ratio (>15x) | Many similar VMs being backed up | Normal for homelab |
191+
| Verify shows errors | Bit rot or corrupt chunks | Restore affected backups to verify, re-backup if needed |
192+
193+
## Monitoring (TODO)
194+
195+
**Not yet implemented** — see RCA `pbs-disk-full-vm-crash-rca.md` for context.
196+
197+
Needed:
198+
- ZFS pool space alerting (warn 80%, critical 90%)
199+
- Backup failure notifications to HA
200+
- VM status monitoring (detect crashed VMs)
201+
202+
## Related Documents
203+
204+
- RCA: `docs/runbooks/pbs-disk-full-vm-crash-rca.md`
205+
- PBS migration: `docs/runbooks/pbs-migration-to-pumped-piglet.md`
206+
- PBS connectivity: `docs/runbooks/proxmox-backup-server-storage-connectivity.md`
207+
- Secret management: `docs/secret-management.md`
208+
- PBS CLI tool: `proxmox/homelab/src/homelab/pbs_cli.py`
209+
- PBS config: `proxmox/homelab/config/pbs-storage.yaml`
210+
211+
## Tags
212+
213+
pbs, proxmox-backup-server, backup, vzdump, prune, garbage-collection, gc, verify, retention, zfs, disk-full, enospc, maintenance, schedule, datastore, homelab-backup, pumped-piglet, lxc-103, runbook, disaster-recovery, proxmox, pve, proxmocks
214+
215+
**Owner**: Homelab
216+
**Last Updated**: 2026-02-28

0 commit comments

Comments
 (0)