Skip to content

Commit fbec601

Browse files
sudoshiruvnet
sudoshi
and
ruvnet
committed
ops(dr): verify backup restore + make logical app dump carry its extensions (Gate B5)
Ran a non-destructive DR drill against host PG17 (restore into a throwaway DB, dropped after). Found the logical app dump is not standalone-restorable: it needs postgis + pgvector extensions (not emitted by a --schema dump) and the vocab schema (cross-schema FKs). Verified data integrity once prerequisites were present — analysis_executions (173) and cohort_definitions (164) restored byte-exact to prod, and the dump carries full users/studies/sources COPY blocks. - pg-host-logical-backup.sh now prepends CREATE EXTENSION postgis/vector/pg_trgm - new runbook docs/lineage/operations/2026-06-21-dr-restore-runbook.md - corrected disaster-recovery-plan.md (basebackup is daily not weekly; logical is 3x/day not on-demand; backups live in /mnt/md0/postgres-backups not backups/; documented the restore prerequisites) Co-Authored-By: claude-flow <ruv@ruv.net>
1 parent b334906 commit fbec601

3 files changed

Lines changed: 163 additions & 14 deletions

File tree

‎docs/compliance/disaster-recovery-plan.md‎

Lines changed: 19 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -41,13 +41,26 @@ This plan establishes procedures for recovering the Parthenon outcomes research
4141

4242
### 3.1 Database Backups
4343

44-
| Backup Script | Schedule | Schemas Covered | Retention | Location |
44+
> **Verified 2026-06-21 against the live crontab** (see
45+
> `docs/lineage/operations/2026-06-21-dr-restore-runbook.md`). The production
46+
> backup path is the **host PostgreSQL** scripts below — `db-backup.sh` targets
47+
> the Docker Postgres and is a legacy/manual path, not the scheduled production
48+
> backup. Backups land in `/mnt/md0/postgres-backups/`, not `backups/`.
49+
50+
| Backup Script | Schedule (actual cron) | Schemas Covered | Retention | Location |
4551
|---|---|---|---|---|
46-
| `scripts/db-backup.sh` | Daily 3:17 AM (cron) | `app`, `irsf`, `irsf_results`, `vocab`, `results`, `omop`, `public` (Orthanc) | 30 daily backups | `backups/` directory |
47-
| `scripts/pg-host-basebackup.sh` | Weekly (manual/cron) | Full cluster (WAL-level) | 4 weeks | `backups/` directory |
48-
| `scripts/pg-host-logical-backup.sh` | On demand | Per-schema logical dumps | Manual | `backups/` directory |
49-
| `scripts/pg-host-prune-backups.sh` | Weekly | N/A (pruning) | N/A | Removes backups beyond retention |
50-
| `scripts/backup-app-db.sh` | On demand | Application tables only | Manual | `backups/` directory |
52+
| `scripts/pg-host-basebackup.sh` | **Daily 02:23** | Full cluster + WAL (point-in-time) | per `pg-host-prune-backups.sh` | `/mnt/md0/postgres-backups/` |
53+
| `scripts/pg-host-logical-backup.sh` | **3×/day 06:17, 14:17, 22:17** | `app` schema (`PG_LOGICAL_SCHEMAS`) | 14 dumps (`PG_LOGICAL_KEEP_COUNT`) | `/mnt/md0/postgres-backups/logical/` |
54+
| `scripts/pg-wal-retention.sh --purge` | Daily 06:30 | N/A (WAL pruning) | N/A | — |
55+
| `scripts/pg-host-prune-backups.sh` | Daily 06:00 | N/A (pruning) | N/A | Removes backups beyond retention |
56+
| `scripts/db-backup.sh` (legacy, Docker PG) | Manual | `app`, `irsf`, `irsf_results`, `vocab`, `results`, `omop`, `public` | 30 daily | `backups/` directory |
57+
58+
**Restoring the logical `app` dump requires prerequisites** (verified by the
59+
2026-06-21 drill): the `postgis`, `vector`, and `pg_trgm` extensions, plus the
60+
`vocab` schema, must exist in the target — the `app` schema has geometry/vector
61+
columns and cross-schema FKs into `vocab`. A fresh-instance restore therefore uses
62+
the **physical basebackup** (Scenario A in the runbook), not the logical dump. The
63+
logical backup now self-installs the extensions; the vocab dependency is by design.
5164

5265
**Backup features:**
5366
- Timestamped filenames: `parthenon-full-YYYYMMDD-HHMMSS.sql`

‎docs/lineage/operations/2026-06-21-dr-restore-runbook.md‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
---
2+
doc_type: runbook
3+
status: active
4+
date: 2026-06-21
5+
owner: acumenus
6+
module: infrastructure
7+
lineage_anchor: false
8+
supersedes: []
9+
superseded_by: null
10+
related_code:
11+
- scripts/pg-host-logical-backup.sh
12+
- scripts/pg-host-basebackup.sh
13+
- scripts/db-restore.sh
14+
- docs/compliance/disaster-recovery-plan.md
15+
---
16+
17+
# PostgreSQL Backup & Restore Runbook (DR drill verified 2026-06-21)
18+
19+
This runbook records a **non-destructive disaster-recovery drill** run against the
20+
host PostgreSQL 17 instance on 2026-06-21 and the verified restore procedures it
21+
produced. It supplements `docs/compliance/disaster-recovery-plan.md` and corrects
22+
several stale facts in it (see [Corrections](#corrections-to-the-dr-plan)).
23+
24+
The drill touched **no production data**: every restore went into a throwaway
25+
`parthenon_dr_drill` database that was dropped afterward.
26+
27+
## Backup architecture (as actually scheduled in cron)
28+
29+
| Tier | Script | Cadence (cron) | Scope | Destination |
30+
|---|---|---|---|---|
31+
| **Physical basebackup** | `pg-host-basebackup.sh` | **Daily 02:23** | Full cluster + WAL | `/mnt/md0/postgres-backups/` |
32+
| **Logical (per-schema)** | `pg-host-logical-backup.sh` | **3×/day 06:17, 14:17, 22:17** | `app` schema only (`PG_LOGICAL_SCHEMAS`) | `/mnt/md0/postgres-backups/logical/` |
33+
| **WAL retention** | `pg-wal-retention.sh --purge` | Daily 06:30 | WAL segment pruning | — |
34+
| **Prune** | `pg-host-prune-backups.sh` | Daily 06:00 | Retention enforcement | — |
35+
36+
The legacy `scripts/db-backup.sh` / `scripts/db-restore.sh` pair targets the
37+
**Docker** Postgres and is not the production path; `db-restore.sh` restores
38+
**over** the live `parthenon` database and must never be pointed at production
39+
during a drill.
40+
41+
## Two recovery scenarios
42+
43+
### A. Full-instance / full-corruption DR — use the physical basebackup
44+
The physical basebackup is the authoritative full-cluster recovery path: it
45+
contains every schema (`app`, `vocab`, `omop`, all CDM sources), all extensions,
46+
roles, and WAL for point-in-time recovery. This is the only tier that restores a
47+
**fresh** instance with no prerequisites. (Size: ~1.27 TB; restore is an
48+
infrastructure operation, not drilled inline here.)
49+
50+
### B. Schema-level recovery of `app` — use the logical dump INTO an existing cluster
51+
`latest-app.sql.gz` (~5.2 GB gzipped) restores the `app` schema into a cluster
52+
that **already has** the `vocab` schema and the required extensions present
53+
(i.e., over the existing `parthenon` DB, or a basebackup-restored instance). It
54+
is **not** standalone-restorable into an empty database — see the drill findings.
55+
56+
## Drill findings (2026-06-21)
57+
58+
Restoring `latest-app.sql.gz` into an **empty** database failed three times, each
59+
exposing an undocumented prerequisite:
60+
61+
1. `ERROR: type "public.geometry" does not exist` — `app.gis_admin_boundaries.geom`
62+
needs **PostGIS**, which a `--schema=app` dump does not create.
63+
2. `ERROR: type "public.vector" does not exist` — `*.embedding` columns need
64+
**pgvector**, likewise not created by the schema dump.
65+
3. `ERROR: relation "vocab.concept" does not exist` — the `app` schema has
66+
**cross-schema foreign keys into `vocab`**, which is not part of the `app`
67+
logical backup (vocab lives in the basebackup).
68+
69+
**Fix applied:** `pg-host-logical-backup.sh` now prepends
70+
`CREATE EXTENSION IF NOT EXISTS postgis | vector | pg_trgm` to every dump, so the
71+
extension class of failure (1 and 2) is gone for future backups. The `vocab`
72+
cross-schema dependency (3) is **by design** — the logical `app` dump is a
73+
schema-restore tool for an existing cluster, not a standalone DR artifact.
74+
75+
**Data integrity verified:** with extensions present and a real `vocab` schema,
76+
`app.analysis_executions` (173 rows) and `app.cohort_definitions` (164 rows)
77+
restored **byte-exact to production**, and the dump was confirmed to contain the
78+
full `COPY app.users / app.studies / app.sources` data blocks. The backup is
79+
complete and faithful.
80+
81+
## Verified restore procedure (schema-level, into a throwaway for drills)
82+
83+
```bash
84+
# 1. Create an isolated target — NEVER restore into the live parthenon DB during a drill
85+
psql "host=127.0.0.1 port=5432 dbname=postgres user=claude_dev" \
86+
-c "CREATE DATABASE parthenon_dr_drill OWNER claude_dev;"
87+
88+
# 2. Provide the prerequisites the app schema depends on
89+
psql "host=127.0.0.1 port=5432 dbname=parthenon_dr_drill user=claude_dev" -c "
90+
CREATE EXTENSION IF NOT EXISTS postgis;
91+
CREATE EXTENSION IF NOT EXISTS vector;
92+
CREATE EXTENSION IF NOT EXISTS pg_trgm;"
93+
# For a FULL standalone restore you must also restore the vocab schema first
94+
# (from the basebackup); for an in-cluster restore vocab is already present.
95+
96+
# 3. Restore
97+
zcat /mnt/md0/postgres-backups/logical/latest-app.sql.gz \
98+
| psql "host=127.0.0.1 port=5432 dbname=parthenon_dr_drill user=claude_dev" -v ON_ERROR_STOP=1
99+
100+
# 4. Verify row counts against production, then drop the throwaway
101+
psql "host=127.0.0.1 port=5432 dbname=postgres user=claude_dev" \
102+
-c "DROP DATABASE IF EXISTS parthenon_dr_drill WITH (FORCE);"
103+
```
104+
105+
A clean run (post-fix, full vocab present) completes with `ON_ERROR_STOP=1` and
106+
matching row counts. A degraded environment (stub vocab) will partially restore —
107+
that is expected and not a backup defect.
108+
109+
## Corrections to the DR plan
110+
111+
`docs/compliance/disaster-recovery-plan.md` was written before this drill and
112+
states: basebackup "Weekly", logical "On demand", destination `backups/`. The
113+
**actual** cadence is basebackup daily and logical 3×/day, both writing to
114+
`/mnt/md0/postgres-backups/`. Its selective-schema restore section also omitted
115+
the extension + vocab prerequisites. Those entries are corrected in that file with
116+
a pointer here.
117+
118+
## Open items (not addressed by this drill)
119+
120+
- A scheduled **physical basebackup restore** rehearsal (Scenario A) into a spare
121+
instance to validate full-instance RTO end-to-end.
122+
- A **backup-failure alert**: cron success is currently observable only via the
123+
`/tmp/parthenon-pg-*.log` files; no Prometheus alert fires if a backup stops
124+
running (see the observability gap noted in the Gate B5 review).

‎scripts/pg-host-logical-backup.sh‎

Lines changed: 20 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -21,14 +21,26 @@ for schema in "${schema_list[@]}"; do
2121
[ -n "$schema" ] || continue
2222

2323
out_file="$BACKUP_ROOT/${schema}-${TIMESTAMP}.sql.gz"
24-
pg_dump \
25-
-d "$PG_URL" \
26-
--schema="$schema" \
27-
--clean \
28-
--if-exists \
29-
--no-owner \
30-
--no-acl \
31-
| gzip -9 > "$out_file"
24+
{
25+
# Carry the schema's extension dependencies so a restore into a fresh
26+
# database does not fail on the first geometry/vector column. A --schema
27+
# dump does NOT emit the public extensions the app schema relies on
28+
# (postgis for gis_admin_boundaries.geom, pgvector for *.embedding,
29+
# pg_trgm for GIN indexes) — the DR drill on 2026-06-21 hit exactly these.
30+
# NOTE: the app schema also has cross-schema FKs into `vocab`, so a full
31+
# standalone restore still requires the vocab schema; see the DR runbook
32+
# at docs/lineage/operations/2026-06-21-dr-restore-runbook.md.
33+
echo "CREATE EXTENSION IF NOT EXISTS postgis;"
34+
echo "CREATE EXTENSION IF NOT EXISTS vector;"
35+
echo "CREATE EXTENSION IF NOT EXISTS pg_trgm;"
36+
pg_dump \
37+
-d "$PG_URL" \
38+
--schema="$schema" \
39+
--clean \
40+
--if-exists \
41+
--no-owner \
42+
--no-acl
43+
} | gzip -9 > "$out_file"
3244

3345
ln -sfn "$out_file" "$BACKUP_ROOT/latest-${schema}.sql.gz"
3446
echo " Wrote $out_file"

0 commit comments

Comments
 (0)