Skip to content

PLAN.md

Latest commit

 

History

History
440 lines (369 loc) · 20.3 KB

PLAN.md

File metadata and controls

440 lines (369 loc) · 20.3 KB
Raw

Kubernetes Multi-Database Backup Operator — Plan

1. Goal

Build a Kubernetes-native operator that backs up databases running in-cluster (starting with PostgreSQL, later MongoDB, MySQL, and vector databases like Milvus and Weaviate) to a remote S3 bucket, and can restore any of those backups into a target database instance on demand. The system must be designed from day one so that adding a new database engine is a matter of adding a new "engine plugin," not redesigning the CRDs or controllers.

The core design principle: separate the generic backup/restore lifecycle (scheduling, state tracking, retention, S3 storage) from the engine-specific logic (how to actually dump/restore each database type). The operator owns the former; small, swappable "engine" components own the latter.

2. Project Structure & Scaffolding

Use kubebuilder v4 to scaffold the project. It generates CRD types, controller stubs, RBAC manifests, webhook defaults, and Makefile targets with a standard Go project layout.

kube-db-backup/
├── PROJECT                          # kubebuilder project metadata
├── Makefile                         # generated Makefile (docker-build, deploy, test, etc.)
├── go.mod / go.sum
├── cmd/
│   └── main.go                      # entrypoint: sets up manager, registers controllers
├── api/
│   └── v1alpha1/
│       ├── databaseinstance_types.go
│       ├── backupconfig_types.go
│       ├── backup_types.go
│       ├── backupschedule_types.go
│       ├── restore_types.go
│       ├── groupversion_info.go
│       └── zz_generated.deepcopy.go
├── internal/
│   └── controller/
│       ├── backup_controller.go
│       ├── backupschedule_controller.go
│       ├── restore_controller.go
│       └── suite_test.go            # envtest suite
├── pkg/
│   └── engine/
│       ├── contract.go              # EngineResult struct, exit codes, env contracts
│       └── registry.go              # static Engine Registry map
├── config/
│   ├── crd/                         # kubebuilder-generated CRD manifests
│   ├── rbac/                        # RBAC manifests
│   ├── manager/                     # Deployment manifest for the operator
│   └── samples/                     # sample CR YAMLs
├── engines/
│   └── postgres/
│       ├── Dockerfile               # pg-engine image
│       ├── backup.sh                # pg_dump + S3 upload script
│       ├── restore.sh               # S3 download + pg_restore script
│       └── entrypoint.sh            # dispatches backup/restore, writes contract JSON
├── test/
│   └── e2e/                         # kind-based integration tests
├── docker-compose.yml               # local dev: operator + postgres + minio (S3 emulator)
└── PLAN.md

3. Engine Abstraction Strategy

This is the most important decision for future extensibility.

  • Define a database-agnostic concept of "Engine" — a pluggable unit that knows how to: (a) take a backup of a specific database type and stream/ upload it to S3, (b) restore a backup into a target instance of that database type, (c) report progress/size/errors in a common format.
  • Each engine is implemented as a separate container image (e.g. pg-engine, mongo-engine, mysql-engine, milvus-engine, weaviate-engine), not as code compiled into the operator binary. The operator only knows how to run a Job with the right image and pass it a standardized set of environment variables / mounted config.
  • Standardize a minimal "contract" every engine image must follow (see §4 — Engine Contract Specification).
  • Maintain an internal "Engine Registry" — essentially a mapping from an engineType field in the CRD (e.g. postgres, mongodb, mysql, milvus, weaviate) to: default container image, required connection-secret fields, and any engine-specific config schema. This registry can start as a static Go map and later move to its own CRD (BackupEngine) if you want to let users register custom/third-party engines without recompiling the operator.

4. Engine Contract Specification

Every engine image must conform to this contract. The operator sets the following environment variables and mounts secrets when creating a Job.

4.1 Environment Variables (Injected by Operator)

Variable Description Present In
OPERATION backup or restore Both
DB_HOST Database hostname Both
DB_PORT Database port (default: 5432) Both
DB_NAME Database name to backup or restore into Both
DB_USER Database user Both
S3_ENDPOINT S3 endpoint URL Both
S3_BUCKET S3 bucket name Both
S3_PREFIX Prefix within bucket (optional, default: "") Both
BACKUP_ID Unique backup identifier Both
ENCRYPTION_KEY Optional client-side encryption passphrase Backup only
DUMP_FORMAT custom or plain (Postgres-specific, default: custom) Backup only
RESTORE_CLEAN true to drop existing objects before restore Restore only
RESTORE_FORCE true to restore even if target DB is non-empty Restore only

4.2 Mounted Secrets

Secrets are mounted at fixed paths as files (never env vars for sensitive data):

Mount Path Content
/etc/db-secret/password Database password
/etc/s3-secret/access-key S3 access key
/etc/s3-secret/secret-key S3 secret key
/etc/backup-secret/key Client-side encryption key (optional)

4.3 Exit Code Conventions

Exit Code Category Operator Behavior
0 Success Mark phase=Succeeded
1 Engine internal error Mark phase=Failed, do not retry
2 Auth failure Mark phase=Failed, do not retry
3 Connectivity failure Mark phase=Failed, retry with backoff
4 Storage failure Mark phase=Failed, retry with backoff

4.4 Result Contract (JSON on stdout)

The engine writes a single JSON line to stdout (surrounded by ___RESULT_START___ and ___RESULT_END___ markers so the operator can extract it from noisy logs):

{
  "status": "succeeded",
  "started_at": "2026-07-11T10:00:00Z",
  "finished_at": "2026-07-11T10:05:00Z",
  "size_bytes": 1048576,
  "checksum_sha256": "abc123...",
  "s3_objects": ["backups/postgres/my-instance/bkp-20260711-a1b2/dump.custom"],
  "tables_count": 42,
  "error_message": ""
}

On failure, status is failed and error_message is populated. The operator reads this from the Job's container logs, parses it, and updates the Backup or Restore status accordingly.

5. S3 Object Layout

All backups are organized under a predictable path structure:

s3://{bucket}/{prefix}/backups/{engineType}/{instanceName}/{backupId}/
├── dump.{format}            # the actual dump file (e.g. dump.custom)
├── backup-manifest.json     # backup metadata manifest

The backup-manifest.json is written alongside every dump and contains a self-describing record that includes the Backup object's metadata, the engine type, checksum, timestamp, and the dump format. This ensures that even if the Kubernetes Backup CRD is deleted, the S3 objects remain discoverable and restorable by an operator or a human in an emergency.

{
  "backup_id": "bkp-20260711-a1b2",
  "engine_type": "postgres",
  "database_name": "mydb",
  "instance_name": "my-instance",
  "dump_format": "custom",
  "started_at": "2026-07-11T10:00:00Z",
  "finished_at": "2026-07-11T10:05:00Z",
  "size_bytes": 1048576,
  "checksum_sha256": "abc123...",
  "s3_objects": ["backups/postgres/my-instance/bkp-20260711-a1b2/dump.custom"],
  "tables_count": 42
}

6. Postgres Engine Decisions (Phase 1)

6.1 Dump tool and format

  • Use pg_dump (not pg_dumpall) — backs up a single database, which maps naturally to DatabaseInstance.databaseName.
  • Default format: custom (-Fc). This is compressed, allows parallel restore via pg_restore -j, and produces a smaller output than plain SQL. Users can override via DUMP_FORMAT=plain in engineConfig.
  • The engine image includes the PostgreSQL client tools (postgresql-client package) and awscli for S3 upload/download.
  • Compression is handled by pg_dump's custom format internally; additional gzip wrapping is not applied since -Fc already compresses.

6.2 Restore behavior

  • The restore engine reads RESTORE_CLEAN (default: true) — drops existing database objects before restoring, matching pg_restore --clean.
  • The restore engine reads RESTORE_FORCE (default: false) — if set to false and the target database has tables, the restore aborts with exit code 1 and an error message. This prevents accidental overwrites.
  • On success, it runs ANALYZE to update statistics.

6.3 Engine image dependencies

# engines/postgres/Dockerfile
FROM postgres:16-alpine
RUN apk add --no-cache aws-cli
COPY backup.sh restore.sh entrypoint.sh /
RUN chmod +x /backup.sh /restore.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

7. CRD Design (Generic Across Engines)

Design CRDs around the lifecycle, not around Postgres specifically, so the same objects work unchanged when MongoDB/MySQL/vector DBs are added later.

  • DatabaseInstance — generic connection descriptor for any backend. Contains engineType (postgres/mongodb/mysql/milvus/weaviate), host/ port, auth secret reference, and an engineConfig free-form field (map/JSON) for engine-specific options (e.g. Postgres sslMode, MongoDB replica set name, Milvus collection list, Weaviate class names). Using a generic engineConfig blob avoids having to change the CRD schema every time a new database type is added.
  • BackupConfig — S3 target, credentials, retention policy, encryption settings. Fully database-agnostic; reused across all engine types. Retention policy is also mostly generic (count/age based) though some engines can support engine-specific pruning (e.g. WAL retention).
  • Backup — one-shot backup request referencing a DatabaseInstance and BackupConfig. Status includes phase, timing, S3 location(s), size, checksum, and an engineType copied from the referenced instance so downstream tooling doesn't need to cross-reference.
  • BackupSchedule — cron-based generator of Backup objects, identical in shape to Kubernetes' own CronJob-to-Job pattern. Generic across engines since scheduling has nothing to do with database type.
  • Restore — references either a Backup object or a raw S3 path, plus a target DatabaseInstance to restore into. The target instance's engineType determines which restore engine image is used; ideally it should match the source backup's engine type, and the controller should validate this and reject mismatched restores early.
  • BackupEngine (optional, phase 2) — registers a new engine type with the operator at runtime: engine name, container image reference, supported operations, and a JSON schema for validating engineConfig. This turns the operator into an open platform instead of a fixed list of hardcoded database types, and lets you or others add MongoDB/MySQL/ Milvus/Weaviate support (or anything else) without a new operator release.

Keep engineType as a first-class, validated (enum today, open string once BackupEngine exists) field everywhere it matters, since it is the dispatch key the operator uses to decide which engine image and which default connection-secret shape to expect.

8. Controller Responsibilities (Unchanged by Engine Type)

  • BackupScheduleController: computes next run time from cron expression, creates Backup objects, prunes old Backup objects per history limits. No engine-specific logic at all.
  • BackupController: validates references, resolves the engineType to a container image via the Engine Registry (or BackupEngine object), creates a Job with the standardized engine contract inputs, watches the Job, and maps the engine's JSON result + exit code back into Backup status. No engine-specific logic beyond the registry lookup.
  • RestoreController: same pattern as BackupController but for restores; additionally validates that the source backup's engineType matches the target instance's engineType before creating the Job.
  • BackupEngineController (phase 2, if the registry becomes a CRD): validates new engine registrations, checks the referenced image exists/ is reachable, and makes the engine available for use in engineConfig validation across the cluster.

Because none of the controllers contain Postgres-specific (or MongoDB- specific, etc.) logic, growing the list of supported databases becomes a matter of shipping a new engine image and registering it — the operator codebase itself barely changes.

9. Retry & Backoff Strategy

The operator uses categorized retry behavior based on the engine's exit code:

  • Exit codes 0, 1, 2 (success, internal error, auth failure): no retry. Mark as terminal and set conditions accordingly.
  • Exit codes 3, 4 (connectivity, storage failure): retry with exponential backoff:
    • Max 3 retries.
    • Delays: 30s, 2m, 5m between attempts.
    • Each retry creates a new Job with the same parameters.
    • After max retries, mark as terminal Failed with a condition indicating exhaustion.
  • Job timeout: each backup/restore Job has an activeDeadlineSeconds of 1 hour (configurable via BackupConfig). If the Job exceeds this, it is treated as a connectivity/storage failure and retried.
  • The backoff state is tracked in the CRD status (e.g. retryCount, nextRetryAt fields).

10. Security & Multi-Tenancy

  • Namespace-scoped CRDs; RBAC so that DatabaseInstance and BackupConfig (which reference credentials) can be governed separately from who is allowed to create Backup/Restore requests.
  • Never store raw credentials in CRD specs — always via Secret references, for both database auth and S3 credentials.
  • Support optional client-side encryption of backups before upload (engine-agnostic, can be applied as a pipe step regardless of which engine produced the data), so the operator's encryption story doesn't need to be reinvented per database type.
  • Least-privilege S3 IAM: scoped to the specific bucket/prefix per BackupConfig.
  • Audit trail: every Backup/Restore object's status/conditions and associated Job logs serve as the audit record; consider emitting Kubernetes Events for key lifecycle transitions.

11. Observability

  • Structured status/conditions on every CRD (Pending, Running, Succeeded, Failed) following the standard Kubernetes conditions pattern, consistent across all engine types.
  • Prometheus metrics from the operator: backup success/failure counts, duration, size, per engineType and per DatabaseInstance, so dashboards/alerts work uniformly regardless of how many database types are onboarded.
  • Job logs remain the detailed engine-level debugging surface; operator status stays high-level and structured.

12. Testing Strategy

  • Unit tests: envtest-based controller tests (part of kubebuilder scaffold) testing reconciliation logic with mock Jobs.
  • Engine integration tests: shell-level tests of each engine's shell scripts — spin up a real Postgres container + MinIO container via Docker Compose, run backup.sh/restore.sh, assert exit codes and result JSON. These run in CI as part of engine image builds.
  • E2E tests: kind cluster + operator deployed + Postgres + MinIO. Create DatabaseInstance / BackupConfig / Backup / Restore CRs and assert the full flow end-to-end, including restore validation (query the restored DB and compare row counts).
  • Test matrix: CI matrix runs tests against Postgres 14, 15, and 16 to catch pg_dump compatibility issues early.
  • No mock of the S3 API — MinIO provides a real S3-compatible backend.

13. Engine-Specific Considerations to Plan For (Not Build Yet)

Even though implementation is deferred, the abstraction should anticipate these differences so the contract doesn't need breaking changes later:

  • PostgreSQL: logical (pg_dump) vs physical/PITR (pgBackRest, WAL-G) backups; may need to expose "backup type" and optional point-in-time target on Restore.
  • MongoDB: replica-set aware backups (need to pick a secondary, ensure oplog consistency); mongodump/mongorestore for logical, or filesystem/volume snapshots for large clusters.
  • MySQL: mysqldump for logical, Percona XtraBackup for physical/hot backups; binlog position tracking if PITR is wanted later.
  • Milvus: backup is typically metadata + object-storage segment references (Milvus already stores vector data in object storage), so the "engine" may mostly orchestrate Milvus's own backup API/tool rather than move raw data itself.
  • Weaviate: has its own native backup module (backend-agnostic snapshot API, including S3 backends); the engine here may largely be a thin wrapper calling Weaviate's REST backup endpoints rather than doing data movement directly.

This confirms the value of the plugin/contract approach: several of these databases already have their own backup tooling with very different mechanics, so the operator should orchestrate rather than reimplement.

14. Phased Roadmap

  • Phase 1 (MVP, Postgres only): DatabaseInstance, BackupConfig, Backup, BackupSchedule, Restore CRDs; static Engine Registry with a single hardcoded postgres entry; pg_dump-based engine image; manual restore into a target Postgres instance. Prove out the Job-based contract end-to-end.
  • Phase 2 (Engine plugin maturity): formalize the Engine Registry as optionally a BackupEngine CRD; add PITR-capable Postgres engine (pgBackRest/WAL-G) as an alternate engine type alongside plain pg_dump, to validate that the contract supports multiple engines per database type, not just one.
  • Phase 3 (Second database family): add MongoDB engine, reusing all existing CRDs and controllers unchanged; this is the real test of the abstraction — if MongoDB support requires touching controller code beyond the registry, the contract needs revisiting.
  • Phase 4 (Relational parity): add MySQL engine following the same pattern.
  • Phase 5 (Vector databases): add Milvus and Weaviate engines, likely thin wrappers around their native backup APIs; validate that the generic engineConfig blob and result contract are flexible enough for non-traditional "backup" semantics (e.g. metadata + object-storage references rather than a single dump file).
  • Phase 6 (Platform hardening): retention/lifecycle automation across engines, cross-engine backup catalog/search, restore-time validation (e.g. checksum verification before restore), and optional multi-cluster/DR support (restoring into a database in a different cluster).

15. Key Success Criteria for the Abstraction

A good signal that the design is working: when MongoDB/MySQL/Milvus/ Weaviate support is added in later phases, the changes should be limited to (a) a new engine container image, (b) a new Engine Registry entry or BackupEngine object, and (c) at most minor additive fields in engineConfig validation — with zero changes to the CRD structure of Backup, Restore, BackupSchedule, or to the core reconcile loops of the existing controllers.