Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
224 lines (155 loc) · 9.02 KB

ARCHITECTURE.md

File metadata and controls

224 lines (155 loc) · 9.02 KB
Raw

kube-db-backup Architecture

What problem this solves

Running production databases on Kubernetes means someone has to answer:

  • How do I back up this database automatically?
  • Where do the backups go?
  • How do I restore into another cluster or instance?
  • How do I do this for multiple database types without writing a different tool for each one?

kube-db-backup solves this by introducing a Kubernetes-native operator with a pluggable engine model. The operator handles scheduling, lifecycle, retries, S3 storage, and observability. Small, swappable engine container images know only how to dump/restore one specific database type. Today it supports PostgreSQL; tomorrow it can support MongoDB, MySQL, Milvus, Weaviate, or anything else by adding a new engine image — without changing the operator.

The core design principle is separation of concerns: the generic lifecycle belongs to the operator; the database-specific mechanics belong to the engine.

High-level data flow

  1. A user creates DatabaseInstance, BackupConfig, and either a Backup (one-shot) or BackupSchedule (cron) resource.
  2. The operator validates the references, looks up the engine type in the Engine Registry, and creates a Kubernetes Job.
  3. The Job runs an engine container image (e.g., pg-engine) with a standardized set of environment variables and mounted secrets.
  4. The engine connects to the database, runs pg_dump/pg_restore, uploads/downloads from S3, and writes a standardized JSON result.
  5. The operator reads the Job status and result, then updates the Backup/Restore CRD status.

Components

1. CRDs (Custom Resource Definitions)

All CRDs live in api/v1alpha1/ and are database-agnostic. They describe the backup lifecycle, not Postgres specifically.

DatabaseInstance

  • Purpose: describes how to connect to a database.
  • Key fields:
    • engineType (e.g., postgres) — tells the operator which engine image to use.
    • host, port, databaseName — connection details.
    • authSecretRef — reference to a Secret containing username and password.
    • engineConfig — free-form key/value map for engine-specific options (e.g., dumpFormat: custom, sslMode: disable).
  • Why it matters: the same CRD works for Postgres, MongoDB, or any future engine. Adding a new engine does not require changing this CRD.

BackupConfig

  • Purpose: describes where backups go in S3 and how long to keep them.
  • Key fields:
    • s3Endpoint, s3Bucket, s3Prefix, s3Region, s3ForcePathStyle — S3 destination.
    • s3SecretRef — reference to a Secret containing access-key and secret-key.
    • retentionPolicy — maxBackups and maxAgeDays (planned lifecycle automation).
    • activeDeadlineSeconds — maximum runtime for a backup Job.
  • Why it matters: storage policy is completely decoupled from the database being backed up.

Backup

  • Purpose: one-shot backup request.
  • Key fields:
    • databaseInstanceRef — what to back up.
    • backupConfigRef — where to put it.
  • Status fields: phase (Pending/Running/Succeeded/Failed), sizeBytes, checksumSha256, s3Objects, tablesCount, retryCount, jobName.
  • Why it matters: it is the unit of work. It maps 1:1 to a Kubernetes Job.

BackupSchedule

  • Purpose: cron-based generator of Backup objects.
  • Key fields:
    • schedule — cron expression.
    • databaseInstanceRef and backupConfigRef.
    • suspend — pause without deleting.
    • successfulJobsHistoryLimit / failedJobsHistoryLimit — how many old Backup objects to keep.
  • Why it matters: scheduling is engine-agnostic. It behaves like Kubernetes CronJob → Job, but for backups.

Restore

  • Purpose: restore a backup into a target database instance.
  • Key fields:
    • backupRef — restore from an existing Backup object.
    • s3Path — alternative: restore from a raw S3 key.
    • targetInstanceRef — where to restore.
    • restoreClean — drop existing objects before restoring.
    • force — allow restoring into a non-empty database.
  • Why it matters: restores are explicit, auditable, and validated (engine type must match between source and target).

2. Engine Registry

Located in pkg/engine/registry.go.

  • A static map from engineType to EngineInfo.
  • Each entry defines the container image and default port for that engine.
  • The registry can be overridden at runtime via environment variables (e.g., PG_ENGINE_IMAGE).
  • Why it matters: this is the only place the operator knows about Postgres. To add MongoDB, you add one entry and ship a new image — no controller changes.

3. Engine Contract

Located in pkg/engine/contract.go.

The contract defines exactly how the operator and an engine image communicate:

  • Input: standardized environment variables (OPERATION, DB_HOST, DB_PORT, DB_NAME, S3_ENDPOINT, S3_BUCKET, etc.).
  • Secrets mounted as files:
    • /etc/db-secret/password
    • /etc/s3-secret/access-key
    • /etc/s3-secret/secret-key
  • Exit codes:
    • 0 success
    • 1 engine internal error
    • 2 auth failure
    • 3 connectivity failure — retried
    • 4 storage failure — retried
  • Output: a JSON result wrapped in ___RESULT_START___ / ___RESULT_END___ markers, written to stdout.

The contract also defines retry behavior: max 3 retries with backoff delays of 30s, 2m, 5m.

4. Controllers

Located in internal/controller/.

BackupReconciler

  • Watches Backup objects.
  • Validates DatabaseInstance and BackupConfig references.
  • Looks up the engine image from the registry.
  • Creates a Kubernetes Job with the engine contract inputs.
  • Polls the Job status and maps the engine result back to Backup status.
  • Implements retry logic for connectivity/storage failures.

BackupScheduleReconciler

  • Watches BackupSchedule objects.
  • Computes the next run time from the cron expression.
  • Creates Backup objects when the schedule fires.
  • Prunes old Backup objects according to history limits.

RestoreReconciler

  • Watches Restore objects.
  • Validates the target DatabaseInstance and BackupConfig.
  • Validates that the source backup's engine type matches the target instance's engine type.
  • Creates a Job to run the restore engine.
  • Polls the Job and updates Restore status.

5. Engine Images

Located in engines/postgres/.

An engine image is a standalone container with native database tooling.

  • Dockerfile — based on postgres:16-alpine, includes aws-cli.
  • entrypoint.sh — dispatches to backup.sh or restore.sh based on OPERATION.
  • backup.sh — runs pg_dump -Fc, writes a backup-manifest.json, and uploads both files to S3.
  • restore.sh — downloads the manifest and dump from S3, validates target DB emptiness unless FORCE=true, runs pg_restore --clean, and runs ANALYZE.

Why this design is powerful: each engine is a black box. It can use completely different tooling (pg_dump, mongodump, pgBackRest, Milvus's backup API) without the operator needing to understand any of it.

6. S3 Object Layout

Backups are organized as:

s3://{bucket}/{prefix}/backups/{engineType}/{databaseName}/{backupId}/
├── dump.{format}
└── backup-manifest.json

The manifest is self-describing and includes checksum, timestamp, size, and S3 object keys. This means even if the Kubernetes Backup CRD is deleted, the S3 objects remain discoverable and restorable.

7. Helm Chart

Located in charts/kube-db-backup/.

The chart packages everything:

  • crds/ — all 5 CRDs (Helm installs these first).
  • templates/ — operator Deployment, RBAC, Secrets, and application CRs (DatabaseInstance, BackupConfig, BackupSchedule).
  • values.yaml — single file where users configure DB connections, S3 credentials, schedules, and engine/operator images.

The operator Deployment receives PG_ENGINE_IMAGE as an environment variable so the chart controls which engine image the operator dispatches.

8. Local Development Stack

  • docker-compose.yml — spins up Postgres 16 and MinIO for local testing.
  • Makefile — build, test, docker-build targets.

What each component abstracts away

Concern Handled by
Scheduling BackupSchedule + BackupScheduleReconciler
Job lifecycle / retries Controllers
Storage target / retention BackupConfig
Database connection DatabaseInstance
Actual dump/restore Engine image
S3 upload/download Engine image
Result contract pkg/engine/contract.go
Multi-engine dispatch Engine Registry
Packaging / install Helm chart

Extensibility

To add a new database engine:

  1. Build a new engine image that follows the contract.
  2. Add its entry to the Engine Registry (or later, a BackupEngine CRD).
  3. Done. No CRD changes, no controller changes.

This is the project's key success criterion: the abstraction is validated when adding a new engine requires only a new image and a registry entry.