Skip to content

REBUILD.md

Latest commit

 

History

History
287 lines (201 loc) · 9.92 KB

REBUILD.md

File metadata and controls

287 lines (201 loc) · 9.92 KB
Raw

Rebuilding kube-db-backup from Scratch

This guide is the single source of truth for recreating the entire project if the repository is lost. Follow the steps in order. The reasoning behind each step is included so the design intent is preserved, not just the commands.

Step 0: Define the ideology first

Before writing any code, decide the architecture:

  1. Separate lifecycle from engine mechanics.
    • The operator owns scheduling, state tracking, retries, and S3 orchestration.
    • The engine images own how to actually dump/restore each database type.
  2. Engine-agnostic CRDs.
    • CRDs describe connection (DatabaseInstance), storage (BackupConfig), and lifecycle (Backup, BackupSchedule, Restore), not Postgres.
  3. Container-per-engine plugin model.
    • Each engine is a separate image. The operator dispatches them as Jobs.
  4. Contract over code sharing.
    • The operator and engine communicate through env vars, mounted secret files, exit codes, and a JSON result on stdout. They do not import each other's code.

Write these four decisions down. Every file you create afterward should honor them.

Step 1: Create the Go module

mkdir kube-db-backup
cd kube-db-backup
go mod init github.com/kube-db-backup/kube-db-backup

Create the directory layout:

├── cmd/                        # operator entrypoint
├── api/v1alpha1/               # CRD Go types
├── internal/controller/        # reconcile loops
├── pkg/engine/                 # contract + registry
├── engines/postgres/           # first engine image
├── config/crd/                 # CRD YAMLs
├── config/rbac/                # RBAC YAMLs
├── config/manager/             # operator deployment
├── config/samples/             # example CRs
├── charts/kube-db-backup/      # Helm chart
├── test/e2e/                   # integration tests
├── docker-compose.yml          # local dev stack
├── Dockerfile                  # operator image
└── Makefile

Why this order: the module and directory structure are the foundation. Everything else hangs on these paths.

Step 2: Define the engine contract

Before writing CRDs, define how the operator and engine will talk. This is the most important contract in the system.

Create pkg/engine/contract.go:

  • Exit code conventions: 0 success, 1 internal, 2 auth, 3 connectivity (retry), 4 storage (retry).
  • Max retries and backoff schedule.
  • Result struct for JSON output.
  • ParseResult helper.

Create pkg/engine/registry.go:

  • Static map from engineType to EngineInfo.
  • Initial entry: postgres → image pg-engine:latest.
  • Allow override via PG_ENGINE_IMAGE env var.

Why this order: if you design the contract after CRDs, the CRDs will leak database-specific assumptions into generic fields. Define the contract first, then force the CRDs to stay generic.

Step 3: Define CRD Go types

Create api/v1alpha1/:

  1. groupversion_info.go — scheme group version.
  2. databaseinstance_types.go
  3. backupconfig_types.go
  4. backup_types.go
  5. backupschedule_types.go
  6. restore_types.go
  7. zz_generated.deepcopy.go — implement DeepCopy methods for every type.

Guidelines while writing:

  • Keep engineType as a first-class enum.
  • Use engineConfig as a map[string]string blob for engine-specific options.
  • Reference credentials via Secret refs; never put secrets in CRD specs.
  • Include status fields for phase, conditions, timing, size, checksum, retry count, and job name.

Why this order: the CRDs are the public API. Get them right before writing controllers that depend on them.

Step 4: Write the operator entrypoint

Create cmd/main.go:

  • Build the runtime scheme.
  • Register the core scheme and your custom scheme.
  • Create a controller-runtime manager.
  • Register the three reconcilers.
  • Add health/readiness probes.

Use the correct manager.Options for your controller-runtime version (e.g., Metrics: metricsserver.Options{BindAddress: ...} in v0.24+).

Why this order: controllers need a manager. You can't run the controllers until this file exists.

Step 5: Write the controllers

Create internal/controller/:

  1. backup_controller.go
  2. backupschedule_controller.go
  3. restore_controller.go
  4. helpers.go — shared condition helpers and exit-code extraction.

BackupController logic

  • Reconcile Backup objects.
  • On Pending or eligible retry: validate DatabaseInstance and BackupConfig refs.
  • Look up engine image in registry.
  • Build a Kubernetes Job with the engine contract env vars and secret volume mounts.
  • Set owner reference on the Job.
  • On Running: poll the Job, parse result, update status.
  • On failure: implement retry with exponential backoff.

BackupScheduleController logic

  • Reconcile BackupSchedule objects.
  • If suspended, skip.
  • Compute next scheduled time.
  • When due, create a Backup object.
  • Prune old Backup objects based on history limits.

RestoreController logic

  • Same pattern as BackupController, but for restores.
  • Validate engine type match between source backup and target instance.
  • Build restore Job with RESTORE_CLEAN and RESTORE_FORCE env vars.

Why this order: controllers are the core logic. They depend on CRDs and the engine contract, so they come after both.

Step 6: Write the first engine image

Create engines/postgres/:

  1. Dockerfile — start from postgres:16-alpine, add aws-cli and bash.
  2. entrypoint.sh — dispatch to backup or restore script based on OPERATION.
  3. backup.sh:
    • Read env vars and mounted secrets.
    • Run pg_dump in custom format (-Fc).
    • Compute size and SHA-256 checksum.
    • Upload dump.custom and backup-manifest.json to S3.
    • Write the standardized result JSON to stdout.
  4. restore.sh:
    • Read env vars and mounted secrets.
    • Download manifest and dump from S3.
    • Check target DB emptiness unless FORCE=true.
    • Run pg_restore --clean.
    • Run ANALYZE.
    • Write result JSON.

Test the engine locally with docker-compose before connecting it to the operator.

Why this order: the engine is a standalone component. Build and validate it independently so you know the contract works before the operator orchestrates it.

Step 7: Create raw Kubernetes manifests

Create config/:

  1. crd/ — one YAML per CRD. Use apiextensions.k8s.io/v1.
  2. rbac/rbac.yaml — ServiceAccount, ClusterRole, ClusterRoleBinding.
  3. manager/namespace.yaml — operator namespace.
  4. manager/deployment.yaml — operator Deployment.
  5. samples/all-in-one.yaml — Secrets, DatabaseInstance, BackupConfig, Backup, BackupSchedule, Restore.

Make sure the operator Deployment sets PG_ENGINE_IMAGE so the registry override works.

Why this order: these manifests prove the system works without Helm. They are also the source material for the Helm chart.

Step 8: Add local development tooling

Create:

  • docker-compose.yml — Postgres 16 + MinIO + bucket creation.
  • Makefile — build, test, docker-build, deploy targets.
  • .gitignore.

Verify locally:

docker compose up -d
docker build -t pg-engine:latest engines/postgres/
go run cmd/main.go
kubectl apply -f config/samples/all-in-one.yaml
kubectl get backups -w

Why this order: a working local loop is the fastest way to find integration bugs between the operator, engine, and S3.

Step 9: Build the Helm chart

Create charts/kube-db-backup/:

  1. Chart.yaml.
  2. values.yaml — user-facing config for operator image, engine image, namespace, DB instances, S3 configs, schedules, secrets.
  3. crds/ — copy the 5 CRD YAMLs here.
  4. templates/:
    • namespace.yaml
    • rbac.yaml
    • deployment.yaml (pass PG_ENGINE_IMAGE as env var)
    • secrets.yaml
    • databaseinstances.yaml
    • backupconfigs.yaml
    • backupschedules.yaml

Make sure helm template and helm lint pass.

Why this order: Helm is packaging. Package only after the raw manifests are correct and tested.

Step 10: Write tests

Add tests in order of value:

  1. pkg/engine/contract_test.go — test result parsing, retry logic, registry lookup.
  2. Engine integration tests — run engines/postgres/backup.sh and restore.sh against docker-compose Postgres + MinIO.
  3. Controller envtest tests — verify reconcile loops create Jobs correctly.
  4. E2E tests in test/e2e/ — kind cluster, full backup/restore flow.

Why this order: start with cheap unit tests, then integration, then expensive e2e. The engine contract tests catch most contract mismatches before you spin up a cluster.

Step 11: Document everything

Write:

  • README.md — quick start.
  • ARCHITECTURE.md — component overview and design rationale.
  • REBUILD.md — this file.
  • PLAN.md — roadmap and abstraction goals.

Why this order: documentation captures intent. Write it while the decisions are fresh.

Common mistakes to avoid

  1. Putting credentials in CRD specs. Always use Secret references.
  2. Leaking engine specifics into CRDs. Keep engineConfig a free-form map.
  3. Hardcoding engine images without override. Use env vars so Helm can configure them.
  4. Forgetting ImagePullPolicy: IfNotPresent. Without it, kind clusters try to pull from a registry.
  5. No backup manifest in S3. The manifest lets you restore even if the Backup CRD is gone.
  6. Designing for all engines at once. Build Postgres end-to-end first, then abstract. Otherwise you design for hypotheticals.

Validation checklist

Before declaring the system done:

  • go build ./... passes.
  • go test ./... passes.
  • go vet ./... passes.
  • helm lint charts/kube-db-backup passes.
  • helm template renders valid YAML.
  • One-shot backup creates a Job, succeeds, and uploads to S3.
  • Restore from that backup works into a second database.
  • Deleting the operator does not delete existing S3 backups.