Migrate the outer DOJO appliance to native NixOS services

[zardus]

Summary

• replace the Compose-managed outer stack with a NixOS image and native systemd services while preserving the inner Docker learner runtime
• migrate persistent configuration and storage safely, including database state, generated secrets, SSH host keys, TLS certificates, workspace images, and learner metadata
• harden multinode HomeFS ownership, legacy-owner reconciliation, transfer streaming, service dependencies, and workspace routing
• update deployment and development tooling, and add CI coverage for native upgrades, multinode HomeFS guarantees, and Splunk forwarding

Validation

• focused single-node suite: 10 passed
• guarded multinode suite: 31 passed, 654 deselected
• certificate unit tests: 2 passed
• Nix flake checks passed
• final dojo image built successfully
• runc and Kata learner workspace smoke checks passed
• zero failed systemd units and zero leaked learner containers after the full multinode run

[codecov]

Codecov Report

❌ Patch coverage is 75.00000% with 4 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
dojo_plugin/worker/__main__.py	0.00%	3 Missing ⚠️
dojo_plugin/pages/workspace.py	85.71%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[zardus]

/data upgrade risks

Fresh-install CI is green, but the persisted Compose-to-NixOS upgrade path has not actually run. The new transition workflow only has workflow_dispatch and schedule triggers; because it is not on the default branch yet, there is no transition run for this PR. The current required jobs all start with fresh data roots.

I would treat the first native boot as a one-way migration unless we restore a snapshot:

• dojo-config rewrites /data/config.env, changes the legacy local DB_HOST=db value, and drops comments or manually-added variables outside its allowlist.
• The NixOS PostgreSQL module replaces /data/postgres/postgresql.conf with a Nix-store symlink. Custom PostgreSQL configuration is lost, and that symlink will be invalid if we simply launch the legacy image again.
• The inner Docker data root is opened by Docker 29.7.2 after previously being managed by 27.5.1. Downgrading the same data root is not a safe rollback plan.
• The workspace Nix store may also be upgraded or populated with newer paths. Garbage collection is off by default, which helps rollback but increases disk use.

Other migration risks:

• HomeFS transition coverage is incomplete. Existing ownership claims are migrated only when the recorded legacy address is in 172.16.0.0/12; custom Docker address pools can leave active homes unclaimable. Topology validation now only accepts node IDs 1 through 15. test/test-native-upgrade.sh does not create and reopen a real learner home.
• First boot recursively changes ownership under PostgreSQL, Redis, CTFd, coverage, IPython, and dojo directories. This happens twice on the initial migration when legacy containers are retired. A large data tree can make startup very slow or exceed Docker's ten-minute startup timeout.
• PostgreSQL stays on major version 17, but moves from 17.5 to 17.11 and changes its libc/runtime environment. We should check for collation-version warnings and reindex before refreshing collation versions. Any custom extensions or tablespaces need separate validation.
• Legacy container cleanup depends on exact Compose labels and the exact /opt/pwn.college/docker-compose.yml path. Older or customized infrastructure containers that do not match can restart under the new daemon and contend with native services for the same persistent files.
• The transition test currently checks a database row, generated secrets, SSH host keys, and inner-Docker image/container metadata. It does not verify Redis state, uploads/dojos, TLS state, backup keys, HomeFS data, Splunk data, multinode migration, or that an old learner container can actually start after the upgrade.
• The outer /data bind must be recreated with :shared, must permit suid and execution, and must have enough free space and inodes for the new workspace closure. BACKUP_AES_KEY_FILE must point at a durable file under /data; a missing configured key now correctly stops startup.

Before using a production data root, I think the minimum bar is:

1. Run the transition workflow on this PR, not only after merge.
2. Extend it to cover HomeFS, Redis, uploads/dojos, and starting a migrated learner container.
3. Rehearse against an offline copy of representative production data, including any custom configuration and multinode/Splunk setup.
4. Take an offline, restorable snapshot of the complete data root immediately before migration. Do not rely on launching the old image as rollback.
5. Add or follow a preflight that validates mount propagation/options, free space and inodes, backup-key location, workspace topology, and the labels of every legacy Compose container.

I would not deploy this against the only production copy of /data until those checks pass.