|
| 1 | +# Pynixd-Nixkube Improvements |
| 2 | + |
| 3 | +Improvements for the interaction between pynixd, nixkube and Kubernetes, discussed |
| 4 | +[2026-05-25](./../../.opencode/logs/2026-05-25.md). |
| 5 | + |
| 6 | +- ✅ Agreed + actionable |
| 7 | +- 👀 Agreed but lower priority |
| 8 | +- ❌ Not a priority (accepted tradeoff or non-issue) |
| 9 | + |
| 10 | +## Builder Manager Resiliency (✅) |
| 11 | + |
| 12 | +Fix race conditions and edge cases in the 726-line autoscaling builder pool: |
| 13 | + |
| 14 | +- **Creation race**: `_ensure_min_builders` and `_maybe_create_builder` both list |
| 15 | + jobs independently then create. Two concurrent calls can over-provision. Fix |
| 16 | + with a creation lock or make the watcher the sole creator. |
| 17 | +- **Orphan reaping**: `_reap_orphaned_builder_pods` runs once at startup only. |
| 18 | + Orphaned pods created later are never cleaned. Run it periodically. |
| 19 | +- **Probe timeout**: `_pending_probes` has no timeout — a hanging probe blocks |
| 20 | + reprobing of that node forever. Add a timeout + retry. |
| 21 | +- **Rate limiting**: `_ensure_min_builders` bypasses cooldown. Rapid Deployment |
| 22 | + scale-up events can burst Job creations. Respect cooldown in all creation |
| 23 | + paths. |
| 24 | +- **Lost registrations on restart**: In-flight `SSHSubprocessStore` connections |
| 25 | + are lost when the cache pod restarts. Persist store registrations to SQLite |
| 26 | + (pynixd already has `use_db`), then re-register on startup and let pynixd |
| 27 | + reschedule failed builds. (Note: losing individual in-flight builds when the |
| 28 | + cache itself dies is acceptable — this is about recovering the pool, not |
| 29 | + preserving individual builds.) |
| 30 | + |
| 31 | +## Observability (✅) |
| 32 | + |
| 33 | +- **Prometheus metrics**: Queue depth, builder utilization, cache hit ratio, CSI |
| 34 | + operation latency. pynixd already has `PrometheusMetrics` — wire it up. |
| 35 | +- **Health endpoint**: Add an HTTP `/healthz` that returns 200 only after |
| 36 | + `server` + `builder_manager` are initialized (stores registered, reconciler |
| 37 | + running). The current SSH TCP probe passes even before initialization. |
| 38 | +- **Structured builder lifecycle**: Add more structured fields (system, |
| 39 | + store_id, pod_name) and correlation IDs across builder lifecycle events. |
| 40 | + |
| 41 | +## GC Owned by Pynixd (✅) |
| 42 | + |
| 43 | +Replace the bash-based GC script with pynixd-managed GC: |
| 44 | + |
| 45 | +- **Current state**: `environments/cache/default.nix` has a dinit-managed GC |
| 46 | + service that runs `nix store delete` via bash + `shuf` for jitter. It has no |
| 47 | + coordination with pynixd — it could delete paths being served. |
| 48 | +- **Target state**: pynixd already has `StoreMonitor` and `use_db` for tracking |
| 49 | + path metadata. Have pynixd periodically delete paths with zero references or |
| 50 | + last-access beyond retention. This removes the bash GC entirely. |
| 51 | +- **Bonus**: The `IS_CACHE=true` env var (needed for store signing in the GC |
| 52 | + script) is never set in the Kubernetes manifests, so store signing never |
| 53 | + actually happens. Either wire it up or remove the dead code. |
| 54 | + |
| 55 | +## Resource-Aware Builder Scheduling (👀) |
| 56 | + |
| 57 | +Builders are created purely by `system` availability. No consideration of: |
| 58 | + |
| 59 | +- Current CPU/memory utilization of existing builders |
| 60 | +- Node resource pressure |
| 61 | +- Build queue size vs. builder count correlation |
| 62 | + |
| 63 | +Fix: expose builder resource metrics (via Prometheus), and let the builder |
| 64 | +manager consider utilization in scaling decisions. Or use the K8s scheduler's |
| 65 | +node selection (`_schedule_builder` picks nodes with available capacity). |
| 66 | + |
| 67 | +## Replace Environments (👀) |
| 68 | + |
| 69 | +The `environments/` directory (dinix-based service sets with bash GC scripts) |
| 70 | +should eventually be replaced by something cleaner. Store signing via the bash |
| 71 | +script is dead code (never actually runs). |
| 72 | + |
| 73 | +## Not A Priority |
| 74 | + |
| 75 | +These were discussed but accepted as tradeoffs or non-issues for now: |
| 76 | + |
| 77 | +- **Bootstrap ordering**: The cache's CSI `init-store` volume creates a circular |
| 78 | + dependency on fresh clusters. Dismissed because the cache can fetch from |
| 79 | + cachix instead. |
| 80 | +- **Secrets management**: The `init-secrets` kluctl hook regenerates SSH keys on |
| 81 | + every deploy (destructive). Current behavior is intentional — it only runs |
| 82 | + when something is missing, and destroying + recreating is acceptable. |
| 83 | +- **In-flight build recovery on cache crash**: Losing in-flight builds when the |
| 84 | + cache pod restarts is an acceptable tradeoff. |
| 85 | +- **Builder security**: Privileged builder pods are an accepted tradeoff — |
| 86 | + pynixd-nixkube should not be used to run untrusted builds. |
0 commit comments