Feat: P2 shared sandbox pool with N:M lease-based routing

[pdettori]

P2 — Shared sandbox pool + harness→sandbox routing

Part of the two-tier FS-free harness epic (#49). Depends on P1 (#45, merged). Implements #46.

Scales Archetype A from one sandbox to a static pool of N single-instance Sandbox CRs: short-lived, fresh-context leaf harnesses load-balance across the pool via Redis leases, each leaf's repo is lazily converged to a pinned git ref, and its work is isolated in a per-leaf git worktree — the harness still touching no filesystem.

Spec: docs/specs/2026-07-02-p2-shared-sandbox-pool-design.md · Plan: docs/superpowers/plans/2026-07-02-p2-shared-sandbox-pool.md

What's in it

• Pool discovery (@sh/k8s-sandbox/pool.ts) — list Running pods by a shared label.
• Redis lease store (harness/sandbox-lease.ts) — per-pod expiry-scored sorted set; atomic Lua acquire (sweep-expired + ZCARD-under-cap); crash reclaim is implicit via TTL.
• Least-loaded selector (harness/select-sandbox.ts) — picks least-loaded pod under a soft cap, throws SandboxPoolSaturatedError when full, falls back to single-pod when no pool selector is set.
• Ref-pinned converge + worktree (harness/converge.ts) — shared /workspace/repo object store (flock-serialized git fetch) + per-leaf git worktree; shell inputs single-quote-escaped.
• Leaf wiring (harness/run-leaf.ts) — LeafEnvelope gains repoUrl/ref; select → converge → heartbeat → run → finally cleanup+release, after the verdict fast-path. FS-free contract preserved (all FS via exec).
• Deploy — sandbox-pool.yaml (3 CRs, own RWO PVC each, common pool label, git in the image), KAGENTI_SANDBOX_POOL_SELECTOR, OCP overlay patched for all pool members.

Design decisions (from brainstorm)

Per-sandbox RWO copy (no RWX on the deployable path; RWX documented as the alternative) · harness-side pick + Redis leases (no new deployable component) · ref-pinned lazy converge (eager pre-warm deferred) · static-N pool (config knob). Soft cap. Kata isolation, the ~20:1 ratio, and autoscaling are P3 (#48).

Testing

• Unit: pool arg-builders, lease pure logic, selector (least-loaded / saturation / fallback / ttl-propagation), converge script builders + injection escaping. Full suites green (harness 130 passed/3 skipped; k8s-sandbox 60 passed/6 skipped).
• Gated live-Redis (POOL_LIVE_SMOKE=1): ACQUIRE_LUA cap enforcement + expiry reclaim against real Redis.
• Live Kind smoke: 6 concurrent leaves distributed exactly 2/2/2 across the 3-pod pool (cap=2), never exceeding cap, clean drain, all 6 correct verdicts — discovery + least-loaded + soft-cap + N:M concurrency + release verified end-to-end. Killing a pool pod mid-run did not wedge the pool; its leases freed (no slot leak) and it self-healed.

Follow-ups (non-blocking)

• Point setup-kind.sh at the pool manifest (currently the single sandbox.yaml).
• Sync-path saturation: bounded wait + 503 Retry-After per spec §4.3 (async requeue path already correct; sync fails fast, fail-safe).
• Direct unit test for the empty-pod-list branch.

Assisted-By: Claude Code