|
| 1 | +# P2 — Shared Sandbox Pool + Harness→Sandbox Routing — Design |
| 2 | + |
| 3 | +Version: 1.0 — July 2, 2026 |
| 4 | +Status: Design (approved for implementation planning) |
| 5 | +Scope: **Phase 2 (P2)** of the [two-tier FS-free harness epic](https://github.com/kagenti/serverless-harness/issues/49) |
| 6 | +([P2 issue #46](https://github.com/kagenti/serverless-harness/issues/46)). Depends on **P1** (FS-free harness, |
| 7 | +[#45](https://github.com/kagenti/serverless-harness/issues/45)), which is merged. Scales Archetype A from one |
| 8 | +sandbox to **many leaf harnesses sharing a smaller pool of sandboxes** (N:M), with the repo distributed as |
| 9 | +per-sandbox copies and no RWX on the deployable path. |
| 10 | + |
| 11 | +--- |
| 12 | + |
| 13 | +## 1. Goal & motivation |
| 14 | + |
| 15 | +After P1, the sandbox tier is a **single** agent-sandbox `Sandbox` CR (`sandbox-0`) with one RWO PVC, serving one |
| 16 | +harness consumer. The harness resolves its exec target by reading `KAGENTI_SANDBOX_NAME` → the CR's |
| 17 | +`.status.selector` → **the first Running pod** ([`packages/k8s-sandbox/src/resolve-pod.ts`](../../packages/k8s-sandbox/src/resolve-pod.ts)). |
| 18 | +That is 1:1. |
| 19 | + |
| 20 | +P2 makes it **N:M**: many short-lived, fresh-context leaf harnesses load-balance across a **static pool of N |
| 21 | +sandbox pods**, so the dense, cheap harness tier is decoupled from the smaller, durable sandbox tier. The exact |
| 22 | +sharing ratio (~20:1) is **not** fixed here — it is an empirical P3 experiment. P2 delivers the *mechanism*: pod |
| 23 | +discovery, lease-based assignment, per-sandbox repo distribution, and per-leaf worktree isolation. |
| 24 | + |
| 25 | +`kubectl exec` targets a **concrete pod name**, so "pick the first Running pod" must become deliberate, |
| 26 | +load-aware pod selection. That is the heart of P2. |
| 27 | + |
| 28 | +## 2. Decisions locked (brainstorm 2026-07-02) |
| 29 | + |
| 30 | +These were settled during brainstorming and are not relitigated here: |
| 31 | + |
| 32 | +| # | Decision | Rationale | |
| 33 | +|---|----------|-----------| |
| 34 | +| D1 | **Storage topology: per-sandbox RWO copy.** Each sandbox pod holds its own repo copy on its own RWO PVC. **No RWX** on the deployable path. RWX (fleet-wide single repo) is documented as the alternative only. | Runs on the EBS-only OCP 4.20 cluster today; matches "the harness mounts nothing"; RWX is heavier infra (EFS/CSI), slower networked FS, cross-pod contention. | |
| 35 | +| D2 | **Routing: harness-side pick + Redis leases.** Selection logic stays in the harness/`@sh/k8s-sandbox` layer; Redis holds per-pod lease counters for least-loaded + capacity backpressure. **No new deployable component.** | Reuses the existing hard Redis dependency; crash-safe lease reclaim via TTL mirrors the existing leaf-resume model. | |
| 36 | +| D3 | **Repo seeding: ref-pinned lazy converge.** The envelope carries a git ref; on leaf start the leased pod's repo is fetched/converged to that ref, then a worktree is created. Idempotent (pod already at ref = no-op). **Eager pre-warm deferred to P3.** | Guarantees batch-wide commit consistency regardless of which pod a leaf lands on; amortizes clone cost across the sharing ratio; survives pod churn for free. | |
| 37 | +| D4 | **Pool scaling: static N, config knob.** N `Sandbox` CRs declared in kustomize; N and per-pod cap are values tuned empirically in P3. **Autoscaling is future work only.** | Deterministic, no new controller; saturation backpressures through the existing async Redis/KEDA queue and a bounded sync wait. | |
| 38 | +| D5 | **Soft capacity cap.** ~20 is an empirical figure, not a safety bound; rare concurrent overshoot is acceptable. | Avoids a hard-CAS hot path; the true safety boundary is Kata at the pod level (P3). | |
| 39 | + |
| 40 | +**Threat model (locked — from epic #49, not relitigated):** there is **no agent in the sandbox** — it is a |
| 41 | +passive `kubectl exec` target. The threat is a **compromised/injected harness** and **kernel exploits**. Blast |
| 42 | +radius is contained by giving the harness zero FS/exec surface (P1) and Kata-isolating only the sandboxes (P3). |
| 43 | +Intra-pod cross-leaf isolation is therefore an explicit **non-goal** for the no-Kata P2 PoC (see §9). |
| 44 | + |
| 45 | +## 3. Pool topology — N single-instance `Sandbox` CRs |
| 46 | + |
| 47 | +agent-sandbox `v1beta1` **removed `spec.replicas`** (the `Sandbox` is a single-instance primitive; a warm pool is |
| 48 | +an open upstream feature request, [kubernetes-sigs/agent-sandbox#34](https://github.com/kubernetes-sigs/agent-sandbox/issues/34)). |
| 49 | +So the P2 pool primitive is **N distinct `Sandbox` CRs** `sandbox-0 … sandbox-{N-1}`: |
| 50 | + |
| 51 | +- **Own RWO PVC each** via the existing `volumeClaimTemplates` — this *is* the per-sandbox repo copy (D1). No |
| 52 | + shared volume, no RWX. |
| 53 | +- **Distinct names** deliberately sidestep the agent-sandbox v0.5.0 **bare-pod name-collision / adopt-error** |
| 54 | + gotcha observed in P0′. |
| 55 | +- **Common pod label** stamped via each CR's `podTemplate.metadata.labels` (e.g. |
| 56 | + `sh.kagenti.io/sandbox-pool: default`). Discovery selects on **this label across all N CRs' pods**, decoupling |
| 57 | + pool membership from any single CR's `.status.selector`. |
| 58 | + |
| 59 | +``` |
| 60 | + label: sh.kagenti.io/sandbox-pool=default |
| 61 | + ┌───────────────┬───────────────┬─────── … ────────┐ |
| 62 | + sandbox-0-0 sandbox-1-0 sandbox-2-0 sandbox-{N-1}-0 |
| 63 | + PVC ws-0 PVC ws-1 PVC ws-2 PVC ws-{N-1} (all RWO) |
| 64 | +``` |
| 65 | + |
| 66 | +## 4. Routing & lease protocol |
| 67 | + |
| 68 | +**Discovery is authoritative from Kubernetes; load is authoritative from Redis.** At leaf start the harness lists |
| 69 | +**Running** pods matching `KAGENTI_SANDBOX_POOL_SELECTOR` (this naturally handles pods added, removed, or crashed), |
| 70 | +then consults Redis for per-pod load and leases the least-loaded pod under the cap. |
| 71 | + |
| 72 | +### 4.1 Lease as an expiry-scored set (crash-reclaim is implicit) |
| 73 | + |
| 74 | +- Per-pod sorted set **`sh:sandbox:<pod>:leases`**, member = `runId`, score = **expiry timestamp** (ms). |
| 75 | +- **Active load** = count of members with `score > now`. |
| 76 | +- **Crash reclaim is implicit:** a crashed leaf's lease member simply ages past its expiry. Every acquire lazily |
| 77 | + sweeps expired members (`ZREMRANGEBYSCORE sh:sandbox:<pod>:leases -inf <now>`) — **no separate reaper process**. |
| 78 | + This mirrors the existing crash-safe leaf-resume model. |
| 79 | +- A running leaf **heartbeats** (re-`ZADD`s its member with `expiry = now + TTL`) on an interval well below the |
| 80 | + TTL, so long-running leaves are never reclaimed mid-run. TTL and heartbeat interval are config values. |
| 81 | + |
| 82 | +### 4.2 Acquire / release (atomic via Lua) |
| 83 | + |
| 84 | +`ACQUIRE(pod, cap, runId, expiry)` — evaluated as a single Redis Lua script: |
| 85 | + |
| 86 | +1. `ZREMRANGEBYSCORE` the pod's set to drop expired members. |
| 87 | +2. If `ZCARD < cap`: `ZADD` `{runId → expiry}`, return `OK`. |
| 88 | +3. Else return `FULL`. |
| 89 | + |
| 90 | +The harness orders candidate pods by last-known load (cheap `ZCARD` reads, best-effort) and calls `ACQUIRE` |
| 91 | +against them in order until one returns `OK`. `RELEASE(pod, runId)` = `ZREM`. Concurrency between two harnesses |
| 92 | +racing the same pod can momentarily overshoot the soft cap by a small margin — accepted per D5. |
| 93 | + |
| 94 | +### 4.3 Saturation (all pods FULL) |
| 95 | + |
| 96 | +Bounded wait with backoff, up to a timeout: |
| 97 | + |
| 98 | +- **Async leaves** overflow naturally: they already sit behind the Redis Streams queue drained by the KEDA |
| 99 | + ScaledJob, so work that cannot acquire stays queued and drains as leases free — no special handling. |
| 100 | +- **Sync leaves** return **`503` with `Retry-After`** on timeout. |
| 101 | + |
| 102 | +## 5. Repo lifecycle & workspace layout |
| 103 | + |
| 104 | +The unit shared across leaves on a pod is the **git object store**, *not* a working tree — so two leaves at |
| 105 | +different commits never contend over a checkout. |
| 106 | + |
| 107 | +- **`/workspace/repo`** — the canonical clone; its object store accumulates every ref ever fetched on that pod. |
| 108 | + **Converge = `git fetch origin <ref>`** (additive, idempotent; a pod already holding the ref is a no-op). |
| 109 | + Concurrent converges on one pod are serialized with a per-pod advisory lock |
| 110 | + (`flock /workspace/repo/.sh-fetch.lock`). |
| 111 | +- **`/workspace/leaves/<runId>`** — the leaf's own `git worktree add <path> <commit>`, pinned to the exact commit |
| 112 | + resolved from the envelope ref. **This path is the leaf's `workspaceRef`.** |
| 113 | +- **Cleanup:** `git worktree remove` on leaf end; orphaned worktrees from crashed leaves are reclaimed by |
| 114 | + `git worktree prune` plus a dir-age sweep performed opportunistically on acquire. |
| 115 | + |
| 116 | +This yields **batch-wide consistency** (each leaf pins its own commit regardless of pod) *and* lets a single pod |
| 117 | +serve many refs concurrently. |
| 118 | + |
| 119 | +## 6. FS-free contract & leaf-flow interaction |
| 120 | + |
| 121 | +The credentialed harness continues to **touch no filesystem** (P1 contract preserved). Every FS mutation — |
| 122 | +converge, `worktree add`, `worktree remove` — is issued as a **sandbox `exec`**; the bytes land in the sandbox, |
| 123 | +never in the harness process. |
| 124 | + |
| 125 | +Sequence added to [`harness/src/run-leaf.ts`](../../harness/src/run-leaf.ts) around the existing model loop: |
| 126 | + |
| 127 | +``` |
| 128 | +runLeaf(envelope): |
| 129 | + pod = poolSelect() # §4: list-by-label + ACQUIRE lease |
| 130 | + exec(pod, convergeAndWorktree(repoUrl, ref, runId)) # §5: fetch + worktree add |
| 131 | + workspaceRef = /workspace/leaves/<runId> |
| 132 | + … existing model loop, all tool exec routed to `pod` … # unchanged (P1) |
| 133 | + exec(pod, worktreeRemove(runId)) # §5 cleanup |
| 134 | + RELEASE(pod, runId) # §4.2 |
| 135 | + … existing verdict write (inline sync / Redis async) … # unchanged (P1) |
| 136 | +``` |
| 137 | + |
| 138 | +Lease heartbeat (§4.1) runs for the duration of the model loop. On crash before `RELEASE`, the lease TTL-expires |
| 139 | +and the existing Redis-backed resume re-runs the leaf, which **re-acquires** (likely a different pod) and |
| 140 | +**re-converges** — idempotent by construction. |
| 141 | + |
| 142 | +## 7. Envelope changes (`LeafEnvelope`) |
| 143 | + |
| 144 | +Additive to the P1 wire contract: |
| 145 | + |
| 146 | +| Field | Type | Meaning | |
| 147 | +|-------|------|---------| |
| 148 | +| `repoUrl` | string | Git remote to converge from. | |
| 149 | +| `ref` | string | Commit SHA (preferred) or branch/tag; the pod converges to it and the worktree pins the resolved commit. | |
| 150 | + |
| 151 | +`workspaceRef` becomes **derived** (`/workspace/leaves/<runId>`) rather than a caller-supplied absolute pod path. |
| 152 | +Callers we own (leaf-orchestrator, dispatch scripts) are updated in the same cut to send `repoUrl`/`ref` and to |
| 153 | +stop supplying `workspaceRef`. |
| 154 | + |
| 155 | +## 8. Deploy changes (`deploy/knative/`) |
| 156 | + |
| 157 | +- **`sandbox.yaml`** → generate **N** `Sandbox` CRs (`sandbox-0..{N-1}`) with the common pool label; N is a |
| 158 | + kustomize config value. Each retains its own `volumeClaimTemplates` (RWO). |
| 159 | +- **`service.yaml`** — replace `KAGENTI_SANDBOX_NAME=sandbox-0` with **`KAGENTI_SANDBOX_POOL_SELECTOR`** |
| 160 | + (label selector). Retain `KAGENTI_SANDBOX_POD` as a single-pod **test/override** that bypasses pool selection. |
| 161 | +- **RBAC** — unchanged surface (`get/list pods`, `create pods/exec`, `get/list Sandboxes`); it already spans all |
| 162 | + pods in the namespace. |
| 163 | +- **OCP overlay** — GHCR image + UID 65532 as in P0′, applied across all N CRs. Operational note: the |
| 164 | + agent-sandbox **StatefulSet PVC is not GC'd on CR delete** (P0′) — pool-shrink requires a manual PVC sweep. |
| 165 | + |
| 166 | +## 9. Isolation (explicit non-goal for P2) |
| 167 | + |
| 168 | +Leaves sharing a pod run in **separate worktree directories under the same UID 65532** — soft, directory-level |
| 169 | +isolation only. A compromised leaf could read sibling worktrees and the shared object store. This is **acceptable |
| 170 | +for P2** per the epic threat model: the sandbox is a passive exec target and the real blast-radius control is |
| 171 | +**Kata at the pod level**, delivered in **P3** (possibly with per-leaf UID/namespace). Leaves sharing a pod are |
| 172 | +one trust domain in P2. Documented here as the known limitation P3 resolves. |
| 173 | + |
| 174 | +## 10. Failure modes |
| 175 | + |
| 176 | +| Event | Behavior | |
| 177 | +|-------|----------| |
| 178 | +| Pod crashes mid-leaf | Lease TTL-expires → reclaimed on next acquire; worktree orphaned then age-pruned; leaf resumes (existing Redis resume) and re-acquires + re-converges (likely a different pod). | |
| 179 | +| Pod removed from pool | K8s list stops returning it → never picked. In-flight leaf on it fails through the existing verdict-error path and resumes elsewhere. | |
| 180 | +| All pods FULL | §4.3 — async stays queued; sync gets `503 Retry-After`. | |
| 181 | +| `git fetch`/converge fails | Leaf errors out via the existing verdict-error path; lease released; resume retries. | |
| 182 | +| Two harnesses race one pod | Soft-cap overshoot by a small margin (D5) — accepted. | |
| 183 | + |
| 184 | +## 11. Testing |
| 185 | + |
| 186 | +- **Unit:** lease Lua (`ACQUIRE` OK/FULL, expiry sweep, heartbeat re-add, `RELEASE`); candidate-pod ordering by |
| 187 | + load; converge idempotency (already-at-ref = no-op); worktree path derivation. |
| 188 | +- **Integration (live Kind, N=2–3):** concurrent leaves land on distinct pods; a mixed-ref batch stays |
| 189 | + commit-consistent per leaf; crash → lease reclaim → re-acquire on a surviving pod; saturation backpressure |
| 190 | + (sync `503`, async drains). |
| 191 | +- **Gated live smoke (OCP 4.20.8):** per-sandbox RWO pool, `SH_MODEL=claude-haiku-4-5`, reusing the existing |
| 192 | + `*_LIVE_SMOKE` env-gate pattern; assert N:M fan-out completes with correct verdicts. |
| 193 | + |
| 194 | +## 12. Acceptance mapping (issue #46) |
| 195 | + |
| 196 | +| Issue #46 scope item | Addressed by | |
| 197 | +|----------------------|--------------| |
| 198 | +| Replace fixed `KAGENTI_SANDBOX_POD` with pool assignment (pod-selection logic) | §4 routing + §8 `KAGENTI_SANDBOX_POOL_SELECTOR` | |
| 199 | +| Within a sandbox: shared repo (read-mostly), per-leaf isolated worktrees, `/workspace/<run>/…` layout | §5 shared object store + per-leaf worktree | |
| 200 | +| Across sandboxes: storage topology — RWX vs per-sandbox copy (design both) | D1 + §3 (per-sandbox RWO chosen; RWX = §13 alternative) | |
| 201 | +| Sandbox tier as a managed set with durable storage | §3 N `Sandbox` CRs, per-CR RWO PVC | |
| 202 | +| Open Q — routing mechanism | §4 (harness-side pick + Redis leases) | |
| 203 | +| Open Q — repo distribution | D3 + §5 (ref-pinned lazy converge) | |
| 204 | +| Open Q — sandbox lifecycle/scaling | D4 + §8 (static N config knob) | |
| 205 | +| Open Q — isolation between shared leaves | §9 (worktree/dir, Kata → P3) | |
| 206 | + |
| 207 | +## 13. Non-goals (this phase) |
| 208 | + |
| 209 | +- **Kata isolation and the ~20:1 ratio experiments** — P3 ([#48](https://github.com/kagenti/serverless-harness/issues/48)). |
| 210 | +- **Eager pod-start pre-warm** — P3 optimization layered on the §5 converge contract. |
| 211 | +- **Autoscaling the pool** — future; P2 is static N. |
| 212 | +- **RWX / fleet-wide single repo** — documented alternative below, not implemented. Would require an RWX |
| 213 | + provisioner (EFS/CSI) on AWS OCP, a single ReadWriteMany PVC mounted by all pods, and cross-pod fetch/worktree |
| 214 | + serialization. Chosen against for the deployable PoC (D1). |
| 215 | + |
| 216 | +## 14. Superseded / doc updates |
| 217 | + |
| 218 | +- Updates the **P2 row** in [`docs/specs/README.md`](README.md): status `planned` → `design ✅`, and corrects the |
| 219 | + description (the prior placeholder referenced `SandboxWarmPool`/`SandboxClaim` and "RWX relocates here" — the |
| 220 | + brainstorm instead landed on **N distinct `Sandbox` CRs + per-sandbox RWO copy**, with RWX as a documented |
| 221 | + alternative only). |
| 222 | +- Does **not** alter P1, P0′, or the epic's locked decisions. |
| 223 | + |
| 224 | +--- |
| 225 | + |
| 226 | +*Assisted-By: Claude (Anthropic AI) — brainstorming + spec authoring for P2.* |
0 commit comments