Skip to content

Commit 3bdaeec

Browse files
pdettoripdettori
authored
Merge pull request #35 from kagenti/docs/executive-overview
docs: Rewrite README with value-oriented overview and quick start
2 parents 306e899 + 2ca35a1 commit 3bdaeec

1 file changed

Lines changed: 263 additions & 12 deletions

File tree

‎README.md‎

Lines changed: 263 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,266 @@
1-
# Serverless Harness (Pi track)
1+
# Serverless Harness
22

3-
## Packages
4-
- `packages/session-backend` — generic append-only `LogStore` + Redis Streams impl.
5-
- `harness` — Pi `SessionStorageBackend` adapter (write-behind) + headless smoke.
6-
- `pi-fork` — pinned Pi (base commit `406a2214`, branch `feat/session-storage-backend`) with the injectable `SessionStorageBackend` seam.
3+
**Run stateful AI coding agents serverless — scale to zero between turns, resume exactly where they left off.**
74

8-
## Milestones
5+
![status](https://img.shields.io/badge/status-MVP%20(Phase%201)-success)
6+
![platform](https://img.shields.io/badge/platform-Knative%20%2B%20KEDA-blue)
7+
![runtime](https://img.shields.io/badge/runtime-Pi%20coding%20agent-informational)
8+
![node](https://img.shields.io/badge/node-22%2B-green)
9+
![license](https://img.shields.io/badge/license-Apache--2.0-blue)
910

10-
- **M1 — Redis SessionStorageBackend** (`packages/session-backend`, `harness`): append-only
11-
`LogStore` backed by Redis Streams; write-behind adapter wired into Pi's `SessionStorageBackend`
12-
seam. Design: `docs/specs/2026-06-16-m1-redis-session-backend-design.md`.
13-
- **M2 — K8sSandboxClient** (`packages/k8s-sandbox`): routes Pi tool execution to
14-
a remote Kubernetes pod via `kubectl exec`. Env-gated (`KAGENTI_SANDBOX_POD`);
15-
off by default. Design: `docs/specs/2026-06-17-m2-k8s-sandbox-client-design.md`.
11+
Serverless Harness turns a long-lived AI agent into a **scale-to-zero workload** on Kubernetes.
12+
An agent process normally has to stay resident — holding its conversation, tool state, and working
13+
directory in memory — even while it sits idle waiting for the next turn or for a human to approve a
14+
step. That idle time is pure cost. The harness decouples the agent's **state** (durable in Redis)
15+
and its **tool execution** (an isolated sandbox pod) from the **agent process** itself, so the agent
16+
runs as a Knative service that drops to zero pods when idle and cold-starts with full session
17+
continuity on the next request.
18+
19+
The result is a **leaf-session backend**: an external orchestrator dispatches isolated units of agent
20+
work ("leaves") over a simple HTTP + shared-volume contract, and the harness runs each one
21+
sync, async (queued), scheduled, or paused-for-approval — all on infrastructure that costs nothing at
22+
rest.
23+
24+
## Table of Contents
25+
26+
- [Why](#why)
27+
- [Architecture](#architecture)
28+
- [Features](#features)
29+
- [Quick Start](#quick-start)
30+
- [How It Works](#how-it-works)
31+
- [Dispatch Archetypes](#dispatch-archetypes)
32+
- [Repository Layout](#repository-layout)
33+
- [Evidence](#evidence)
34+
- [Roadmap](#roadmap)
35+
- [Documentation](#documentation)
36+
- [Status & License](#status--license)
37+
38+
---
39+
40+
## Why
41+
42+
| Persistent agent | Serverless Harness |
43+
|------------------|--------------------|
44+
| Process stays resident between turns | Scales to **zero** when idle, cold-starts in sub-second |
45+
| State lives in process memory — lost on crash/evict | State lives in **Redis** — survives eviction, restart, and cold start |
46+
| Tools execute in the agent process | Tools execute in an **isolated sandbox pod** (brain/hands split) |
47+
| Idle compute billed continuously | **Only Redis + sandbox** stay resident (2 pods at rest) |
48+
| One invocation model | **Four**: sync, async fan-out, scheduled, human-gated |
49+
50+
In an idle-heavy workload [experiment](deploy/knative/EXPERIMENTS.md), the serverless path consumed
51+
roughly **a quarter** of the pod-seconds of an equivalent always-on agent — because the expensive
52+
part (the agent process) exists only while a turn is actively running.
53+
54+
---
55+
56+
## Architecture
57+
58+
```mermaid
59+
flowchart LR
60+
O[External Orchestrator] -->|POST /run-leaf| K[Knative Service<br/>scale-to-zero]
61+
C[CronJob<br/>cron-dispatch] -->|schedule| K
62+
K -->|sync| R[runLeaf]
63+
K -->|async: true| Q[(Redis Streams<br/>queue + session state)]
64+
Q --> W[KEDA ScaledJob<br/>leaf-worker 0..N]
65+
W --> R
66+
R -->|kubectl exec| S[sandbox-0<br/>tool / code execution]
67+
R <-->|session state| Q
68+
```
69+
70+
| Component | Role |
71+
|-----------|------|
72+
| **Knative Service** | Scale-to-zero HTTP endpoint; runs a turn inline (sync) or enqueues it (async) |
73+
| **Redis** | Durable session state (resume by `sessionId`), work queue (Streams), gate state |
74+
| **KEDA ScaledJob** | Autoscales `leaf-worker` pods 0→N on queue depth (`lagCount` + `pendingEntriesCount`) |
75+
| **sandbox-0** | Persistent pod where all tool/code execution runs; reached via `kubectl exec` |
76+
| **Shared PVC** | Volume-envelope contract — inputs, results, and markers travel as files |
77+
| **CronJob** | Scheduled dispatch (`cron-dispatch`) for periodic batch work |
78+
79+
> **Note:** The Knative Service and the `leaf-worker` are the **same container image** with two entry
80+
> points (`server.ts` vs `leaf-job.ts`). Both converge on `runLeaf()`, which routes execution into
81+
> `sandbox-0`. The "brain" (model inference + session logic) runs in whichever pod called `runLeaf()`;
82+
> the "hands" (actual command/tool execution) always run in the sandbox.
83+
84+
---
85+
86+
## Features
87+
88+
- **Scale-to-zero turns** — Knative drops the agent to zero pods between turns; the activator
89+
cold-starts a fresh pod on the next request.
90+
- **Durable resume** — sessions are append-only logs in Redis; a cold-started pod recalls full
91+
conversation and state by `sessionId`, surviving pod eviction.
92+
- **Brain/hands isolation** — the agent never executes tools in its own process; everything runs in a
93+
separate hardened `sandbox-0` pod via a persistent in-pod channel.
94+
- **Four dispatch modes** — one `/run-leaf` endpoint serves sync, async-queued, cron-scheduled, and
95+
human-gated execution (see [Dispatch Archetypes](#dispatch-archetypes)).
96+
- **Human-in-the-loop gates** — a leaf can pause mid-run, report `awaiting_approval`, and resume on an
97+
external approve/reject/abort verdict — scaling to zero while it waits.
98+
- **Volume-envelope contract** — orchestrators pass inputs and collect results as files on a shared
99+
PVC, decoupling result size from HTTP limits.
100+
- **Hardened by default** — non-root UID, read-only root filesystem, all capabilities dropped,
101+
`RuntimeDefault` seccomp, no service-account token automount.
102+
- **Built on Pi** — wraps a pinned [`kagenti/pi`](https://github.com/kagenti/pi) coding agent through
103+
an injectable `SessionStorageBackend` seam; the agent itself is unmodified.
104+
105+
---
106+
107+
## Quick Start
108+
109+
Bring up the full stack on a local [Kind](https://kind.sigs.k8s.io/) cluster and drive an agent that
110+
scales to zero and resumes from cold.
111+
112+
> **Prerequisites:** `kind`, `kubectl`, `docker`, and an Anthropic-compatible model credential.
113+
114+
```bash
115+
# 1. Clone (the Pi agent is a submodule)
116+
git clone --recurse-submodules https://github.com/kagenti/serverless-harness.git
117+
cd serverless-harness
118+
119+
# 2. Provide a model credential — either a direct key...
120+
export ANTHROPIC_API_KEY=sk-...
121+
# ...or a Bearer-token gateway (e.g. LiteLLM):
122+
# export ANTHROPIC_BASE_URL=https://your-gateway
123+
# export ANTHROPIC_AUTH_TOKEN=...
124+
125+
# 3. One-shot: create cluster, install Knative + Kourier, deploy Redis + sandbox + harness
126+
./deploy/knative/setup-kind.sh
127+
```
128+
129+
In a second terminal, expose the gateway and watch pods:
130+
131+
```bash
132+
kubectl port-forward -n kourier-system svc/kourier 8080:80 # leave running
133+
watch -n5 'kubectl get pods' # in another pane
134+
```
135+
136+
**Send the first turn** — a pod cold-starts to handle it, then scales back to zero:
137+
138+
```bash
139+
curl -s -H "Host: serverless-harness.default.example.com" \
140+
-H "Content-Type: application/json" \
141+
-d '{"prompt":"Remember the secret word: pineapple. Reply only with OK."}' \
142+
http://localhost:8080/turn | jq .
143+
# => { "sessionId": "019ed8e8-...", "response": "OK" }
144+
```
145+
146+
**Resume across a cold start** — wait ~90s for the pod to terminate, then ask on the *same* session.
147+
A fresh pod spins up from zero and still remembers the state from Redis:
148+
149+
```bash
150+
export SID="<sessionId from above>"
151+
curl -s -H "Host: serverless-harness.default.example.com" \
152+
-H "Content-Type: application/json" \
153+
-d "{\"sessionId\":\"$SID\",\"prompt\":\"What was the secret word?\"}" \
154+
http://localhost:8080/turn | jq .
155+
# => response contains "pineapple"
156+
```
157+
158+
See [`serverless-harness-demo.md`](serverless-harness-demo.md) for the full guided walkthrough
159+
(including sandbox command execution) and [`deploy/knative/SMOKE.md`](deploy/knative/SMOKE.md) for the
160+
verified smoke-test claims.
161+
162+
---
163+
164+
## How It Works
165+
166+
1. **An orchestrator POSTs a leaf** to `/run-leaf` (or `/turn` for a single interactive turn).
167+
2. The **Knative Service** wakes from zero, and either runs the leaf inline (`sync`) or pushes the
168+
envelope onto **Redis Streams** and returns `202` (`async: true`), then idles back to zero.
169+
3. For async work, a **KEDA ScaledJob** scales `leaf-worker` pods up on queue depth and drains items.
170+
4. Both paths call **`runLeaf()`**, which executes all tools inside **`sandbox-0`** via `kubectl exec`.
171+
5. **Session state streams to Redis** as it goes, so the leaf is resumable by `sessionId` even if its
172+
pod dies mid-run.
173+
6. **Results land as files** on a shared PVC, and a **done-marker** signals completion to the
174+
orchestrator.
175+
176+
---
177+
178+
## Dispatch Archetypes
179+
180+
The same backend serves three orchestration patterns, all validated end-to-end on Kind:
181+
182+
| Archetype | Pattern | Example use case |
183+
|-----------|---------|------------------|
184+
| **A — Async fan-out** | `{async:true}` → Redis Streams → KEDA scales workers 0→N → done-markers | "Research 10 topics concurrently" |
185+
| **B — Human gate** | Leaf pauses → `awaiting_approval` → external verdict → resume/terminate | "Draft a clause, pause for legal sign-off, finalize" |
186+
| **C — Scheduled** | CronJob → `cron-dispatch` reads a config list → posts each as async | "Summarize yesterday's tickets at 02:00 daily" |
187+
188+
---
189+
190+
## Repository Layout
191+
192+
```text
193+
serverless-harness/
194+
├── packages/
195+
│ ├── session-backend/ # Generic append-only LogStore + Redis Streams impl
196+
│ ├── k8s-sandbox/ # Routes Pi tool execution to a remote pod (kubectl exec)
197+
│ ├── knative-server/ # HTTP server (server.ts) + leaf-worker (leaf-job.ts) entry points
198+
│ └── work-queue/ # Redis Streams work queue (async dispatch)
199+
├── harness/ # Pi SessionStorageBackend adapter (write-behind) + headless smoke
200+
├── pi-fork/ # Pinned Pi coding agent (submodule) with the injectable backend seam
201+
├── deploy/knative/ # Kind setup, manifests, smoke + experiment drivers
202+
├── experiments/ # @sh/experiments — reproducible cost/behaviour experiments
203+
└── docs/specs/ # Design specs (per-milestone) + milestone registry
204+
```
205+
206+
---
207+
208+
## Evidence
209+
210+
Behaviour and economics are backed by reproducible experiments rather than claims:
211+
212+
- **[`deploy/knative/EXPERIMENTS.md`](deploy/knative/EXPERIMENTS.md)** — cluster experiments E1
213+
(economics), E3 (mobility), E4 (recovery), run live on Kind.
214+
- **[`docs/experiment-results.md`](docs/experiment-results.md)** — E2 (reconstruction cost) and E5
215+
(budget enforcement) from the `@sh/experiments` workspace.
216+
- **[`deploy/knative/SMOKE.md`](deploy/knative/SMOKE.md)** — the 6/6 cold-start + resume smoke claims.
217+
218+
---
219+
220+
## Roadmap
221+
222+
**Phase 1 — Decoupled Harness (built):** Redis session backend, remote sandbox client, persistent
223+
in-pod channel, Knative wrapper, compaction-checkpoint fast path, experiments, and the leaf-session
224+
backend with all three dispatch archetypes. See the
225+
[milestone registry](docs/specs/README.md) for the source-of-truth status of every milestone.
226+
227+
**Phase 2 — Zero-Trust Credential Plane (design complete, deferred):** a credential plane where
228+
*no component influenced by model output ever holds a raw secret.*
229+
230+
| ID | Adds |
231+
|----|------|
232+
| Z1 | Per-session SPIFFE identity (SPIRE) |
233+
| Z2 | Secret-free, default-deny harness lock-down |
234+
| Z3 | Inference injector — provider-key chokepoint, mTLS to the LLM gateway |
235+
| Z4 | MCP code-mode in the sandbox |
236+
| Z5 | Generalized credentialed egress (sandbox forward proxy) |
237+
| Z6 | Subagents as isolated child sessions |
238+
| Z7 | Red-team + formal validation of the credential plane |
239+
240+
Today the harness uses a trust-the-operator model: the model credential is a pre-provisioned
241+
Kubernetes Secret, there is no egress policy, and all leaves share one service-account identity. Those
242+
gaps are exactly what Phase 2 closes.
243+
244+
---
245+
246+
## Documentation
247+
248+
- [Executive overview — leaf-session backend](docs/executive-overview-leaf-session.md)
249+
- [Milestone registry](docs/specs/README.md) — authoritative milestone numbering and status
250+
- [Design specs](docs/specs/) — one dated design doc per milestone
251+
- [`harness/README.md`](harness/README.md) — local dev build (Pi workspace build order, headless smoke)
252+
253+
---
254+
255+
## Status & License
256+
257+
This is a **private research repository** (`kagenti/serverless-harness`). It is an MVP — the
258+
scale-to-zero, durable-resume, sandbox-isolation, and dispatch features above are built and
259+
smoke-verified; the zero-trust credential plane is designed but not yet implemented. Interfaces may
260+
change.
261+
262+
Licensed under the [Apache License 2.0](LICENSE).
263+
264+
---
265+
266+
*Assisted-By: Claude Code*

0 commit comments

Comments
 (0)