Merge pull request #51 from kagenti/docs/p0prime-ocp-fs-free-deployment

Deploy the FS-free harness on OpenShift with a durable sandbox (P0′)

Author: pdettori

deploy/knative/README-ocp.md

@@ -3,22 +3,26 @@
 `deploy/knative/setup-ocp.sh` stands up the serverless-harness stack on
 **OpenShift 4.20+** — the OpenShift-native sibling of [`setup-kind.sh`](setup-kind.sh).
 It installs OpenShift Serverless (Knative + Kourier), Redis, the sandbox pod, the
-`leaf-work` PVC, the LLM-credentials secret, and the harness Knative Service,
-reachable over its **auto-created OpenShift Route**.
+LLM-credentials secret, and the harness Knative Service, reachable over its
+**auto-created OpenShift Route**.
 
 Base bring-up only — see [Scope](#scope) for what is deferred.
 
 ## Prerequisites
 
 - **`oc`**, logged in to an OpenShift **4.20+** cluster as **cluster-admin**
   (operator installs + SCC assignment require it).
-- A default **StorageClass** for the `leaf-work` PVC (the script fails fast if
+- A default **StorageClass** for the sandbox's durable `/workspace` PVC (the script fails fast if
   none exists). See the [storage caveat](#storage--scc).
 - A model credential:
   - `ANTHROPIC_API_KEY` (direct), **or**
   - `ANTHROPIC_AUTH_TOKEN` + `ANTHROPIC_BASE_URL` (Bearer-token gateway, e.g. LiteLLM).
 - The harness image. By default the script pulls the published
   `ghcr.io/kagenti/serverless-harness:latest`; override with `--image`.
+- **agent-sandbox controller** (kubernetes-sigs v0.5.0) is installed by the script
+  (`sandboxes.agents.x-k8s.io`); it creates the `sandbox-0` pod from the Sandbox CR
+  and provisions its durable `/workspace` PVC. The harness resolves the pod via the
+  CR's `.status.selector` and `kubectl exec`s tool calls into it.
 
 ## Quick start
 
@@ -63,7 +67,7 @@ creates a real Route per Knative Service (`oc get ksvc serverless-harness -o jso
 | Knative config | Autoscaler tuning + the `podspec-persistent-volume-claim`/`-write`/`-securitycontext` feature flags are set in the **`KnativeServing` CR spec** (the operator reverts direct `config-*` ConfigMap patches). |
 | Redis | Lightweight in-repo Deployment (`redis:7-alpine`), runs under `restricted-v2`. |
 | Sandbox | Pre-baked image ([`sandbox.Dockerfile`](sandbox.Dockerfile), `USER 65532`), built in-cluster against the internal registry (or supplied via `--sandbox-image`). |
-| `leaf-work` PVC | `ReadWriteOnce`, cluster-default StorageClass. |
+| Sandbox `/workspace` PVC | `ReadWriteOnce` (Sandbox CR `volumeClaimTemplates`), cluster-default StorageClass. |
 | Harness | Knative Service applied via the [`overlays/ocp`](overlays/ocp) kustomize overlay; SA granted the `nonroot-v2` SCC. |
 | Ingress | Auto-created OpenShift Route. |
 
@@ -144,11 +148,12 @@ and verifying the async-leaf path itself on OpenShift is a further step.
 
 ## Storage & SCC
 
-- **Storage / RWX.** `leaf-work` is `ReadWriteOnce`. On block storage (e.g. AWS EBS
+- **Storage / RWX.** The sandbox's `/workspace` PVC is `ReadWriteOnce`. On block storage (e.g. AWS EBS
   `gp3-csi`) it binds to a single node — fine for a single harness consumer.
   Concurrent multi-node scale-out, or co-mounting with the leaf-orchestrator, needs
   a **RWX** StorageClass (a filesystem provisioner). The base bring-up does not deploy
   the orchestrator. Set a specific class by making it the cluster default before install.
+  RWX, if ever needed for a shared sandbox pool, lives on the sandbox tier (P2) — never the harness.
 - **SCC.** The published harness image declares no `USER` (defaults to root), so it
   runs as an explicit non-root UID (65532) and the script grants the harness
   ServiceAccount the `nonroot-v2` SCC (`oc adm policy add-scc-to-user nonroot-v2 -z
@@ -174,7 +179,7 @@ and verifying the async-leaf path itself on OpenShift is a further step.
 |---------|-------------|
 | `ksvc` never Ready, pod `CreateContainerConfigError: container has runAsNonRoot and image will run as root` | The `nonroot-v2` SCC grant didn't apply. Re-run the script, or `oc adm policy add-scc-to-user nonroot-v2 -z serverless-harness -n <ns>`. |
 | `ksvc` never Ready, pod `CrashLoopBackOff` with `ERR_MODULE_NOT_FOUND` | The harness image is broken/stale. Use a newer `--image` (the fix shipped in the image build; see the repo history). |
-| `leaf-work` PVC stuck `Pending` | No (default) StorageClass. Set one, or ensure a provisioner is installed. |
+| Sandbox `/workspace` PVC stuck `Pending` | No (default) StorageClass. Set one, or ensure a provisioner is installed. |
 | `oc apply -k overlays/ocp` fails with a load-restrictor / "not in or below" error | The overlay references shared base YAMLs one level up. Render with `oc kustomize --load-restrictor LoadRestrictionsNone deploy/knative/overlays/ocp \| oc apply -f -` — `setup-ocp.sh` does this for you. |
 | `/turn` returns `"Connection error"` | The harness can't reach its configured Anthropic endpoint from the cluster (egress/gateway reachability). `/health` and session creation still work. |
 
@@ -183,7 +188,11 @@ and verifying the async-leaf path itself on OpenShift is a further step.
 ```bash
 oc delete ksvc serverless-harness -n default
 oc delete -k <(oc kustomize --load-restrictor LoadRestrictionsNone deploy/knative/overlays/ocp) 2>/dev/null || true
-oc delete pod sandbox-0 deployment/redis svc/redis pvc/leaf-work secret/llm-credentials -n default
+oc delete sandbox sandbox-0 deployment/redis svc/redis secret/llm-credentials -n default
+# The durable /workspace PVC is provisioned StatefulSet-style from the Sandbox CR's
+# volumeClaimTemplates and is NOT garbage-collected when the CR is deleted — remove it
+# explicitly to reclaim the backing EBS volume:
+oc delete pvc workspace-sandbox-0 -n default
 # Operators (optional): oc delete knativeserving knative-serving -n knative-serving; oc delete subscription serverless-operator -n openshift-serverless
 ```
 

deploy/knative/leaf-smoke.sh

@@ -40,7 +40,7 @@ dispatch_item() {
   local body
   body=$(jq -nc --arg s "$sid" --arg m "$model" --arg id "$id" --arg f "$file" --arg p "$pat" --arg ws "$SBOX_REPO" \
     '{sessionId:$s, model:$m, workspaceRef:$ws, item:{item_id:$id, file:$f, pattern:$p}}')
-  curl -s --max-time 240 -H "$HOST_HEADER" -H "Content-Type: application/json" -d "$body" "$BASE/runs"
+  curl -s $CURL_OPTS --max-time 240 ${CURL_HDR[@]+"${CURL_HDR[@]}"} -H "Content-Type: application/json" -d "$body" "$BASE/runs"
 }
 
 # dispatch <item_id> [model] -> echoes terminal JSON from /runs
@@ -129,7 +129,7 @@ if [ "$bogus" = "failed" ] && [ "$good" = "done" ]; then ok "bogus model -> fail
 # is the server's isLeafEnvelope/validateItem guard: a body with no `item` must return HTTP 400.
 claim 5 "Malformed envelope (missing item) is rejected with HTTP 400"
 neg_body=$(jq -nc --arg s "$RUN/ineg" '{sessionId:$s}')
-neg_code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 -H "$HOST_HEADER" -H "Content-Type: application/json" -d "$neg_body" "$BASE/runs")
+neg_code=$(curl -s $CURL_OPTS -o /dev/null -w '%{http_code}' --max-time 30 ${CURL_HDR[@]+"${CURL_HDR[@]}"} -H "Content-Type: application/json" -d "$neg_body" "$BASE/runs")
 if [ "$neg_code" = "400" ]; then ok "malformed envelope rejected (HTTP 400)"; else ko "expected HTTP 400, got $neg_code"; fi
 
 # --- Claim 6: idempotent re-invoke returns a valid verdict ---

deploy/knative/overlays/ocp/kustomization.yaml

@@ -6,42 +6,36 @@
 #   - Harness image: pull the published multi-arch image from GHCR instead of the
 #     Kind-local dev.local tag. Override with `kustomize edit set image` /
 #     setup-ocp.sh --image.
-#   - Sandbox image: the pre-baked image built in-cluster (sandbox.Dockerfile),
-#     referenced by its internal-registry pullspec. Default assumes namespace
-#     `default`; setup-ocp.sh rewrites this for --namespace.
-#   - Sandbox pod: no root apk-add at start-up (tools are pre-baked), plus
-#     restricted-v2-compatible securityContext and a writable /workspace emptyDir
-#     (patch-sandbox.yaml).
+#   - Sandbox image + pod: the pre-baked image (sandbox.Dockerfile) and the
+#     non-root securityContext are set via a JSON6902 patch onto the Sandbox CR
+#     (patch-sandbox.yaml): runAsUser/fsGroup 65532, seccomp RuntimeDefault,
+#     drop ALL caps, and serviceAccountName serverless-harness-sandbox. The
+#     command is `sleep infinity` (no root apk-add).
 #
-# The harness Service keeps the base runAsUser/fsGroup 65532: the published GHCR
-# image declares no USER (defaults to root), so a non-root UID must be set
-# explicitly — the SCC does not reliably inject one. setup-ocp.sh grants the
-# harness ServiceAccount the `nonroot-v2` SCC so that explicit non-root UID is
-# admitted (issue #41 item #4, approach b). The pre-baked sandbox image, by
-# contrast, sets USER 65532 itself, so its pod needs no SCC grant.
+# Both tiers run non-root: setup-ocp.sh grants nonroot-v2 to the harness SA
+# (serverless-harness) AND the sandbox SA (serverless-harness-sandbox), since the
+# GHCR harness image declares no USER and the SCC does not reliably inject a UID.
 #
-# The leaf-work PVC keeps ReadWriteOnce (the only mode AWS EBS / gp3-csi offers);
-# see SMOKE.md for the RWX caveat.
+# The sandbox's /workspace is backed by the Sandbox CR's durable RWO PVC
+# (volumeClaimTemplates) — the only mode AWS EBS / gp3-csi offers. RWX is a P2
+# concern on the sandbox tier, never the harness. There is no leaf-work PVC after P1.
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 
 resources:
   - ../../redis.yaml
   - ../../sandbox.yaml
-  - ../../leaf-pvc.yaml
   - ../../service.yaml
 
 images:
   - name: dev.local/serverless-harness
     newName: ghcr.io/kagenti/serverless-harness
     newTag: latest
-  - name: alpine
-    newName: image-registry.openshift-image-registry.svc:5000/default/serverless-harness-sandbox
-    newTag: latest
 
 patches:
   - path: patch-sandbox.yaml
     target:
-      version: v1
-      kind: Pod
+      group: agents.x-k8s.io
+      version: v1beta1
+      kind: Sandbox
       name: sandbox-0

deploy/knative/overlays/ocp/patch-sandbox.yaml

@@ -1,28 +1,32 @@
-# OpenShift variant of the sandbox pod. The base runs `apk add ...` as root at
-# start-up (blocked by restricted-v2); the tools are pre-baked into the image
-# (sandbox.Dockerfile, swapped in via the overlay `images:` transformer), so the
-# command is just `sleep infinity`. Adds restricted-v2-compatible securityContext
-# and mounts an emptyDir at /workspace (fsGroup-owned → writable by the assigned
-# UID) so the sandbox can write regardless of root-filesystem permissions.
-apiVersion: v1
-kind: Pod
-metadata:
-  name: sandbox-0
-spec:
-  securityContext:
+# OpenShift variant of the sandbox pod, applied as a JSON6902 patch onto the
+# Sandbox CR (agents.x-k8s.io/v1beta1). The base (Kind) runs `apk add ...` as root
+# at start-up — blocked by OCP SCCs. Here: tools are pre-baked into the image
+# (sandbox.Dockerfile), so the command is just `sleep infinity`; the image is the
+# in-cluster-built internal-registry pullspec; the pod runs non-root (UID 65532,
+# admitted by nonroot-v2 on the serverless-harness-sandbox SA) with fsGroup 65532
+# so the Sandbox CR's durable /workspace PVC (from volumeClaimTemplates) is writable.
+# NOTE: we deliberately do NOT touch volumeClaimTemplates/volumeMounts — /workspace
+# stays on the durable RWO PVC (the point of P0′), not an emptyDir.
+- op: replace
+  path: /spec/podTemplate/spec/containers/0/image
+  value: image-registry.openshift-image-registry.svc:5000/default/serverless-harness-sandbox:latest
+- op: replace
+  path: /spec/podTemplate/spec/containers/0/command
+  value: ["sleep", "infinity"]
+- op: add
+  path: /spec/podTemplate/spec/serviceAccountName
+  value: serverless-harness-sandbox
+- op: add
+  path: /spec/podTemplate/spec/securityContext
+  value:
+    runAsUser: 65532
     runAsNonRoot: true
+    fsGroup: 65532
     seccompProfile:
       type: RuntimeDefault
-  containers:
-    - name: sandbox
-      command: ["sleep", "infinity"]
-      securityContext:
-        allowPrivilegeEscalation: false
-        capabilities:
-          drop: ["ALL"]
-      volumeMounts:
-        - name: workspace
-          mountPath: /workspace
-  volumes:
-    - name: workspace
-      emptyDir: {}
+- op: add
+  path: /spec/podTemplate/spec/containers/0/securityContext
+  value:
+    allowPrivilegeEscalation: false
+    capabilities:
+      drop: ["ALL"]

deploy/knative/sandbox.Dockerfile

@@ -10,16 +10,14 @@
 # Built in-cluster against the OpenShift internal registry by setup-ocp.sh
 # (oc new-build --binary --strategy=docker). The harness routes agent tool
 # execution into this pod via `kubectl exec`, so it needs bash + GNU coreutils,
-# findutils and grep on PATH.
+# findutils, grep and ripgrep (the agent's find/grep tools shell out to `rg`) on PATH.
 FROM alpine:3.20
 
-RUN apk add --no-cache bash coreutils findutils grep
+RUN apk add --no-cache bash coreutils findutils grep ripgrep
 
-# OpenShift assigns an arbitrary UID at runtime that belongs to the root group
-# (GID 0). Make /workspace owned by and writable for the root group so the
-# sandbox can write there regardless of which UID the SCC injects. (setup-ocp.sh
-# also mounts an emptyDir over /workspace, which fsGroup makes writable — this is
-# belt-and-suspenders for direct `docker run` / non-mounted use.)
+# OpenShift assigns the pod a non-root UID; the OCP overlay pins runAsUser/fsGroup
+# 65532 and backs /workspace with the Sandbox CR's durable PVC (fsGroup-owned, so
+# writable). The chgrp/chmod below is belt-and-suspenders for GID-0 arbitrary-UID use.
 RUN mkdir -p /workspace \
     && chgrp -R 0 /workspace \
     && chmod -R g=u /workspace