promoted-workflow-demo.md

Demo: "The workflow you built on your laptop, running in the cluster"

A ~8-minute walkthrough of workflow promotion: you author an agent workflow in Claude Code — a skill, a CLAUDE.md, a memory file, a slash command — and run it unchanged in the harness by adding one field to a dispatch.

The task — write a ship note — is just a vehicle. The real show is what the remote agent knows. You will send the same prompt twice, to the same cluster, differing only by configRef, and watch one run ask what a ship note even is while the other cites an incident id that exists nowhere but your laptop's memory directory.

laptop                                    cluster
------                                    -------
/tmp/sh-demo               promote        harness pod (fs-free)      sandbox pod
  CLAUDE.md          --> canonical tar -->  /tmp/sh-config/<digest>/   /workspace/.sh-config/<digest>/
  .claude/skills/          sha256 in           skills/  context/            skills/
  .claude/commands/         Redis              (system prompt)         (what `read` can see)
  .claude/projects/*/memory/

One digest names both halves. The dispatch carries the digest and nothing else about the workflow.

Act	What "move my workflow to the server" normally needs	What promotion needs
1 — Author small	A hand-written manifest listing what to ship, kept in sync by hand	One env var. HOME=$SH_DEMO_SANDBOX makes the sandbox its own user scope
2 — Promote	An image rebuild, a redeploy, a registry push	/promote, inside Claude Code — a 12 KB tar, content-addressed; re-promotion uploads nothing
3 — Run it	A bespoke endpoint that knows about your skills	One field. "configRef": "sha256:…" on the existing prompt envelope
4 — Know it landed	Read the pod logs and hope	The run cites a fact only your memory holds, and a token only the sandbox can read

Promotion runs from inside Claude Code as /promote, in the same session you authored the workflow in — Act 0 installs it. Prefer it non-interactive? make demo-promoted-workflow does all of this and asserts every claim, driving the CLI directly since it has no Claude session to run in. This document is the version you drive by hand so you can explain each move.

---

Act 0: Install

You need a warm harness cluster whose image contains the promotion feature (merged in #214). If you do not have one:

git clone --recurse-submodules https://github.com/rossoctl/serverless-harness.git
cd serverless-harness
cd pi-fork && npm ci && npm run build && cd ..
pnpm install

export ANTHROPIC_API_KEY=sk-...    # ...or a gateway: ANTHROPIC_BASE_URL + ANTHROPIC_AUTH_TOKEN
./deploy/knative/setup-kind.sh

The model must be reachable from the cluster — both runs are real model calls.

If your cluster is already warm but predates #214, rebuild and force a new Revision. The image tag is mutable, so re-applying an unchanged spec rolls nothing and you would keep serving the old code:

docker build --load -t dev.local/serverless-harness:local .
kind load docker-image dev.local/serverless-harness:local --name sh-knative
kubectl -n default patch ksvc serverless-harness --type merge \
  -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"deploy.sh/build-ts\":\"$(date +%s)\"}}}}}"
kubectl wait ksvc/serverless-harness -n default --for=condition=Ready --timeout=180s

Set the convenience vars used throughout:

export NS=default KSVC=serverless-harness
export HOSTHDR='Host: serverless-harness.default.example.com'
export BASE=http://localhost:8080

# The SH_* names are the ones demo-promoted-workflow.sh and the make targets read, so the
# hand-driven and scripted halves of this walkthrough stay on the same sandbox and port.
export SH_DEMO_SANDBOX=/tmp/sh-demo
export SH_DEMO_REDIS_PORT=16379
export SH_MODEL=claude-haiku-4-5

# Where this repo is checked out. `/promote` reads this, because from Act 1b onward your shell's
# cwd is the sandbox, not the checkout -- and the CLI has no installed binary.
export SH_HARNESS_DIR=$(pwd)

Install the /promote command

Act 2 runs promotion from inside Claude Code, and there are two places to install it. They differ in one thing that matters for this demo: whether the session you author in resembles the one that runs remotely.

Option A — real user scope. Simpler; use this if you are just following along.

mkdir -p ~/.claude/commands
cp deploy/claude/commands/promote.md ~/.claude/commands/promote.md

Claude Code sees /promote in every project, and promote — reading the sandbox as user scope — never sees it, so it cannot travel. The cost: your authoring session also loads your real ~/.claude, so locally the agent has every skill you own while the promoted run gets the sandbox's one.

Option B — in the sandbox, for actual parity. Put the command in the sandbox, so the local agent can be made to see exactly what the promoted run will:

mkdir -p $SH_DEMO_SANDBOX/.claude/commands
cp deploy/claude/commands/promote.md $SH_DEMO_SANDBOX/.claude/commands/promote.md

Installing is all that happens here. The matching launch (HOME=$SH_DEMO_SANDBOX claude) belongs in Act 1b, and only after Act 1a has populated the sandbox: run it now and Claude Code would start on an empty sandbox with no tunnels open, and your shell would be left in $SH_DEMO_SANDBOX, where Act 1a's repo-relative cp paths cannot resolve.

This is what --exclude-prompt is for. With HOME pointed at the sandbox, that .claude/commands/ is promote's prompts directory, and every markdown file in it travels — so without the flag the command would ship itself into every bundle as a prompt template. /promote detects that it lives in the project and adds --exclude-prompt promote itself; you do not pass it by hand. It adds the flag only in that case, because an exclusion matching nothing warns — the flag's own typo guard.

The trade: a fresh HOME means Claude Code re-authenticates, and you lose your own skills for the duration. That is the standard dev/prod-parity cost, and parity is the whole claim this demo makes, so Option B is the more honest way to perform it.

Either option produces the same bundle, so every digest quoted below holds for both. Measured: Option B without the exclusion is 19456 bytes (the command itself travelling); with it, 12288 bytes and sha256:43b8c4c0… — byte-identical to Option A.

Open the two tunnels

kubectl port-forward -n kourier-system svc/kourier 8080:80 &
kubectl port-forward -n default svc/redis 16379:6379 &
export REDIS_URL=redis://localhost:16379

Why 16379 and not 6379. A listener on 6379 is not evidence it is this cluster's Redis. This repo's own test container (sh-tdd-redis) publishes 0.0.0.0:6379; promote will happily upload into it, print uploaded, and then the harness — reading the cluster's Redis — fails with config bundle not found for the digest sitting right there in your terminal. Bind a private port instead. This cost the author two model calls before the failure surfaced.

0a. Prove the cluster implements promotion before you trust anything else

An unknown digest must fail before any model call. An image without the feature ignores the field and answers normally — which would make Act 3's contrast look like a model mood swing rather than a missing deployment:

curl -s -H "$HOSTHDR" -H 'Content-Type: application/json' \
  -d '{"sessionId":"probe-1","kind":"prompt","prompt":"say hi",
       "configRef":"sha256:ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"}' \
  $BASE/runs | jq -c .

{ "status": "failed", "reason": "error", "message": "config bundle not found: sha256:ffff…" }

If you get {"status":"responded", …} instead, the cluster is serving a pre-#214 image. Stop and do the forced roll above. A green demo on the wrong image proves nothing.

---

Act 1: Author in a sandbox, not in your whole ~/.claude

Say this out loud, because it is the part people get wrong. The tempting demo is "promote my real Claude Code setup". Don't. Measured on the author's laptop: a real ~/.claude resolves 62 skills, 56 of which travel, into an ~8.6 MB bundle with dozens of preflight warnings — nearly all from skills the workflow never uses. A curated sandbox ships one skill in 12 KB with zero findings. That is the design's own recommendation (spec §11) and this act is it.

1a. Build the workflow

Four files. A skill that defines a format, a CLAUDE.md that sets a house rule, a memory file holding a fact, and a slash command as the entry point:

mkdir -p $SH_DEMO_SANDBOX/.claude/skills/ship-note/references $SH_DEMO_SANDBOX/.claude/commands
cp -R deploy/knative/fixtures/promoted-demo/skills/. $SH_DEMO_SANDBOX/.claude/skills/
cp -R deploy/knative/fixtures/promoted-demo/commands/. $SH_DEMO_SANDBOX/.claude/commands/
cp deploy/knative/fixtures/promoted-demo/CLAUDE.md $SH_DEMO_SANDBOX/CLAUDE.md

# Claude Code's own memory layout: every path separator in the project path becomes '-'
MEM="$SH_DEMO_SANDBOX/.claude/projects/-$(echo "${SH_DEMO_SANDBOX#/}" | tr '/' '-')/memory"
mkdir -p "$MEM" && cp deploy/knative/fixtures/promoted-demo/memory/*.md "$MEM/"

git -C $SH_DEMO_SANDBOX init -q

# Claim this directory as the demo's own. demo-promoted-workflow.sh gates every destructive
# touch -- including `--teardown` -- on this marker, so without it the scripted teardown will
# (correctly) refuse to remove the sandbox you just built.
touch $SH_DEMO_SANDBOX/.sh-demo-sandbox

Why git init. promote bounds its CLAUDE.md walk at a .git entry. Without one it climbs past your sandbox into ancestor directories and sweeps their context files — including a personal ~/CLAUDE.md — into a bundle bound for a shared store.

The memory file is the one to read aloud, because it is what makes Act 3 undeniable:

cat "$MEM/auth-timeout-incident.md"

It records that the regression is tracked as KAG-4471 and ships behind auth.idle_reaper_v2. Nothing else in the demo knows that.

1b. Watch it work locally first

cd $SH_DEMO_SANDBOX && claude

Under Option B, launch it this way instead — a sandbox HOME is what makes the local agent's configuration identical to the promoted run's:

cd $SH_DEMO_SANDBOX && HOME=$SH_DEMO_SANDBOX claude

Ask it: Write the ship note for the auth timeout fix. You get the house SHIP NOTE block with the ticket and the risk line. This is the "works on my laptop" baseline — the thing that normally does not survive the trip.

Leave this Claude Code session open. You will drive the rest of the demo from it: it is the terminal where the workflow was authored, so promoting and dispatching from here is the point.

---

Act 2: Promote

2a. One slash command, from the session you authored in

In the Claude Code session from Act 1b — still in the sandbox — run:

/promote ship-note

Claude checks the three guards, starts the Redis tunnel if it is not up, and runs the CLI for you:

project:    /tmp/sh-demo
inventory:  …/sandbox-inventory/ghcr.io_rossoctl_serverless-harness-sandbox_latest.json (347 binaries)
  resolved   1 skills
  travels    1
  dropped    0
  context    2 file(s), 1 memory file(s)
  secrets    no blocking findings
  entry      ship-note

preflight: no findings

  bundle     sha256:43b8c4c0…  (12288 bytes, uploaded)
  lockfile   .claude/promoted.lock.json

dispatch with:  {"sessionId":"<run>/<item>","kind":"prompt","prompt":"…","configRef":"sha256:43b8c4c0…"}

The HOME override is the demo, not a shortcut — /promote sets it for you, but say what it does. promote reads user scope from $HOME/.claude; pointing HOME at the sandbox makes the sandbox its own user scope, so the bundle holds this workflow and nothing else. It is also what makes this project's entry prompt travel at all, since promote reads prompts from user scope only. The command it runs is:

HOME="$PWD" REDIS_URL=redis://localhost:16379 \
  pnpm --dir "$SH_HARNESS_DIR/harness" promote --entry ship-note --project "$PWD"
#   ...plus --exclude-prompt promote under Option B, which /promote adds for you

That trailing flag is not cosmetic: under Option B the command is in the sandbox, so without it the bundle is 19456 bytes rather than the 12288 printed above.

--project "$PWD" looks redundant and is not: without it the CLI promotes the directory the process started in, which through pnpm --dir is the harness checkout. Measured — it reports project: …/serverless-harness/harness.

Note travels 1, preflight: no findings, and 12288 bytes. Those three numbers are the argument for sandbox-first authoring, and the last line hands you the exact dispatch envelope.

The lockfile is committable, and it is also the most convenient place to read the digest back from:

export DIGEST=$(jq -r .digest $SH_DEMO_SANDBOX/.claude/promoted.lock.json)
echo $DIGEST
# => sha256:43b8c4c04fc084ed0df724572082ab35b381cd803d86675b8e52a2efb5e7cee6

2b. Verify it landed in the cluster's Redis, not somewhere else

Read the key back through the cluster's own client rather than your port-forward. A forward pointing at the wrong Redis passes every check up to here:

kubectl exec -n $NS deploy/redis -- redis-cli EXISTS "config:bundle:$DIGEST"
# => 1

2c. Re-promotion is free

Run it again, unchanged:

/promote ship-note

  bundle     sha256:43b8c4c0…  (12288 bytes, unchanged — upload skipped)

Same digest, no upload. The bundle is content-addressed over a canonical tar — sorted paths, normalised mtimes and modes — so re-promoting unchanged configuration is a no-op, and the digest does not even depend on which directory you authored in.

---

Act 3: The same prompt, one field apart

Both dispatches are byte-identical but for configRef. Same cluster, same model, same prompt.

3a. Purge the shared cache first, or your control is not a control

Do not skip this on a warm cluster. The overlay materialises the bundle into a digest-keyed cache inside the shared pool sandbox and leaves it there for reuse. It is world-readable, it contains context/agents/0-CLAUDE.md and memory/, and it outlives the leaf:

# The leaf may have leased ANY pool sandbox, so ask the pool rather than guessing a pod.
export POOL=$(kubectl get pods -n $NS -l 'sh.kagenti.io/sandbox-pool=default' -o name | sed 's|pod/||')
for p in $POOL; do kubectl exec -n $NS "$p" -- find /workspace/.sh-config -type f 2>/dev/null; done \
  | sed 's|.*/sha256-[0-9a-f]*/||'
# context/MEMORY.md
# context/agents/0-CLAUDE.md
# memory/auth-timeout-incident.md
# skills/ship-note/SKILL.md
# skills/ship-note/references/release-token.md

This bit the author, so it will bite you. On the second run of this demo the bare arm answered TICKET: KAG-4471 / TOKEN: SHIPNOTE-7F3A-SANDBOX-OK and opened with "following the house rules" — having been told none of it. A bare leaf that leases a sandbox where a previous promoted leaf ran can simply explore the filesystem and answer from someone else's promoted workflow. The A/B still looked plausible; it had just stopped proving anything.

for p in $POOL; do
  kubectl exec -n $NS "$p" -- sh -c 'rm -rf /workspace/.sh-config/sha256-*'
done

Empty the whole cache, not just this digest. Purging only the digest you are about to dispatch is not enough: any other cached bundle carrying the same memory fact answers the bare arm just as well. That is not hypothetical — a bundle built from this same fixture before Prettier normalised one emphasis marker in the memory file has a different content digest, sits alongside, and made this control fail while a per-digest purge reported success.

3b. Run A — bare

curl -s -H "$HOSTHDR" -H 'Content-Type: application/json' \
  -d "$(jq -nc --arg m "$SH_MODEL" '{sessionId:"demo-bare-1", kind:"prompt", model:$m,
        prompt:"Write the ship note for the auth timeout fix."}')" \
  $BASE/runs | jq -r .text

The workspace appears to be empty. Could you provide me with:

1. **Where is the code/project?** …
3. **Format for the ship note:**
   - Is there an existing SHIP_NOTES, CHANGELOG, or similar file I should follow?

It does not know what a ship note is here, has no incident to cite, and asks you. This is the harness's normal behaviour, unchanged — which is the back-compat guarantee, stated as output.

3c. Run B — the same prompt, plus the digest

curl -s -H "$HOSTHDR" -H 'Content-Type: application/json' \
  -d "$(jq -nc --arg m "$SH_MODEL" --arg c "$DIGEST" '{sessionId:"demo-promoted-1", kind:"prompt", model:$m,
        prompt:"Write the ship note for the auth timeout fix.", configRef:$c}')" \
  $BASE/runs | jq -r .text

SHIP NOTE · auth-idle-reaper-fix
WHAT:   Fixed sessions disconnecting after 30 seconds of inactivity
WHY:    The idle reaper was measuring time since connection open instead of last activity
TICKET: KAG-4471
RISK:   low — fix is reversible behind auth.idle_reaper_v2 feature flag
TOKEN:  SHIPNOTE-7F3A-SANDBOX-OK

Pause here and read the four lines back one at a time. Each one is a different channel arriving, and each is separately checkable:

• The format came from the skill. Pi puts only a skill's name and description in the system prompt and tells the model to read the body when it matches — and that read runs in the sandbox pod.
• TICKET: KAG-4471 came from your memory directory. The model cannot guess it; Run A demonstrably did not produce it. Memory travels read-only (ADR-0031), so a promoted run consumes what you taught it and stays replayable.
• RISK: came from CLAUDE.md, injected as an agents file — the deterministic channel.
• TOKEN: is the load-bearing one. It lives in references/release-token.md inside the skill's own directory. Producing it means the bundle materialised on the sandbox side of the fs-free split and a relative sibling path resolved against the absolute skills root the leaf injects. The skill even says "if you cannot read that file, write TOKEN: unavailable", so a failed read shows up as an honest gap instead of a hallucinated token.

3d. Prove the sandbox half on the filesystem, not in the model's prose

The TOKEN: line is the model telling you it read the file. Now check the claim directly — this holds even on a run where the model declines to read:

TOKEN_PATH="/workspace/.sh-config/sha256-${DIGEST#sha256:}/skills/ship-note/references/release-token.md"
for p in $POOL; do
  kubectl exec -n $NS "$p" -- cat "$TOKEN_PATH" 2>/dev/null && echo "  ^ in $p" && break
done
# => SHIPNOTE-7F3A-SANDBOX-OK
#      ^ in sandbox-0

Which pod it lands in is not fixed — the leaf leases whichever pool sandbox is free, which is why this loops instead of naming sandbox-0. A hardcoded pod here fails as "broken feature" when it was only a wrong guess.

You purged this path in 3a and dispatched nothing but a digest. It is back, references/ and all, because the overlay put it there under flock and made it read-only with chmod -R a-w — which is how ADR-0031's read-only guarantee is enforced by the filesystem rather than by convention.

3e. Status is durable, not just a response body

curl -s -H "$HOSTHDR" "$BASE/runs/status?sessionId=demo-promoted-1" | jq -c '{status, reason}'
# => {"status":"responded","reason":null}

The verdict is persisted under the session id, so a re-dispatch of the same id resumes rather than re-pays, and a fan-out driver can collect results after the fact. This is the existing leaf idempotency contract; promotion did not change it.

---

What just happened

You moved a workflow off your laptop without writing a manifest:

1. Authored small — one skill, a CLAUDE.md, one memory file, one slash command, in a throwaway sandbox. HOME=$SH_DEMO_SANDBOX made it its own user scope: 1 skill, 12 KB, zero preflight findings, against 56 skills and ~8.6 MB for a real ~/.claude (Act 1, 2a).
2. Promoted with one slash command, without leaving Claude Code — /promote ship-note, in the same session you authored the workflow in. Content-addressed over a canonical tar, so the second promotion uploaded nothing and the digest was identical (Act 2a, 2c).
3. Dispatched with one field — configRef on the existing prompt envelope. No new endpoint, no redeploy, no image rebuild (Act 3c).
4. Proved all three channels arrived — the skill's format, the CLAUDE.md rule, and an incident id that exists only in your memory directory, with a bare run standing next to it as a control you made honest by purging the shared cache first (Act 3a, 3b, 3c).
5. Proved the sandbox half twice — once in the model's words (a token readable only by a read executed in the separate sandbox pod, through a path the leaf translated) and once on the filesystem, which does not depend on the model cooperating (Act 3c, 3d).
6. Kept the old contract — the bare run behaved exactly as the harness always did, and /runs/status replayed the verdict (Act 3b, 3e).

To replay all of it non-interactively with every claim asserted:

make demo-promoted-workflow

---

Cleanup

make demo-promoted-workflow-teardown        # removes the sandbox (needs the Act 1a marker)
pkill -f 'kubectl port-forward -n kourier-system svc/kourier'
pkill -f 'kubectl port-forward -n default svc/redis'

The bundle stays in Redis under a 30-day TTL. It is immutable and content-addressed, so leaving it costs 12 KB and makes the next run of this demo skip the upload. To drop it:

kubectl exec -n $NS deploy/redis -- redis-cli DEL "config:bundle:$DIGEST"

---

Notes and limits

• Option A's authoring session is not what runs remotely. With your real HOME, Claude Code loads your whole ~/.claude while the promoted run gets the sandbox's one skill — so "it behaved the same locally" is weaker evidence than it looks. Option B closes that with HOME=$SH_DEMO_SANDBOX plus --exclude-prompt, at the cost of a re-auth. Say which one you ran if someone asks whether local matched remote.
• The TOKEN: line is the one model-dependent claim. The format and the token both require the model to choose to read the skill body. CLAUDE.md says it MUST, and the deterministic channel carries that instruction, so it is reliable in practice — but it is not a mechanical guarantee the way TICKET: is. If it ever comes back unavailable, that is the skill being honest, not the overlay silently failing.
• The shared config cache is visible to other leaves, and that is worth saying out loud. The digest-keyed directory in the pool sandbox is what makes reuse cheap, and the overlay makes it read-only — but read-only is not invisible. Any leaf leasing that sandbox can read another workflow's promoted CLAUDE.md and memory/, with or without a configRef of its own. Act 3a works around it for the demo's sake. Tracked as #216: the narrow point is that the cache outlives the leaf that made it, so a configRef-less leaf can answer from it — which makes spec §2 goal 6 ("absent a promoted bundle, harness behavior is unchanged") true in the harness process but not observably true. Cross-leaf reading itself is an accepted non-goal (P2 §9, one trust domain; Kata isolation is P3/#48), and ADR-0031 speaks to write-protection rather than to visibility or lifetime. Once the cache no longer outlives its leaf, Act 3a's purge can go.
• This demo does not show MCP servers or subagents. Both are explicitly out of scope for promotion (spec §2, §9). Do not let a room infer that a promoted workflow carries its MCP config.
• Memory is read-only in the promoted run. The remote agent cannot write back what it learns; discoveries surface in the leaf result instead. That is deliberate (ADR-0031) and it is what keeps leaf replay reproducible.
• promote reads slash commands from user scope only. A project-scope .claude/commands/ file does not travel unless HOME points at that project — which the sandbox pattern does anyway, but it will surprise someone promoting a real repo.
• Preflight blocks on facts and warns on heuristics. The secret scan refuses the upload only on structural credential formats (AWS key ids, PEM blocks, GitHub/Slack/OpenAI tokens); prose-shaped matches warn. Do not promise that promotion cannot leak a pasted password — it can, and it warns. A curated sandbox like this one yields zero findings, which is the state in which blocking preflight would become safe to restore (spec §6 D10, §11).
• --dry-run shows you the bundle without uploading it, which is the safe way to inspect what a real ~/.claude would ship before you ship it.

Reference: ../../deploy/knative/demo-promoted-workflow.sh, ADR-0030, ADR-0031.