E8: decouple turn-concurrency measurement from the sandbox tier (spec amendment + driver arms)

[pdettori]

Summary

E8 currently measures turn concurrency coupled to the sandbox tier, and its parameterisation makes
the harness knee unreachable. Two arms are wanted instead:

• a capacity arm that measures the supervisor + worker tier's turn concurrency and throughput with
  the sandbox tier deliberately non-binding and non-competing;
• the existing realism arm, unchanged, which keeps the calibrated duty and answers "does it still
  hold up when tools cost something".

The sandbox tier's own capacity is P4's subject
(docs/specs/2026-09-09-p4-microvm-sandbox-design.md §7.2 E10, §7.3 E11), so E8 should stop competing
with it. This is a scope change against P6's spec as written, not just a driver tweak — §5.2
deliberately keeps the sandbox tier as a watched variable. The spec must be amended, not quietly
ignored, or the next run passes preconditions that no longer mean anything.

Blocked on #250 landing (→ #253). Do not open the PR for this before then.

Why the current methodology cannot answer the decoupled question

Three independent defects, from the calibrated run recorded in
docs/serverless-harness/2026-09-14-p6-vm-density-findings.md, which lives outside this repo — the per-rung tables and the derivations below are reproduced in the first comment, so this issue is self-contained:
knee_floor=32, saturated=no, bound=not-observed, duty 0.0735, 960/960 at c=32, p50 1530–1539 ms,
p95 1546 → 2457 ms, worker loop lag ~11 ms flat.

1. The tail we measured belongs to the sandbox tier. Three containers at ~78% busy at c=32
(32 × 0.0735 = 2.35 sandbox-equivalents of 3). The signature confirms it: p50 flat within 9 ms and loop
lag flat while p95 rises 1.59× is queueing at a shared downstream with few servers, not worker
saturation.

2. The sandbox work consumes the CPU under measurement. ~2.35 of 8 cores run git inside the tier
we want to exclude. Adding containers fixes queueing but not this.

3. The stub's latency, not the machine, sets the ladder's reach. ~1.42 s of the 1.537 s turn is the
stub's programmed wait (SH_STUB_TTFT_MS=300, SH_STUB_TOKEN_DELAY_MS=12, SH_STUB_OUTPUT_TOKENS=64).
An in-flight turn is mostly a sleeping promise costing single-digit ms of CPU, so 32 in-flight turns
load the worker tier at a few percent and the ladder tops out at the admission cap W×S=32 — a config
choice. Remove the sandboxes entirely and the result would still be bound=not-observed, because the
cap binds first. This is the defect that matters most and it is independent of the sandbox tier.

Corollary to fix in the reporting: "88% of ideal throughput" is computed against 32 ÷ 1.537 s, a
denominator that is mostly sleep. It measures how well the harness hides a fixed wait — a
concurrency-plumbing check, not a capacity ceiling. It must not be reported as one.

Part 1 — Spec amendment

docs/specs/2026-09-08-p6-vm-process-manager-design.md

1a. §5.2 (line 535) — split E8 into two arms

State the arms explicitly, each with its own purpose, preconditions and reporting rules:

Arm	Sandbox tier	Stub profile	Answers
capacity	non-binding and non-competing (trivial exec)	fast	how many concurrent turns the supervisor + workers push, and where the worker tier knees
realism	calibrated duty (e6-ocp)	300/12/64	whether the density figure survives a real tool cost

Record for each arm that its number is not the other's: the capacity arm is an upper bound on the
harness tier and not a deployable density; the realism arm is the deployable one and is sandbox-shaped
by construction.

§5.2 today reasons: "whether the sandbox tier holds the duty of its pinned basis (§2.3) under this load
is an open question — which is also why each run records which basis it used. Without these two metrics
a sandbox-bound run would be reported as a harness density limit." Keep that reasoning verbatim for the
realism arm. For the capacity arm, invert its conclusion: the sandbox tier is held out of the way on
purpose, and the guard becomes a utilisation ceiling rather than a pool floor.

Add to the §5.2 metric table: per-turn worker CPU (worker CPU-seconds ÷ turns served), attributing a
knee to "the worker tier is actually full" as distinct from "the stub is slow". This is the metric whose
absence makes the current record ambiguous.

1b. §5.4 (line 600) — duty calibration is per-arm

§5.4 currently requires the stub calibrated to a duty basis. Amend so the requirement binds the
realism arm, and the capacity arm is required to record its (near-zero) duty explicitly so the
arm can never be mistaken for a density claim. Keep "measure duty, never model it" for both — that
discipline is not what is being relaxed; see the wrong-model failure documented in
deploy/vm/tests/workload-duty.test.sh.

1c. §2.3 (line 97) — unchanged, but cite it from both arms

The three-row basis table stays as is. Note only that the capacity arm takes no row, and that
"no basis" is a distinct recorded value from "basis unrecorded" — the same distinction
check_sandbox_floor already makes between an unobserved pool and an empty one.

1d. Note the P4 seam

Cross-reference P4 §7.2/§7.3 so the division of labour is legible from both sides. P4 §7.3 already
guards this boundary from its side, listing "lease saturation one tier up — a harness-side refusal
misread as a VM-tier limit" as a per-rung metric. E8's capacity arm is the mirror of that.

Part 2 — Driver changes

2a. deploy/vm/prepare-workload.sh — a trivial-exec mode

Keep exactly one exec per turn so the relay hop, lease acquire/release, transport framing and exec
plumbing all still run per turn — that work is the harness's and must stay on the path. Only the
command's cost goes away.

turn_cmd() (line 173) builds the git chain from a repeat count. Add a mode that emits a trivial
command (e.g. true) and skips calibration entirely — there is nothing to converge. It must still print
an SH_STUB_TOOL_INPUT in the same shape (line 395) and still assert an exec was actually recorded, via
the existing counting path (turn_cmd_counting, line 245): a capacity arm whose tool call silently
never reaches a sandbox is the /turn-ran-tools-locally defect again, and the counting probe is what
catches it.

Record the resulting duty (~0.005) rather than suppressing it.

2b. deploy/vm/e8-density.sh — invert the precondition for the capacity arm

check_sandbox_floor() (line 142) asserts sandbox_pool_size >= ceil(W × S × duty) via
duty_basis_sandbox_floor (lib-vm.sh:402 → experiments/src/basis.ts sandboxFloor). For the
capacity arm that is the wrong direction. Add a ceiling check:

• keep the "pool never observed" refusal exactly as it is — it catches a turn that bypassed pool
  selection, and that hazard is identical in both arms (see the two documented failures in the comment
  at lines 120–134);
• replace the floor comparison with a utilisation ceiling: refuse the run if measured sandbox
  utilisation exceeds ~25%, computed from the recorded sandbox_cpu delta (lines 205, 284) against
  wall time × container count, so it is measured rather than predicted from a duty the arm does not
  have.

Both arms must fail loudly rather than proceed; a capacity arm running at 78% sandbox utilisation
produces the number this issue exists to stop producing.

2c. Two stub profiles

SH_STUB_TTFT_MS / SH_STUB_TOKEN_DELAY_MS / SH_STUB_OUTPUT_TOKENS are already env knobs, so this is
a parameter, not new machinery. Capacity arm: near-zero latency (suggest TTFT 10 ms, token delay 1 ms,
tokens unchanged) so turn duration approximates the harness's own cost and concurrency can climb to
where the event loop queues. Realism arm: 300/12/64 unchanged.

assert_stub_pinned (lib-vm.sh:279) already refuses a supervisor pointed at a different stub than the
one the driver is reading. Extend the same idea to the profile: the run's recorded profile must match
what /profile returns, so an arm cannot be labelled "capacity" while measuring the slow stub.

2d. Get the admission cap out of the way

Not the W/S sweep that was previously (correctly) vetoed. That veto held because the cap bound and
nothing else could saturate, so another sweep bought another honest not-observed. Here the cap stops
being the variable: set SH_WORKERS × SH_TURNS_PER_WORKER well past the expected knee (several hundred
admitted) and sweep offered concurrency upward until the knee appears. KAGENTI_SANDBOX_CAP
(deploy/vm/env/supervisor.env.example) must rise with it or leases become the new artificial cap — the
exact confusion 2b guards against.

2e. Per-turn worker CPU

Add the worker-side counterpart to sandbox_cpu: worker process CPU-seconds per rung, reported as
CPU-seconds ÷ turns served. Recorded per rung beside loop_lag_p99 / rss_bytes (lines 275–276) and
folded into the attribution block (line ~397) so a knee can be attributed to real worker CPU.

Part 3 — Two open questions to resolve while implementing

Loop lag may lack the resolution to be the capacity arm's headline diagnostic. It read ~11 ms flat
at every rung including c=1, where one turn is in flight and the tier is nearly idle. That looks like a
sampling or timer artefact rather than real delay. Since §5.2 makes it the metric that attributes a knee
to worker CPU, verify it against a deliberately CPU-starved worker before trusting it. Unverified
suspicion, not a finding — the sampling implementation was not inspected.

file_op_p95_ms is still NaN (e8-density.sh:277, acknowledged at line 459: no counter exists
anywhere). It matters less for the capacity arm, but it is the metric that would let the relay hop be
subtracted from a turn that keeps an exec on the path. Decide whether to build it now or record the gap
again.

Free check to do first, before any code moves

Read sandbox_cpu and rss_bytes out of the existing calibrated run record and confirm defect 2
quantitatively — that ~2.35 of 8 cores went to the sandbox tier. The driver already records both per
rung, so this needs no new run and no hardware. If the number disagrees with the arithmetic, this
issue's premise needs revisiting before the amendment is written.

Acceptance

• P6 §5.2 / §5.4 amended: two arms, per-arm preconditions, per-arm reporting rules, P4 cross-reference
• prepare-workload.sh trivial-exec mode, one exec per turn, exec presence still asserted, duty recorded
• check_sandbox_floor gains a measured utilisation ceiling for the capacity arm; the unobserved-pool refusal is untouched
• Stub profile recorded and pinned per arm; capacity profile fast
• Admission cap and lease cap raised past the expected knee; offered concurrency swept
• Per-turn worker CPU recorded per rung and used in attribution
• Reporting no longer presents "% of ideal throughput" as a capacity figure when the ideal is stub-sleep-bound
• The free check above run and its result recorded
• Guard tests under deploy/vm/tests/ extended: an arm mislabelled against its stub profile, and a capacity arm over the utilisation ceiling, must both be refused

Base the PR on feat/p6-experiments after #250 → #253 lands.

[pdettori]

Evidence, extracted inline

The findings record lives outside this repo, so the numbers the issue reasons from are reproduced here.
Three runs, all on c6i.2xlarge (8 vCPU / 16 GiB) with an off-box c6i.xlarge generator:
two at a ~0 ms tool call (feat/p6-experiments @ dbb0c46) and one at the calibrated duty
(@ 36411c4).

Defect 3 is already demonstrated, not predicted

The issue argues that removing the sandbox cost would still yield bound=not-observed because the
admission cap binds first. The zero-duty runs already show exactly that. With a tool call costing
~0 ms — the sandbox tier structurally present but load-free:

W=4, S=16 (64 admitted, 6 sandboxes)

c	ok/attempts	p50 ms	p95 ms	turns/s	worst loop lag
1	30/30	1092	1414	0.895	10.5 ms
8	240/240	1087	1420	7.025	11.3 ms
32	960/960	1092	1423	27.655	11.6 ms
64	1920/1920	1091	1420	54.377	11.0 ms

saturated=no. p95 moves 6 ms across a 64× concurrency range, throughput 95% of ideal, loop lag
flat. The W=4, S=8 zero-duty run behaves identically (32/32 admitted sustained, p95 1425 → 1421 ms).

So both halves of the decoupling are already evidenced: with the sandbox cost removed the ladder still
finds nothing, at 64 concurrent turns, because W × S is the top rung by construction. That is the
argument for raising the cap (item 2d) — it is the only change that makes a knee findable.

The per-turn harness cost, as a residual — why the capacity arm is worth building

From the calibrated run: mean turn 1537 ms, per-exec 113 ms, so 1424 ms is non-sandbox. The
stub's programmed latency for the two model responses is TTFT 300 + 64 × 12 = 1068 ms for the full
response plus TTFT-plus-a-few-tokens (~310–400 ms) for the short tool_use one — call it 1380–1460 ms.

The residual, i.e. everything the harness itself costs per turn, is therefore under ~45 ms and
plausibly under 10. The record does not state the tool_use response's token count, so this is a
bound rather than a measurement — but it is bounded well below 3% of the turn either way.

At the top rung's 18.3 turns/s that is at most ~0.8 cores of 8, and likely a fraction of one. The
worker tier was ~2–10% utilised in the run we published as a density result. That is the quantitative
form of defect 3, and it is why a fast stub (item 2c) is not a nicety: with the 300/12 profile the box
cannot be filled at any concurrency the cap permits.

Defects 1 and 2, from the calibrated run

E8_RESULT knee_floor=32 degrade_x=2 min_c=4 workers=4 s=8 saturated=no bound=not-observed

c	ok/attempts	p50 ms	p95 ms	turns/s	lease_sat	worst loop lag
1	30/30	1534	1546	0.648	0.33	11.1 ms
2	60/60	1537	1546	1.284	0.33	11.2 ms
4	120/120	1530	2000	2.539	0.33	11.1 ms
8	240/240	1531	1984	5.010	0.67	10.6 ms
16	480/480	1530	1983	9.637	0.33	11.3 ms
32	960/960	1539	2457	18.314	0.33	11.3 ms

Calibrated workload: repeat count 11, per-exec 113 ms (spread 4%, three sandboxes agreeing within
1 ms at 110/110/111), execs per turn 1.0 counted, mean turn 1537 ms, duty 0.0735 in the
0.061–0.079 e6-ocp band; an independent probe gave 0.0736.

The sandbox arithmetic the issue cites, verbatim from the record: "c=32 keeps 32 × 0.0735 = 2.35
sandbox-equivalents busy out of 3 — 78% utilisation", and duty_basis_sandbox_floor independently
derives K ≥ ceil(32 × 0.079) = 3. The record's own conclusion: "a taller rung at this pool size would
likely bind on the pool, which is exactly what the floor formula predicts." The capacity arm exists so
that binding never happens.

Note the p95 shape against the zero-duty runs: flat 1546 for c=1–2, then 1983–2000 from c=4, then 2457
at c=32 — where the zero-duty ladder was flat within 6 ms to c=64. The record attributes this to "the
sandbox tier doing real work and beginning to contend". That is the tail the capacity arm must not
inherit.

A worked example of the failure mode item 2b guards against

The same S=8 ladder run with the shipped KAGENTI_SANDBOX_CAP=4 instead of 13:

c	ok/attempts	rate	turns/s	result
1–8	all	1.00	0.9 → 7.0	healthy
16	360/480	0.75	10.388	below the 0.95 success floor
32	360/960	0.38	10.371	below the 0.95 success floor

knee_floor=8, bound=sandbox-pool. Exactly 360 successes at both rungs = 12 leases × 30 turns, with
throughput pinned at 10.4/s ≈ 12 ÷ 1.15 s. This run measured the lease pool and would have been
published as a VM density figure had the success-rate floor not caught it. It is the concrete reason
the >5% failure refusal is non-negotiable in both arms, and the reason the capacity arm needs a
utilisation ceiling rather than trusting a configured cap.

Loop lag: the resolution suspicion, now with more support

Across all three runs — two workloads with turn durations of ~1090 ms and ~1537 ms, and S values of
8 and 16 — worst-worker p99 loop lag stayed in 10.3–11.6 ms, including at c=1 where a single turn is
in flight. It does not move with concurrency, with turn duration, or with worker count.

A metric that reads ~11 ms on a nearly idle tier and the same ~11 ms at 64 concurrent turns is not
discriminating anything. Since P6 §5.2 makes it the attribution for worker CPU saturation, the capacity
arm cannot rely on it until it is verified against a deliberately CPU-starved worker. Still a suspicion —
the sampling implementation has not been inspected — but it now rests on three runs rather than one.

Implementation warnings for whoever picks this up

Straight from the record's caveats; each one can produce a wrong number quietly.

1. Do not compute utilisation from the record's cores on this box: 4 field. It describes the
   generator, not the target, and exists only to normalise contention_load1 (also generator-side).
   The target has 8. Item 2b's utilisation ceiling needs the target's core count from somewhere else.
2. lease_saturation cannot serve as the utilisation metric. It is leases per sandbox, saturating
   at KAGENTI_SANDBOX_CAP (13), so the recorded 0.33–1.33 is ~2–10% of pool capacity, not 33–133%. The
   attribution threshold compares it against 0.95, a scale mismatch that is an open methodology decision
   in its own right. Compute the ceiling from the sandbox_cpu delta instead, as item 2b says.
3. file_op_p95_ms is NaN for two different reasons across the runs — in the zero-duty runs
   partly because nothing performed file operations, and in the calibrated run because no counter exists
   anywhere in the tree. The relay tier is unattributed, not exonerated.
4. The capacity arm will run many more turns than anything published. Raising the cap to several
   hundred and sweeping past it multiplies turn count, and defect 13 (a Redis client opened per turn on
   three paths, ~3.5 connections/turn) killed all four workers ~13 minutes in at lower rates: ~10k
   turns produced 35,654 connections against maxclients 10000. It is fixed and verified (120 turns →
   14 connections, a second 120 → 2, connected_clients delta 0), but no run since has been longer
   than ~4 minutes and there is still no soak. A capacity arm that runs long enough to fill the box is
   also long enough to re-enter that territory.
5. vm_turn had no deadline, so when the workers died the stranded turns hung indefinitely and the
   driver stalled rather than erroring. A --max-time is now passed; at higher concurrency verify a
   single stuck turn still cannot hang the whole ladder.
6. Assert the exec actually reached a sandbox. Defect 1 was /turn never consulting the pool, so
   tool calls ran inside the worker process — the hands tier absent and its cost charged to the worker
   tier. In a trivial-exec arm that failure is nearly invisible, which is why item 2a keeps the counting
   probe.
7. Latency is not comparable across the two arms. The baseline moved 1068 → 1537 ms purely from the
   113 ms exec, and throughput linearity fell 95% → 88% for the same reason. Compare density, never
   latency.