Merge pull request #262 from runcycles/agent/benchmark-latency-context

Benchmark reserve fan-out from 1 to 200 clients

Author: amavashev

AUDIT.md

@@ -5,6 +5,29 @@
 
 ---
 
+### 2026-07-30 — controlled 1/10/50/200-client reserve evidence (no version bump)
+
+Benchmark-only correction and expansion. External review correctly noted that
+the 200-client homepage number lacked a same-harness low-concurrency
+denominator. The reserve fan-out now runs at 1, 10, 50, and 200 clients for
+both a shared tenant ledger and independent agent leaf ledgers. A system
+property can select one client level so every published cell is measured in a
+fresh Maven, Spring, and Testcontainers process. Warmup now performs at least
+50 requests, reaches every logical client, and uses a controlled ramp capped
+at 50 concurrent requests before resetting ledger state and starting the
+five-second measured window.
+
+Three fresh-process trials per shape and level completed 70,046 reservations
+with zero request errors and zero Redis ledger mismatches. Shared-ledger p99
+medians were 32.4ms at 1 client, 40.6ms at 10, 113.8ms at 50, and 1,325.9ms
+at 200-client saturation. Independent-ledger medians were 32.8ms, 42.6ms,
+117.0ms, and 929.1ms respectively. The earlier same-day 531.7ms shared and
+446.8ms isolated 200-client figures are superseded: review found they inherited
+warm state from preceding suite tests and their sequential warmups did not
+control wide-fan-out connection state. Production Java, Lua, Redis behavior,
+wire formats, dependencies, and release version remain unchanged.
+`[benchmark-skip]`
+
 ### 2026-07-30 — 200-client reserve fan-out benchmark (no version bump)
 
 Benchmark-only change. The concurrent suite now measures the synchronous

BENCHMARKS.md

@@ -5,31 +5,57 @@ Performance benchmark history across versions. All benchmarks use `CyclesProtoco
 
 Results are environment-dependent. Use for relative comparison across versions on the same hardware, not as absolute SLA targets. Latencies include the full HTTP round-trip: Spring Boot request handling, auth filter, JSON serialization, Redis EVALSHA, Lua execution, and response building.
 
-The concurrent suite also runs two sustained 200-client reserve fan-outs after
-50 warmups: one contending on a shared tenant budget and one sharded across
-independent agent-level leaf budgets. It records reserve p99, throughput, and
-error rate, and fails on any mismatch between successful reservations and the
-resulting Redis `reserved` totals.
+The concurrent suite also runs sustained reserve fan-outs at 1, 10, 50, and
+200 clients in two shapes: one contending on a shared tenant budget and one
+sharded across independent agent-level leaf budgets. It records reserve p99,
+throughput, and error rate, and fails on any mismatch between successful
+reservations and the resulting Redis `reserved` totals.
 
-### 200-client reserve fan-out — v0.1.25.59 (2026-07-30)
+### Fresh-process reserve fan-out — v0.1.25.59 (2026-07-30)
 
 Reference environment: AMD Ryzen Threadripper 3990X (64 cores), Java 21,
 Spring Boot 3.5.16, Redis 7 Alpine in Testcontainers, Docker Desktop 29.6.1,
-and localhost networking. The table reports the median of three fresh-process
-trials. Each trial performed 50 warmups, then measured one five-second window
-per shape.
-
-| Budget shape | Reserve p99 median (range) | Throughput median (range) | Errors | Ledger mismatches |
-|---|---:|---:|---:|---:|
-| Shared tenant budget | 531.7ms (524.8–532.3ms) | 1,328.6 reserves/s (1,295.0–1,354.6) | 0 | 0 |
-| Independent agent leaf budgets | 446.8ms (438.8–455.7ms) | 1,683.4 reserves/s (1,655.8–1,687.0) | 0 | 0 |
-
-Across all six measured windows, 45,022 successful reservations completed
-with zero request errors and zero Redis ledger mismatches. These localhost
-results are reference evidence, not an SLO; network topology, server sizing,
-Redis placement, and client connection pools affect tail latency.
-
-Run benchmarks: `mvn test -Pbenchmark` (requires Docker).
+and localhost networking. Every table cell reports the median and range of
+three trials that each started a fresh Maven, Spring, and Testcontainers
+process. A controlled warmup performs at least 50 requests and reaches every
+logical client, with warmup concurrency capped at 50. The ledger is then reset
+before one five-second measured window. This gives every client level the same
+startup and warmup treatment without opening 200 cold connections at once.
+
+| Clients | Budget shape | Reserve p99 median (range) | Throughput median (range) | Errors | Ledger mismatches |
+|---:|---|---:|---:|---:|---:|
+| 1 | Shared tenant budget | 32.4ms (30.8–34.5ms) | 49.6 reserves/s (46.6–58.0) | 0 | 0 |
+| 1 | Independent agent leaf budgets | 32.8ms (26.4–40.7ms) | 56.2 reserves/s (52.0–60.4) | 0 | 0 |
+| 10 | Shared tenant budget | 40.6ms (40.1–46.4ms) | 430.0 reserves/s (424.8–435.8) | 0 | 0 |
+| 10 | Independent agent leaf budgets | 42.6ms (37.9–43.2ms) | 426.0 reserves/s (426.0–434.6) | 0 | 0 |
+| 50 | Shared tenant budget | 113.8ms (100.1–126.5ms) | 998.8 reserves/s (994.8–1,022.4) | 0 | 0 |
+| 50 | Independent agent leaf budgets | 117.0ms (115.4–165.6ms) | 986.0 reserves/s (817.6–990.6) | 0 | 0 |
+| 200 | Shared tenant budget | 1,325.9ms (1,131.0–1,738.1ms) | 927.8 reserves/s (886.8–954.2) | 0 | 0 |
+| 200 | Independent agent leaf budgets | 929.1ms (651.8–1,040.6ms) | 833.0 reserves/s (832.6–864.6) | 0 | 0 |
+
+Across all 24 measured windows, 70,046 successful reservations completed with
+zero request errors and zero Redis ledger mismatches. Shared and isolated
+shapes are close through 50 clients. At 200 clients both shapes saturate the
+single application instance and its HTTP/Redis capacity, while the shared
+atomic ledger adds tail latency. These localhost results are reference
+evidence, not an SLO; network topology, server sizing, Redis placement, and
+client connection pools affect latency.
+
+This matrix supersedes the earlier same-day 200-only figures of 531.7ms shared
+and 446.8ms isolated. Review showed those cells inherited warmup from preceding
+suite tests, while their 50 sequential warmups did not control wide-fan-out
+connection state. They are retained in Git history but are not used as current
+evidence.
+
+Run all benchmark levels with `mvn test -Pbenchmark` (requires Docker). For a
+reproducible single cell, set `-Dbenchmark.fanout.clients=<level>` and select
+one fan-out test method, for example:
+
+```bash
+mvn -pl cycles-protocol-service-api -am test -Pbenchmark \
+  -Dtest='CyclesProtocolConcurrentBenchmarkTest#concurrentReserve_sharedBudget' \
+  -Dbenchmark.fanout.clients=10
+```
 
 ### Release coverage
 

benchmarks/README.md

@@ -44,10 +44,17 @@ https://raw.githubusercontent.com/runcycles/cycles-server/benchmark-data/benchma
 
 ## Metrics tracked
 
-Nine established regression signals are supplemented by eight fields from the
-200-client reserve fan-out. The Java benchmark itself fails above a 1% request
-error rate or on any Redis ledger mismatch; p99 remains informational in the
-cross-run regression gate because shared-runner tails are noisy.
+The Java suite measures reserve fan-out at 1, 10, 50, and 200 clients for both
+shared and isolated ledger shapes. Nine established regression signals are
+supplemented by eight machine-history fields from the 200-client saturation
+level. Lower-concurrency reference medians and ranges are curated in
+[`../BENCHMARKS.md`](../BENCHMARKS.md); the history pipeline keeps the widest
+level to avoid a breaking expansion of existing baseline records. History is
+a consistent full-suite regression signal and can inherit warm state from
+earlier tests; it is not interchangeable with the independently launched
+fresh-process reference cells. The Java benchmark itself fails above a 1%
+request error rate or on any Redis ledger mismatch; p99 remains informational
+in the cross-run regression gate because shared-runner tails are noisy.
 
 | Metric | Source test | Why |
 |---|---|---|
@@ -82,9 +89,12 @@ Each line is a standalone JSON object:
 
 The example fan-out values illustrate the record shape; they are not a
 published measurement. Each fan-out result measures reserve HTTP latency for
-five sustained seconds after 50 warmups. `shared` sends every client through
-one tenant budget; `isolated` removes that parent budget and assigns each
-client its own agent-level leaf budget.
+five sustained seconds after a controlled warmup of at least 50 requests that
+reaches every logical client, with warmup concurrency capped at 50. The ledger
+is reset before timing starts. `shared` sends every client through one tenant
+budget; `isolated` removes that parent budget and assigns each client its own
+agent-level leaf budget. Use `-Dbenchmark.fanout.clients=<level>` plus one
+fan-out test method to reproduce an individual fresh-process cell.
 
 ## Baseline format (`baseline.json`)
 

cycles-protocol-service/cycles-protocol-service-api/src/test/java/io/runcycles/protocol/api/CyclesProtocolConcurrentBenchmarkTest.java

@@ -19,13 +19,16 @@
  * multiple threads executing Reserve→Commit lifecycles simultaneously.
  *
  * Lifecycle tests ramp from 8 → 16 → 32 concurrent threads. Reserve-only
- * tests add a 200-client fan-out in two shapes: one shared budget (the
- * contention case) and 200 independent leaf budgets (the sharded case).
+ * tests ramp from 1 → 10 → 50 → 200 clients in two shapes: one shared
+ * budget (the contention case) and independent leaf budgets (the sharded
+ * case).
  *
  * Results are CI-environment sensitive — latency and throughput depend on
  * container resources, Redis container networking, and JVM warm-up.
  *
  * Run separately: mvn test -Pbenchmark
+ * Select fan-out levels for isolated fresh-process trials with
+ * -Dbenchmark.fanout.clients=1,10,50,200.
  */
 @DisplayName("Concurrent Load Benchmarks")
 @Tag("benchmark")
@@ -34,8 +37,12 @@
 class CyclesProtocolConcurrentBenchmarkTest extends BaseIntegrationTest {
 
     private static final int WARMUP_OPS = 50;
+    private static final int MAX_WARMUP_CONCURRENCY = 50;
     private static final long MEASURE_DURATION_MS = 5_000;
-    private static final int FANOUT_CLIENTS = 200;
+    private static final List<Integer> DEFAULT_FANOUT_CLIENT_LEVELS =
+            List.of(1, 10, 50, 200);
+    private static final List<Integer> FANOUT_CLIENT_LEVELS =
+            configuredFanoutClientLevels();
     private static final long FANOUT_ALLOCATION = 1_000_000_000_000L;
     private static final long FANOUT_RESERVE_AMOUNT = 100L;
     /** Max acceptable error rate (%) before failing the test */
@@ -111,16 +118,20 @@ void concurrentLifecycle_32threads() throws Exception {
 
     @Test
     @Order(4)
-    @DisplayName("Reserve fan-out at 200 clients on one shared budget")
-    void concurrentReserve_200clients_sharedBudget() throws Exception {
-        runConcurrentReserveFanout(false);
+    @DisplayName("Reserve fan-out from 1 to 200 clients on one shared budget")
+    void concurrentReserve_sharedBudget() throws Exception {
+        for (int clients : FANOUT_CLIENT_LEVELS) {
+            runConcurrentReserveFanout(false, clients);
+        }
     }
 
     @Test
     @Order(5)
-    @DisplayName("Reserve fan-out at 200 clients on independent leaf budgets")
-    void concurrentReserve_200clients_independentBudgets() throws Exception {
-        runConcurrentReserveFanout(true);
+    @DisplayName("Reserve fan-out from 1 to 200 clients on independent leaf budgets")
+    void concurrentReserve_independentBudgets() throws Exception {
+        for (int clients : FANOUT_CLIENT_LEVELS) {
+            runConcurrentReserveFanout(true, clients);
+        }
     }
 
     private void runConcurrentLifecycle(int threadCount) throws Exception {
@@ -227,30 +238,27 @@ private void runConcurrentLifecycle(int threadCount) throws Exception {
         }
     }
 
-    private void runConcurrentReserveFanout(boolean isolated) throws Exception {
-        prepareFanoutBudgets(isolated);
+    private void runConcurrentReserveFanout(
+            boolean isolated,
+            int clientCount) throws Exception {
+        prepareFanoutBudgets(isolated, clientCount);
 
-        // Prime JIT, auth cache, HTTP connection management, and EVALSHA before
-        // resetting the measured ledger state.
-        for (int i = 0; i < WARMUP_OPS; i++) {
-            ResponseEntity<Map> response = post(
-                    "/v1/reservations",
-                    API_KEY_SECRET_A,
-                    fanoutReservationBody(isolated, i % FANOUT_CLIENTS));
-            assertThat(response.getStatusCode().value()).isEqualTo(200);
-            assertThat(response.getBody()).containsKey("reservation_id");
-        }
-        prepareFanoutBudgets(isolated);
+        // Prime JIT, auth cache, HTTP connection management, and EVALSHA
+        // before resetting the ledger. The bounded ramp exercises every
+        // logical client without turning 200 simultaneous cold connections
+        // into a warmup-only transport failure.
+        warmUpFanout(isolated, clientCount);
+        prepareFanoutBudgets(isolated, clientCount);
 
-        ExecutorService executor = Executors.newFixedThreadPool(FANOUT_CLIENTS);
+        ExecutorService executor = Executors.newFixedThreadPool(clientCount);
         try {
             ConcurrentLinkedQueue<Long> timings = new ConcurrentLinkedQueue<>();
             AtomicInteger errorCount = new AtomicInteger();
-            AtomicLongArray successesByClient = new AtomicLongArray(FANOUT_CLIENTS);
+            AtomicLongArray successesByClient = new AtomicLongArray(clientCount);
             CountDownLatch startLatch = new CountDownLatch(1);
             AtomicBoolean running = new AtomicBoolean(true);
 
-            for (int client = 0; client < FANOUT_CLIENTS; client++) {
+            for (int client = 0; client < clientCount; client++) {
                 int clientIndex = client;
                 executor.submit(() -> {
                     try {
@@ -307,13 +315,13 @@ private void runConcurrentReserveFanout(boolean isolated) throws Exception {
                     : 0.0;
             double opsPerSec = totalOps / (MEASURE_DURATION_MS / 1000.0);
             long ledgerMismatches = countFanoutLedgerMismatches(
-                    isolated, successesByClient, totalOps);
+                    isolated, successesByClient, totalOps, clientCount);
 
             FanoutResult result;
             if (totalOps > 0) {
                 result = new FanoutResult(
                         isolated ? "isolated" : "shared",
-                        FANOUT_CLIENTS,
+                        clientCount,
                         totalOps,
                         opsPerSec,
                         p(sorted, 50),
@@ -327,7 +335,7 @@ private void runConcurrentReserveFanout(boolean isolated) throws Exception {
             } else {
                 result = new FanoutResult(
                         isolated ? "isolated" : "shared",
-                        FANOUT_CLIENTS,
+                        clientCount,
                         0, 0, 0, 0, 0, 0, 0,
                         errors, errorRate, ledgerMismatches);
             }
@@ -346,27 +354,61 @@ private void runConcurrentReserveFanout(boolean isolated) throws Exception {
                     result.errorRatePercent, result.ledgerMismatches);
 
             assertThat(errorRate)
-                    .as("Reserve error rate for %s 200-client fan-out", result.shape)
+                    .as("Reserve error rate for %s %d-client fan-out",
+                            result.shape, result.clients)
                     .isLessThan(MAX_ERROR_RATE_PERCENT);
             assertThat(totalOps)
-                    .as("Successful reserves for %s 200-client fan-out", result.shape)
+                    .as("Successful reserves for %s %d-client fan-out",
+                            result.shape, result.clients)
                     .isGreaterThan(0);
             assertThat(ledgerMismatches)
-                    .as("Ledger mismatches for %s 200-client fan-out", result.shape)
+                    .as("Ledger mismatches for %s %d-client fan-out",
+                            result.shape, result.clients)
                     .isZero();
         } finally {
             executor.shutdownNow();
         }
     }
 
-    private void prepareFanoutBudgets(boolean isolated) {
+    private void warmUpFanout(boolean isolated, int clientCount)
+            throws Exception {
+        int warmupOps = Math.max(WARMUP_OPS, clientCount);
+        int warmupConcurrency =
+                Math.min(clientCount, MAX_WARMUP_CONCURRENCY);
+        ExecutorService executor =
+                Executors.newFixedThreadPool(warmupConcurrency);
+        try {
+            List<Future<ResponseEntity<Map>>> responses =
+                    new ArrayList<>(warmupOps);
+            for (int operation = 0; operation < warmupOps; operation++) {
+                int clientIndex = operation % clientCount;
+                responses.add(executor.submit(() -> post(
+                        "/v1/reservations",
+                        API_KEY_SECRET_A,
+                        fanoutReservationBody(isolated, clientIndex))));
+            }
+            executor.shutdown();
+            for (Future<ResponseEntity<Map>> future : responses) {
+                ResponseEntity<Map> response = future.get(30, TimeUnit.SECONDS);
+                assertThat(response.getStatusCode().value()).isEqualTo(200);
+                assertThat(response.getBody()).containsKey("reservation_id");
+            }
+            assertThat(executor.awaitTermination(30, TimeUnit.SECONDS))
+                    .as("Fan-out warmup workers terminated")
+                    .isTrue();
+        } finally {
+            executor.shutdownNow();
+        }
+    }
+
+    private void prepareFanoutBudgets(boolean isolated, int clientCount) {
         try (var jedis = jedisPool.getResource()) {
             if (!isolated) {
                 seedBudget(jedis, TENANT_A, "TOKENS", FANOUT_ALLOCATION);
                 return;
             }
             jedis.del("budget:tenant:" + TENANT_A + ":TOKENS");
-            for (int client = 0; client < FANOUT_CLIENTS; client++) {
+            for (int client = 0; client < clientCount; client++) {
                 seedScopeBudget(
                         jedis,
                         fanoutScope(client),
@@ -391,7 +433,8 @@ private Map<String, Object> fanoutReservationBody(boolean isolated, int client)
     private long countFanoutLedgerMismatches(
             boolean isolated,
             AtomicLongArray successesByClient,
-            int totalOps) {
+            int totalOps,
+            int clientCount) {
         try (var jedis = jedisPool.getResource()) {
             if (!isolated) {
                 long actual = Long.parseLong(jedis.hget(
@@ -400,7 +443,7 @@ private long countFanoutLedgerMismatches(
                 return actual == totalOps * FANOUT_RESERVE_AMOUNT ? 0 : 1;
             }
             long mismatches = 0;
-            for (int client = 0; client < FANOUT_CLIENTS; client++) {
+            for (int client = 0; client < clientCount; client++) {
                 long actual = Long.parseLong(jedis.hget(
                         "budget:" + fanoutScope(client) + ":TOKENS",
                         "reserved"));
@@ -422,6 +465,28 @@ private static String fanoutScope(int client) {
         return "tenant:" + TENANT_A + "/agent:" + fanoutAgent(client);
     }
 
+    private static List<Integer> configuredFanoutClientLevels() {
+        String configured = System.getProperty("benchmark.fanout.clients");
+        if (configured == null || configured.isBlank()) {
+            return DEFAULT_FANOUT_CLIENT_LEVELS;
+        }
+        try {
+            List<Integer> levels = Arrays.stream(configured.split(","))
+                    .map(String::trim)
+                    .map(Integer::parseInt)
+                    .toList();
+            if (levels.isEmpty() || levels.stream().anyMatch(level -> level <= 0)) {
+                throw new IllegalArgumentException(
+                        "benchmark.fanout.clients values must be positive");
+            }
+            return levels;
+        } catch (NumberFormatException e) {
+            throw new IllegalArgumentException(
+                    "benchmark.fanout.clients must be a comma-separated list of integers",
+                    e);
+        }
+    }
+
     private static long p(long[] sorted, int percentile) {
         return sorted[percentileIndex(sorted.length, percentile)];
     }