Merge pull request #97 from runcycles/feature/v0.1.25.10-metrics-and-resilience

feat(metrics): v0.1.25.10 — custom Micrometer counters + Redis-disconnect test

Author: amavashev

AUDIT.md

@@ -1,6 +1,7 @@
 # Cycles Protocol v0.1.25 — Server Implementation Audit
 
-**Date:** 2026-04-14 (v0.1.25.9 — second-wave test additions: overdraft property, expire.lua conformance, admin-release race, multi-scope attribution, idempotency-cache expiry, clock-skew, metrics correctness, audit-log completeness),
+**Date:** 2026-04-14 (v0.1.25.10 — custom Micrometer counters for reserve/commit/release/extend/expired/events + overdraft, plus Redis-disconnect resilience test; dormant emitExpiredEvent key-prefix bug fixed as a side effect),
+2026-04-14 (v0.1.25.9 — second-wave test additions: overdraft property, expire.lua conformance, admin-release race, multi-scope attribution, idempotency-cache expiry, clock-skew, metrics correctness, audit-log completeness),
 2026-04-14 (property-based concurrent budget-exhaustion test + jqwik-spring lifecycle and tries-override follow-up fixes; passing green on Docker Desktop),
 2026-04-12 (spec endpoint-coverage report — parity with admin),
 2026-04-12 (spec tracking: pinned SHA → cycles-protocol@main for immediate drift detection),
@@ -13,6 +14,59 @@
 
 ---
 
+### 2026-04-14 — v0.1.25.10: custom business metrics + resilience test
+
+Addresses the largest remaining gap flagged in the v0.1.25.9 retrospective: the service emitted no domain-level metrics, only Spring Boot's generic `http.server.requests` timer. Operators answering "how many denials in the last 5 minutes by reason and tenant" could only infer it from HTTP status codes. This release wires domain counters through a new shared component and extends the existing Micrometer integration test to cover them.
+
+**New class: `CyclesMetrics` (`cycles-protocol-service-data/.../metrics/`)**
+Centralised Micrometer instrumentation. One `record*` method per operation, each mapping to a counter under the `cycles.*` namespace. Tag set prioritises operational signal (tenant, decision, reason, overage_policy, actor_type) while keeping cardinality bounded; the only high-card tag (`tenant`) is toggleable via `cycles.metrics.tenant-tag.enabled` for deployments with many thousands of tenants.
+
+**Counters emitted:**
+- `cycles.reservations.reserve` — every reserve outcome, tagged {tenant, decision, reason, overage_policy}. Idempotent replays record `reason=IDEMPOTENT_REPLAY` so an operator can tell real ALLOWs from cached replays.
+- `cycles.reservations.commit` — every commit outcome, same tag set.
+- `cycles.reservations.release` — every release, tagged {tenant, actor_type=tenant|admin_on_behalf_of, decision, reason}. The actor_type split was v0.1.25.8's dual-auth surface — now directly queryable.
+- `cycles.reservations.extend` — every extend outcome.
+- `cycles.reservations.expired` — one per actual ACTIVE→EXPIRED transition from the sweep. Does NOT bump for grace-period skips or already-finalised candidates.
+- `cycles.events` — every event outcome, same four-tag shape as reserve/commit.
+- `cycles.overdraft.incurred` — incremented whenever a commit or event actually accrued non-zero debt. Unit-free (the amount is in the balance store; this is "how often did we go into overdraft").
+
+**Modified:**
+- `RedisReservationRepository` — wraps each of `createReservation`, `commitReservation`, `releaseReservation`, `extendReservation`, `createEvent` so the counter emits on both success and exception paths. Method signatures for `commitReservation` and `releaseReservation` gained a `tenant` parameter (and `releaseReservation` an `actorType`); callers in `ReservationController` updated.
+- `ReservationExpiryService` — increments `cycles.reservations.expired` for each Lua-reported EXPIRED result (not per sweep candidate).
+- `cycles-protocol-service-data/pom.xml` — added `io.micrometer:micrometer-core`.
+
+**Dormant bug surfaced and fixed:** `ReservationExpiryService.emitExpiredEvent` was reading `reservation:<id>` instead of `reservation:res_<id>`. Because `jedis.hgetAll` on a missing key returns an empty map (not an error), the method silently no-op'd on every expiry in production — the `reservation.expired` event was never actually emitted. The new counter test exposed it immediately. Existing unit tests (`ReservationExpiryServiceTest`) used mocks keyed to the same wrong prefix so they were self-consistent but didn't catch the real path divergence; test mocks aligned to production in the same commit.
+
+**New integration test: `RedisDisconnectResilienceIntegrationTest`**
+Uses a dedicated Testcontainers Redis (not the shared one from `BaseIntegrationTest`, to avoid breaking parallel tests). Pauses the container mid-request via Docker pause, asserts the commit operation fails with a structured error (not a hang, not a silent 200), resumes the container, asserts a retry succeeds using the still-valid pre-outage reservation, and verifies the TTL index has no orphaned entry post-recovery. Guards the failure class this codebase's positioning claims to prevent (silent failures under a paused downstream).
+
+**Extended: `MetricsCorrectnessIntegrationTest`**
+Adds five nested classes covering every new counter. Each test seeds a clean state, reads the aggregate counter count, drives a known workload, and asserts the exact delta. Uses `Search.counters().stream().mapToDouble(...).sum()` rather than `Search.counter()` because multiple counters can match a partial tag filter (e.g. same tenant+decision but different overage_policy) and `.counter()` on ambiguous searches returns an arbitrary one — the aggregate is what the test needs.
+
+**New unit test: `CyclesMetricsTest`**
+10 tests in `cycles-protocol-service-data` covering every `record*` method's tag shape, null/blank normalisation to the `UNKNOWN` sentinel, and the `cycles.metrics.tenant-tag.enabled=false` path (verifies `tenant` tag is omitted for high-cardinality deployments).
+
+**Wire format:** Unchanged. Response bodies, Lua scripts, error codes, idempotency semantics all identical to v0.1.25.9.
+
+**Verification:**
+- `mvn -B verify --file cycles-protocol-service/pom.xml`: 133 api + 320 data = 453 tests, 0 failures. JaCoCo coverage met (≥95%). Spec coverage 9/9.
+- Property-tests profile unchanged, still passes.
+
+**Modified files:**
+- `cycles-protocol-service/pom.xml` — `<revision>` → `0.1.25.10`.
+- `cycles-protocol-service/cycles-protocol-service-data/pom.xml` — `micrometer-core` dep.
+- `cycles-protocol-service/cycles-protocol-service-data/src/main/java/io/runcycles/protocol/data/metrics/CyclesMetrics.java` — NEW.
+- `cycles-protocol-service/cycles-protocol-service-data/src/main/java/io/runcycles/protocol/data/repository/RedisReservationRepository.java` — instrumented; commit/release signatures gained `tenant`/`actorType`.
+- `cycles-protocol-service/cycles-protocol-service-data/src/main/java/io/runcycles/protocol/data/service/ReservationExpiryService.java` — counter emission + `res_` prefix fix.
+- `cycles-protocol-service/cycles-protocol-service-api/src/main/java/io/runcycles/protocol/api/controller/ReservationController.java` — pass `tenant` + `actorType` to repo.
+- `cycles-protocol-service/cycles-protocol-service-data/src/test/java/io/runcycles/protocol/data/metrics/CyclesMetricsTest.java` — NEW.
+- `cycles-protocol-service/cycles-protocol-service-api/src/test/java/io/runcycles/protocol/api/MetricsCorrectnessIntegrationTest.java` — extended with 8 new tests.
+- `cycles-protocol-service/cycles-protocol-service-api/src/test/java/io/runcycles/protocol/api/RedisDisconnectResilienceIntegrationTest.java` — NEW.
+- Updated mocks/test signatures in `ReservationControllerTest`, `RedisReservationCommitReleaseTest`, `RedisReservationEdgeCaseTest`, `ReservationExpiryServiceTest`, `BaseRedisReservationRepositoryTest`, `BalanceControllerTest`, `DecisionControllerTest`, `EventControllerTest` to match new signatures / provide `CyclesMetrics` mock bean.
+- `AUDIT.md`, `README.md` — this entry + version bump.
+
+---
+
 ### 2026-04-14 — v0.1.25.9: second-wave test additions
 
 Follow-up to v0.1.25.8's `BudgetExhaustionConcurrentPropertyTest`. A test-quality review flagged eight further high-leverage gaps (ordered by expected bug-catch density). This release lands all eight in one PR. Every addition reuses the existing `BaseIntegrationTest` + Testcontainers Redis harness and the `@Tag("property-tests")` convention for long-running jqwik suites.

README.md

@@ -117,7 +117,7 @@ mvn clean install
 ./build-all.sh
 ```
 
-The fat JAR is produced at `cycles-protocol-service-api/target/cycles-protocol-service-api-<version>.jar` (where `<version>` is the `revision` property in `cycles-protocol-service/pom.xml` — e.g. `0.1.25.9`).
+The fat JAR is produced at `cycles-protocol-service-api/target/cycles-protocol-service-api-<version>.jar` (where `<version>` is the `revision` property in `cycles-protocol-service/pom.xml` — e.g. `0.1.25.10`).
 
 ## Docker Deployment
 
@@ -136,7 +136,7 @@ Pre-built images are published to GitHub Container Registry on each release:
 
 ```
 ghcr.io/runcycles/cycles-server:latest
-ghcr.io/runcycles/cycles-server:<version>    # e.g. 0.1.25.9
+ghcr.io/runcycles/cycles-server:<version>    # e.g. 0.1.25.10
 ```
 
 ## Testing
@@ -232,6 +232,44 @@ GET /actuator/prometheus
 
 Exposes JVM, HTTP, and Spring Boot metrics in Prometheus format. Both endpoints are unauthenticated. Configure your Prometheus scrape target to `http://<host>:7878/actuator/prometheus`.
 
+#### Domain counters (v0.1.25.10+)
+
+In addition to Spring Boot's auto-emitted `http_server_requests_seconds`, the service exposes domain-level counters for every reservation-lifecycle operation. Operators can alert on denial rates, overdraft incidence, and per-tenant activity without having to reverse-engineer it from HTTP status codes.
+
+| Metric | Tags | Incremented on |
+|---|---|---|
+| `cycles_reservations_reserve_total` | `tenant`, `decision` (`ALLOW`/`ALLOW_WITH_CAPS`/`DENY`), `reason` (`OK`/`IDEMPOTENT_REPLAY`/error code), `overage_policy` | every `POST /v1/reservations` outcome |
+| `cycles_reservations_commit_total` | `tenant`, `decision` (`COMMITTED`/`DENY`), `reason`, `overage_policy` | every `POST /v1/reservations/{id}/commit` outcome |
+| `cycles_reservations_release_total` | `tenant`, `actor_type` (`tenant`/`admin_on_behalf_of`), `decision`, `reason` | every `POST /v1/reservations/{id}/release` outcome |
+| `cycles_reservations_extend_total` | `tenant`, `decision` (`ACTIVE`/`DENY`), `reason` | every `POST /v1/reservations/{id}/extend` outcome |
+| `cycles_reservations_expired_total` | `tenant` | once per reservation the expiry sweep transitions ACTIVE→EXPIRED (grace-period skips and already-finalised candidates do NOT increment) |
+| `cycles_events_total` | `tenant`, `decision` (`APPLIED`/`DENY`), `reason`, `overage_policy` | every `POST /v1/events` outcome |
+| `cycles_overdraft_incurred_total` | `tenant` | every commit or event that actually accrued non-zero debt |
+
+**Reason codes** use the same error enum the API returns — `BUDGET_EXCEEDED`, `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`, `BUDGET_FROZEN`, `BUDGET_CLOSED`, `RESERVATION_FINALIZED`, `RESERVATION_EXPIRED`, `IDEMPOTENCY_MISMATCH`, `UNIT_MISMATCH`, `INTERNAL_ERROR`, `NOT_FOUND`. Successful outcomes report `reason=OK`; idempotent replays report `reason=IDEMPOTENT_REPLAY`.
+
+**Tag-cardinality control.** The `tenant` tag is the only high-cardinality dimension. For deployments with thousands of tenants it can be turned off:
+
+```properties
+# application.properties
+cycles.metrics.tenant-tag.enabled=false   # default: true
+```
+
+When disabled, the `tenant` tag is omitted from every `cycles_*_total` series. Per-tenant drill-down is lost, but the time-series count drops to O(decision × reason × overage_policy) which is bounded and small.
+
+**Example queries.**
+
+```promql
+# Denial rate by reason, last 5 minutes:
+sum by (reason) (rate(cycles_reservations_reserve_total{decision="DENY"}[5m]))
+
+# Overdraft incidence per tenant:
+sum by (tenant) (rate(cycles_overdraft_incurred_total[5m]))
+
+# Admin-driven releases (compliance signal):
+sum(rate(cycles_reservations_release_total{actor_type="admin_on_behalf_of"}[1h]))
+```
+
 ## Documentation
 
 - [Cycles Documentation](https://runcycles.io) — full docs site

cycles-protocol-service/cycles-protocol-service-api/src/main/java/io/runcycles/protocol/api/controller/ReservationController.java

@@ -121,7 +121,7 @@ public ResponseEntity<CommitResponse> commit(
         validateIdempotencyHeader(idempotencyHeader, request.getIdempotencyKey());
         String tenant = repository.findReservationTenantById(reservationId);
         authorizeTenant(tenant);
-        CommitResponse response = repository.commitReservation(reservationId, request);
+        CommitResponse response = repository.commitReservation(reservationId, request, tenant);
         try {
             Actor actor = buildActor(httpRequest);
             // Spec: emit commit_overage when committed actual > estimated amount
@@ -166,7 +166,8 @@ public ResponseEntity<ReleaseResponse> release(
         validateIdempotencyHeader(idempotencyHeader, request.getIdempotencyKey());
         String tenant = repository.findReservationTenantById(reservationId);
         authorizeTenant(tenant);
-        ReleaseResponse response = repository.releaseReservation(reservationId, request);
+        String actorType = isAdminAuth() ? "admin_on_behalf_of" : "tenant";
+        ReleaseResponse response = repository.releaseReservation(reservationId, request, tenant, actorType);
 
         // v0.1.25.8: on admin-driven release, write an audit-log entry
         // to the shared Redis store. Entry surfaces in the governance