Merge pull request #107 from runcycles/feat/trace-id-correlation-v0.1.25.14

feat(trace-id): W3C Trace Context cross-surface correlation (v0.1.25.14)

Author: amavashev

AUDIT.md

@@ -1,6 +1,7 @@
 # Cycles Protocol v0.1.25 — Server Implementation Audit
 
-**Date:** 2026-04-16 (v0.1.25.13 — hydration cap + enum wire annotations on the sorted `GET /v1/reservations` path; `SORTED_HYDRATE_CAP=2000` guard on the in-memory sort hydration with WARN-on-cap, matches admin plane's v0.1.25.24 pattern; `@JsonValue`/`@JsonCreator fromWire` on `ReservationSortBy` + `SortDirection` to mirror admin's `SortSpec`/`SortDirection` contract),
+**Date:** 2026-04-18 (v0.1.25.14 — trace_id (W3C Trace Context) cross-surface correlation per cycles-protocol revision 2026-04-18; new `TraceContextFilter` extracts `traceparent` or `X-Cycles-Trace-Id` from inbound requests or generates a fresh 128-bit id, echoes `X-Cycles-Trace-Id` on every response, populates `trace_id` on `ErrorResponse` / `Event` / `WebhookDelivery` / `AuditLogEntry`),
+2026-04-16 (v0.1.25.13 — hydration cap + enum wire annotations on the sorted `GET /v1/reservations` path; `SORTED_HYDRATE_CAP=2000` guard on the in-memory sort hydration with WARN-on-cap, matches admin plane's v0.1.25.24 pattern; `@JsonValue`/`@JsonCreator fromWire` on `ReservationSortBy` + `SortDirection` to mirror admin's `SortSpec`/`SortDirection` contract),
 2026-04-16 (v0.1.25.12 — `sort_by` + `sort_dir` on `GET /v1/reservations` per cycles-protocol spec revision 2026-04-16; 7-value sort enum, opaque cursor binds `(sort_by, sort_dir, filters)` tuple, legacy SCAN-cursor path preserved when both params omitted),
 2026-04-14 (automated performance regression detection — nightly trend + release gate, no version bump),
 2026-04-14 (nightly soak test — long-duration stability coverage, no version bump),
@@ -19,6 +20,70 @@
 
 ---
 
+### 2026-04-18 — v0.1.25.14: trace_id cross-surface correlation (W3C Trace Context)
+
+Implements the `CORRELATION AND TRACING` normative section added to `cycles-protocol-v0.yaml` in spec revision 2026-04-18 (commit `8d65959`). Introduces a third correlation identifier — `trace_id` — that is W3C Trace Context-compatible (OpenTelemetry-native) and links every HTTP request to its `ErrorResponse`, audit-log entry, emitted events, and outbound webhook deliveries under one logical-operation grain.
+
+**Inbound header extraction** (new `TraceContextFilter`, `@Order(0)`, runs before `RequestIdFilter`):
+
+1. `traceparent` header — parsed as W3C Trace Context version 00 (`^00-[0-9a-f]{32}-[0-9a-f]{16}-[0-9a-f]{2}$`). Must be non-all-zero trace-id and span-id per W3C §3.2.2.3.
+2. `X-Cycles-Trace-Id` header — parsed as flat 32 lowercase hex; must be non-all-zero. Used when `traceparent` is absent or malformed.
+3. Server generates — `SecureRandom` 16 bytes → 32 lowercase hex; re-rolled if all-zero.
+
+Malformed headers are silently ignored (spec: MUST NOT reject). When both valid headers are present and disagree, `traceparent` wins per OpenTelemetry interop precedence.
+
+Inbound trace-flags preservation on outbound webhooks is deferred to the sibling `cycles-server-events` PR — that PR needs to add a `trace_flags` field to the `WebhookDelivery` schema (or accept the v0 default of `01`/sampled on every outbound delivery). Not a wire-compliance gap for this PR: only the outbound webhook `traceparent` is affected, and it's emitted by the events service, not this server.
+
+**Outbound propagation:**
+
+- Every response (2xx/4xx/5xx) echoes `X-Cycles-Trace-Id` header.
+- `ErrorResponse.trace_id` populated across all five exception-handler paths (`CyclesProtocolException`, `MethodArgumentNotValidException`, `ConstraintViolationException`, `HttpMessageNotReadableException`, generic 500).
+- `Event.trace_id` populated for every event emitted via `EventEmitterService.emit(...)` and `emitBalanceEvents(...)` — RESERVATION_DENIED, RESERVATION_COMMIT_OVERAGE, BUDGET_EXHAUSTED, BUDGET_OVER_LIMIT_ENTERED, BUDGET_DEBT_INCURRED, and RESERVATION_EXPIRED.
+- `WebhookDelivery.trace_id` copied from `Event.trace_id` in `EventEmitterRepository.createDelivery` so the events service can lift it into outbound headers without re-parsing the event body.
+- `AuditLogEntry.trace_id` populated on admin-driven releases (`ReservationController.release`).
+- `ReservationExpiryService` mints a fresh trace_id per sweep batch so all reservation.expired events in one sweep correlate to each other. Per spec, `request_id` remains null on sweeper-generated events (no originating HTTP request).
+
+**SLF4J MDC:** filter sets `traceId` key on entry and removes it in `finally` so every log line produced during the request carries the trace_id automatically. Existing `requestId` MDC key behavior unchanged.
+
+**Contract impact:** purely additive. `trace_id` is an OPTIONAL property on `ErrorResponse` (schema preserves `additionalProperties: false` via a declared property). Response header is additive; clients that don't read it are unaffected. Inbound `traceparent` / `X-Cycles-Trace-Id` headers are additive; clients that don't send them are unaffected.
+
+**Admin spec v0.1.25.28 alignment (WebhookDelivery):** also adds the two companion fields defined by governance-admin spec revision 2026-04-18 on the shared `WebhookDelivery` schema:
+
+- `trace_flags` (`^[0-9a-f]{2}$`) — W3C Trace Context trace-flags byte. Preserves the inbound sampling decision when the originating request carried a valid `traceparent`; defaults to `01` (sampled) when the trace was derived from `X-Cycles-Trace-Id` or server-generated.
+- `traceparent_inbound_valid` (boolean) — whether the originating HTTP request presented a valid inbound W3C `traceparent`. Consumed by the `cycles-server-events` sidecar to decide whether to preserve `trace_flags` on the outbound delivery or default to `01`.
+
+Both fields are threaded from the `TraceContextFilter` through an internal `TraceContext` record (`cycles-protocol-service-data/.../data/util/TraceContext.java`), which also collapses the per-request correlation trio (`trace_id` + `trace_flags` + `traceparent_inbound_valid`) into a single positional parameter on `EventEmitterService.emit(...)` and `emitBalanceEvents(...)` — addressing the param-sprawl concern flagged in the simplify review without adding 2 extra positional args.
+
+`Event.java` carries the two companion fields as `@JsonIgnore` transient properties so they travel with the `Event` object through the async emit path without bleeding into the `Event` wire contract (the spec only declares these on `WebhookDelivery`).
+
+**Out of scope (sibling PRs):**
+
+- `cycles-server-events` (separate repo): outbound webhook `X-Cycles-Trace-Id` and `traceparent` headers. Reads `trace_id`, `trace_flags`, and `traceparent_inbound_valid` directly off the `WebhookDelivery` row from Redis. This PR prepares the row fully so the events-service PR is a straight read-and-forward.
+- `cycles-server-admin`: admin-plane `AuditLogEntry.trace_id` surfacing and new `listEvents`/`listAuditLogs` `trace_id`/`request_id` filter query parameters.
+
+**Files changed:**
+
+- **NEW** `cycles-protocol-service-api/src/main/java/io/runcycles/protocol/api/filter/TraceContextFilter.java` — the filter, regex-based traceparent parsing.
+- **NEW** `cycles-protocol-service-data/src/main/java/io/runcycles/protocol/data/util/TraceIdGenerator.java` — shared pure-function helper (`SecureRandom` → 32-hex with all-zero re-roll) used by the filter fallback path and by `ReservationExpiryService`.
+- **NEW** `cycles-protocol-service-data/src/main/java/io/runcycles/protocol/data/util/TraceContext.java` — record bundling `trace_id` + `trace_flags` + `traceparent_inbound_valid`; threaded through `EventEmitterService.emit(...)` as a single positional param instead of three parallel Strings/Booleans.
+- `cycles-protocol-service-model/src/main/java/io/runcycles/protocol/model/ErrorResponse.java` — `trace_id` field.
+- `cycles-protocol-service-model/src/main/java/io/runcycles/protocol/model/event/Event.java` — `trace_id` field.
+- `cycles-protocol-service-model/src/main/java/io/runcycles/protocol/model/webhook/WebhookDelivery.java` — `trace_id` + `trace_flags` + `traceparent_inbound_valid` fields (admin spec v0.1.25.28).
+- `cycles-protocol-service-model/src/main/java/io/runcycles/protocol/model/audit/AuditLogEntry.java` — `trace_id` field.
+- `cycles-protocol-service-api/src/main/java/io/runcycles/protocol/api/controller/BaseController.java` — new `resolveRequestId` + `resolveTraceId` helpers.
+- `cycles-protocol-service-api/src/main/java/io/runcycles/protocol/api/controller/{ReservationController,DecisionController,EventController}.java` — pass `resolveRequestId(httpRequest)` and `resolveTraceId(httpRequest)` into every event-emission call site.
+- `cycles-protocol-service-api/src/main/java/io/runcycles/protocol/api/exception/GlobalExceptionHandler.java` — populate `trace_id` on every `ErrorResponse.builder()` across all five `@ExceptionHandler` paths.
+- `cycles-protocol-service-data/src/main/java/io/runcycles/protocol/data/service/EventEmitterService.java` — `traceId` parameter appended to `emit(...)` and the full `emitBalanceEvents(...)` signature; three prior `emitBalanceEvents(...)` overloads retained as delegating wrappers for source compatibility.
+- `cycles-protocol-service-data/src/main/java/io/runcycles/protocol/data/service/ReservationExpiryService.java` — batch-scope `TraceIdGenerator.generate()` per sweep, threaded into `emitExpiredEvent`.
+- `cycles-protocol-service-data/src/main/java/io/runcycles/protocol/data/repository/EventEmitterRepository.java` — `createDelivery` copies `event.getTraceId()` onto the `WebhookDelivery`.
+
+**Tests:**
+
+- **NEW** `TraceContextFilterTest` (13 cases) — traceparent valid/malformed/all-zero, X-Cycles-Trace-Id valid/malformed/all-zero/uppercase, both-present disagreement, fallthrough generation, response header echo, request attribute set, `currentTraceId(null)` handling, non-v00 version rejection, trace-flags=00 round-trip.
+- `GlobalExceptionHandlerTest` — `trace_id` assertions across all five handler paths; null-trace when filter didn't run.
+
+---
+
 ### 2026-04-16 — v0.1.25.13: hydration cap + enum wire annotations on the sorted list path
 
 Closes two follow-up gaps surfaced by the three-step review of v0.1.25.12 against the admin-plane implementation of the same feature in `cycles-server-admin` v0.1.25.24:

BENCHMARKS.md

@@ -29,6 +29,15 @@ running them would only measure environmental noise. Skipped releases:
   unchanged and all existing benchmarks exercise the legacy path.
   Benchmarks for the sorted path are worth adding once real tenant
   populations exercise the O(N) full-SCAN; see OPERATIONS.md.
+- **v0.1.25.13** — hydration cap + enum wire annotations on the sorted
+  list path. Write-path unchanged.
+- **v0.1.25.14** — trace_id (W3C Trace Context) correlation. The new
+  `TraceContextFilter` adds two regex matches + one secure-random
+  read per request (~microseconds, no Redis / no locking / no
+  allocation beyond a 32-char String). Request-path code only;
+  Lua / reserve / commit / release / extend hot paths untouched.
+  Benchmarks deliberately skipped — they would only measure
+  environmental noise. [benchmark-skip]
 
 Last benchmarked release: **v0.1.25.7**.
 

CHANGELOG.md

@@ -14,6 +14,45 @@ changes to request/response bodies or Lua-script semantics would require a
 minor bump. "Internal signature changes" (e.g. Java method parameters) are
 called out but are not breaking to API clients.
 
+## [0.1.25.14] — 2026-04-18
+
+### Added
+
+- W3C Trace Context correlation per `cycles-protocol-v0.yaml` revision
+  2026-04-18. Every response now carries an `X-Cycles-Trace-Id`
+  header. The server accepts a `traceparent` (W3C version 00) or
+  `X-Cycles-Trace-Id` header on inbound requests and echoes back the
+  same trace_id; when neither is present it generates a fresh 128-bit
+  id (32 lowercase hex). Malformed headers are silently ignored; the
+  server never rejects a request for a bad correlation header.
+- `trace_id` field on `ErrorResponse`, `Event`, `WebhookDelivery`,
+  and `AuditLogEntry` bodies. Optional for wire back-compat; conformant
+  servers populate it on every payload causally downstream of the
+  request.
+- `trace_flags` (`^[0-9a-f]{2}$`) and `traceparent_inbound_valid`
+  (boolean) on `WebhookDelivery` per governance-admin spec v0.1.25.28.
+  These preserve the upstream W3C sampling decision so the events
+  sidecar can reconstruct an outbound `traceparent` with the correct
+  trace-flags byte instead of defaulting to `01`.
+- SLF4J MDC now carries `traceId` alongside `requestId` for every
+  request — log aggregators can group by trace_id to see all lines
+  produced during a single logical operation.
+- `ReservationExpiryService` mints a fresh trace_id per sweep batch
+  so `reservation.expired` events emitted in the same sweep correlate
+  to each other.
+
+### Internal
+
+- New `TraceContextFilter` (`@Order(0)`) runs before `RequestIdFilter`
+  and sets the `cyclesTraceId` request attribute for downstream code.
+- `EventEmitterService.emit(...)` gains a final `String traceId`
+  parameter. The full-arity `emitBalanceEvents(...)` signature
+  likewise. Three prior overloads kept as delegating wrappers
+  (`traceId = null`) for source compatibility with existing tests.
+- `BaseController` exposes protected `resolveRequestId` and
+  `resolveTraceId` helpers that controllers use to thread the ids
+  into event-emission and audit-log calls.
+
 ## [0.1.25.13] — 2026-04-16
 
 ### Fixed

OPERATIONS.md

@@ -15,7 +15,8 @@ section of [`README.md`](README.md) first.
 3. [SLO definitions](#slo-definitions)
 4. [Dashboards](#dashboards)
 5. [Incident playbook](#incident-playbook)
-6. [Configuration tuning](#configuration-tuning)
+6. [Correlation and tracing](#correlation-and-tracing)
+7. [Configuration tuning](#configuration-tuning)
 
 ---
 
@@ -351,6 +352,55 @@ reservations. Either:
 
 ---
 
+## Correlation and tracing
+
+Every response carries two correlation identifiers (v0.1.25.14+):
+
+| Header | Shape | Grain |
+|---|---|---|
+| `X-Request-Id` | UUIDv4 | One HTTP request |
+| `X-Cycles-Trace-Id` | 32 lowercase hex (W3C Trace Context) | One logical operation |
+
+### Client-supplied trace_id
+
+Clients can set the trace_id themselves by sending either:
+
+- `traceparent: 00-{trace_id}-{span_id}-{trace_flags}` — standard W3C Trace Context header (OpenTelemetry-native). Takes precedence.
+- `X-Cycles-Trace-Id: {trace_id}` — flat 32-hex, for clients not using OpenTelemetry.
+
+If both are present and disagree, `traceparent` wins. Malformed values are silently ignored (the server never 400s for a bad correlation header).
+
+### Propagation
+
+The trace_id is attached to:
+
+- The response (`X-Cycles-Trace-Id` header).
+- `ErrorResponse` bodies (`trace_id` field).
+- Emitted events (`trace_id` field in the event body and on the `WebhookDelivery` row).
+- Audit-log entries for admin-driven releases (`trace_id` field in `AuditLogEntry`).
+
+Sweeper-generated events (`reservation.expired`) get a fresh trace_id per sweep batch — they have no originating HTTP request.
+
+### Log correlation
+
+Every log line produced during a request carries both `requestId` and `traceId` MDC keys. Grep a trace_id across all log lines to see everything that happened in one logical operation:
+
+```bash
+grep '4bf92f3577b34da6a3ce929d0e0e4736' server.log
+```
+
+### Webhook correlation
+
+Outbound webhook deliveries (from `cycles-server-events`) carry the same `trace_id` in the `X-Cycles-Trace-Id` and `traceparent` headers, plus the `trace_id` field in the event body. Subscribers can correlate their downstream processing back to the originating Cycles request. This server persists three correlation fields on the `WebhookDelivery` Redis row so the events sidecar can lift them straight into the outbound HTTP request:
+
+- `trace_id` — the trace identifier.
+- `trace_flags` — the W3C trace-flags byte. When an inbound request carried a valid `traceparent`, this is the inbound byte (preserves sampling decision); otherwise `01`.
+- `traceparent_inbound_valid` — boolean. Tells the sidecar whether to preserve the above `trace_flags` (`true`) or default to `01` (`false`).
+
+Wire-up of the outbound HTTP headers themselves happens in the `cycles-server-events` repo.
+
+---
+
 ## Configuration tuning
 
 All configurable via `application.properties` or environment variables.

README.md

@@ -117,7 +117,7 @@ mvn clean install
 ./build-all.sh
 ```
 
-The fat JAR is produced at `cycles-protocol-service-api/target/cycles-protocol-service-api-<version>.jar` (where `<version>` is the `revision` property in `cycles-protocol-service/pom.xml` — e.g. `0.1.25.13`).
+The fat JAR is produced at `cycles-protocol-service-api/target/cycles-protocol-service-api-<version>.jar` (where `<version>` is the `revision` property in `cycles-protocol-service/pom.xml` — e.g. `0.1.25.14`).
 
 ## Docker Deployment
 
@@ -136,7 +136,7 @@ Pre-built images are published to GitHub Container Registry on each release:
 
 ```
 ghcr.io/runcycles/cycles-server:latest
-ghcr.io/runcycles/cycles-server:<version>    # e.g. 0.1.25.13
+ghcr.io/runcycles/cycles-server:<version>    # e.g. 0.1.25.14
 ```
 
 ## Testing

cycles-protocol-service/README.md

@@ -227,13 +227,14 @@ For operations by resource ID (e.g. `GET /v1/reservations/{id}`), the server fir
 
 | Order | Filter | Responsibility |
 |---|---|---|
+| 0 | `TraceContextFilter` | Extracts `traceparent` / `X-Cycles-Trace-Id` (or generates) → request attribute, MDC, `X-Cycles-Trace-Id` response header |
 | 1 | `RequestIdFilter` | Generates `X-Request-Id` UUID, stored as request attribute and response header |
 | 2 | `RateLimitHeaderFilter` | Adds `X-RateLimit-Remaining` / `X-RateLimit-Reset` headers (sentinel values in v0) |
 | 3 | `ApiKeyAuthenticationFilter` | Validates key, populates `SecurityContext`, sets `X-Cycles-Tenant` header |
 
 Public paths (Swagger UI, actuator health, etc.) bypass authentication.
 
-All responses include an `X-Request-Id` header. All timestamps are Unix milliseconds (int64).
+All responses include `X-Request-Id` and `X-Cycles-Trace-Id` headers. All timestamps are Unix milliseconds (int64). `trace_id` is W3C Trace Context-compatible (32 lowercase hex); clients may pre-supply it via `traceparent` (OpenTelemetry-native) or `X-Cycles-Trace-Id`. See `OPERATIONS.md` → "Correlation and tracing" for the full contract.
 
 ---
 
@@ -688,6 +689,7 @@ cycles-protocol-service/
         │   ├── ApiKeyAuthentication.java        # SecurityContext token
         │   └── SecurityConfig.java              # Spring Security config
         ├── filter/
+        │   ├── TraceContextFilter.java          # X-Cycles-Trace-Id / traceparent extraction
         │   ├── RequestIdFilter.java             # X-Request-Id generation
         │   └── RateLimitHeaderFilter.java       # rate limit headers (stub)
         └── exception/GlobalExceptionHandler.java