Skip to content

Commit 8f4d8d4

Browse files
authored
Merge pull request #76 from runcycles/agent/durable-commit-journal
Harden durable settlement journal conformance
2 parents a9b8bdd + 17ce27f commit 8f4d8d4

26 files changed

Lines changed: 2739 additions & 202 deletions

.github/workflows/ci.yml

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,9 +18,28 @@ jobs:
1818
rust-versions: '["stable", "1.88"]'
1919
cargo-args: '--all-features'
2020

21+
recovery-conformance:
22+
name: Durable recovery conformance
23+
runs-on: ubuntu-latest
24+
steps:
25+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
26+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
27+
with:
28+
repository: runcycles/cycles-protocol
29+
ref: 594631c14710da08ad5e00125d899d642213c296
30+
path: .cycles-protocol
31+
persist-credentials: false
32+
- uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable @ 2026-05-02
33+
- run: python -m pip install PyYAML
34+
- name: Run shared durable recovery scenarios
35+
run: >-
36+
python .cycles-protocol/scripts/run_client_recovery_conformance.py
37+
--claim durable
38+
--adapter python scripts/recovery_conformance_adapter.py
39+
2140
publish:
2241
name: Publish to crates.io
23-
needs: ci
42+
needs: [ci, recovery-conformance]
2443
if: startsWith(github.ref, 'refs/tags/v')
2544
runs-on: ubuntu-latest
2645
concurrency:

AUDIT.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,19 @@
11
# Protocol Conformance Audit — Rust Client
22

3+
- **Date:** 2026-07-28 (v0.3.2 — durable pending-commit journal aligned with
4+
the cross-SDK recovery profile. Actual spend is atomically journaled before
5+
the first commit request; retry/auth/ambiguous outcomes survive restart;
6+
expired commits switch durably to event mode; 429 floors persist as absolute
7+
times; tenant identity survives key rotation; corrupt records quarantine;
8+
concurrent replay is same-key safe. Async and blocking clients expose
9+
bounded flush. Settlement records are removed only for schema-valid HTTP 200
10+
commits or HTTP 201 events; other 2xx outcomes remain ambiguous.
11+
`Error::CommitPending` prevents unresolved durable outcomes from
12+
masquerading as safe-to-compensate terminal errors. All 250 unit,
13+
integration, and doc tests pass (12 live tests ignored); tarpaulin line
14+
coverage is 95.06%, including restart replay, event-mode replay, and
15+
rate-limit persistence.),
16+
317
- **Date:** 2026-07-17 (v0.2.7 — commit-retry wiring fix: `CommitRetryEngine` (`src/retry.rs`) existed with complete backoff logic but carried `#[allow(dead_code)]` and was never instantiated outside its own unit tests, so the documented `retry_*` config knobs (builder methods, `CYCLES_RETRY_*` env vars, README) were silent no-ops — and a transient commit failure permanently leaked the reservation until server-side TTL expiry, because `guard.commit()` sets `finalized = true` and consumes the guard before the network call, so `Drop` performs no best-effort release and the caller cannot retry. `ReservationGuard::commit` now retries retryable failures (transport errors, 5xx, error codes the protocol classifies transient per `Error::is_retryable`, incl. the `Unknown` forward-compat arm) **inline** with exponential backoff — the fire-and-forget design originally sketched in the dead code was rejected in adversarial review because it (a) let the retry window outlive the cancelled heartbeat and die on `RESERVATION_EXPIRED`, (b) broke the "commit `Err` is final" invariant, enabling double-charge via caller compensation racing a late background commit, and (c) silently lost pending retries on runtime shutdown (detached `tokio::spawn`). Inline semantics restore all three properties: the heartbeat now stays alive until the commit outcome is final (cancelled after, with `Drop` as backstop), `Ok`/`Err` from `commit()` is definitive, and no detached task exists. Retries reuse the original `CommitRequest` — same idempotency key — so a commit that already landed server-side cannot double-charge. Both `#[allow(dead_code)]` attributes removed; `CyclesClientBuilder` gained the three missing retry setters (`retry_initial_delay`, `retry_multiplier`, `retry_max_delay`). `tests/retry_test.rs` — previously titled "Tests for CommitRetryEngine" while never exercising the engine — rewritten as four end-to-end reserve→commit wiremock tests (retry-until-success with idempotency-key-reuse and upper-bound `.expect` assertions, exhaustion, non-retryable-final, retry-disabled-final); shared reserve-mock scaffolding extracted to `tests/common/mod.rs` (also used by `guard_test.rs`). README documents the retry semantics. Coverage 96.12%. Follow-up same day: `Cargo.lock` `quinn-proto` 0.11.14 → 0.11.16 for RUSTSEC-2026-0185 (remote memory exhaustion; flagged by the scheduled cargo-audit run of 2026-07-13) and `anyhow` 1.0.102 → 1.0.103 for RUSTSEC-2026-0190 (`Error::downcast_mut()` unsoundness; the CI gate runs `cargo audit --deny warnings`, so unsound-warnings fail too). Both transitive dependencies.), 2026-07-10 (v0.2.7 — `TENANT_CLOSED` error-code support per runtime spec v0.1.25.13 (`cycles-protocol-v0.yaml`, runcycles/cycles-protocol#125): `ErrorCode::TenantClosed` variant with serde string mapping `"TENANT_CLOSED"`, plus `Error::is_tenant_closed()` helper mirroring `is_budget_exceeded()`. Purely additive — previously the code hit the `#[serde(other)] Unknown` forward-compat arm, which deserialized cleanly but reported the 409 as retryable via `ErrorCode::Unknown.is_retryable()`; now typed and non-retryable. The 409→`Error::BudgetExceeded` classification is intentionally unchanged (TENANT_CLOSED is tenant-state, not budget-family; it surfaces as `Error::Api`). Serde roundtrip + `Error` helper + wiremock regression tests added. Also `LIMIT_EXCEEDED` per runtime spec v0.1.25.12 (revision 2026-07-04, HTTP 429 rate limiting on the public evidence/JWKS endpoints, `Retry-After` / `X-RateLimit-Reset` headers): `ErrorCode::LimitExceeded` variant added in spec declaration order (`TenantClosed` relocated after it to mirror the spec exactly), classified retryable by `ErrorCode::is_retryable()` — 429 is transient; `Error::is_retryable()` inherits this via the code arm, preserving the prior `Unknown → retryable` fallback semantics, now typed. Enum-only, matching the `BudgetFrozen`/`BudgetClosed` sibling pattern (no `Error` helper, no 409-classification change). Serde roundtrip + retryability + wiremock 429 tests added.), 2026-07-04 (v0.2.7 — `reserve()` no longer panics on additive `Decision` values (fleet audit, #56 item 1): an unknown decision deserializes to `Decision::Unknown` via `#[serde(other)]`, bypassed `is_denied()`, and hit `.expect("reservation_id must be present…")`. Unknown/additive decisions now return `Error::Validation` regardless of `reservation_id` presence — `reserve()` gates on positive `Decision::is_allowed()`, not merely non-denial (review follow-up on the first cut, which still built a guard when an id happened to be present). Wiremock regression test added; full suite green. Remaining audit findings tracked in #56.), 2026-05-22 (v0.2.6 — `expires_*` / `finalized_*` ISO-8601 window-filter fields added to `ListReservationsParams` plus optional `finalized_at_ms` field added to `ReservationSummary` per `cycles-protocol-v0.yaml` revision 2026-05-22 (runcycles/cycles-protocol#98); closes the Rust-client side of runcycles/cycles-server#162. Four new `Option<String>` fields on the params struct (`expires_from`, `expires_to`, `finalized_from`, `finalized_to`), one new `Option<u64>` field on the response struct (`finalized_at_ms`, with `#[serde(default)]` for back-compat with pre-v0.1.25.21 servers). Wire-format regression tests + finalized_at_ms deserialization tests added. 134 tests pass; clippy + doc-tests clean.), 2026-05-21 (v0.2.5 — `from` / `to` ISO-8601 window-filter fields added to `ListReservationsParams` per `cycles-protocol-v0.yaml` revision 2026-05-21; closes the Rust-client side of runcycles/cycles-server#159. Both `Option<String>`, both inclusive bounds on `created_at_ms`, both serialize via `#[serde(rename = "...")]` to land on the wire under the spec-mandated names. Pure additive struct change — callers using `Default::default()` or struct-update syntax stay compile-clean. Wire-format regression test added using wiremock's `query_param` matcher. 134 tests pass; clippy + doc-tests clean.), 2026-04-10 (protocol conformance), 2026-04-19 (supply-chain coverage — cargo-audit workflow added), 2026-05-08 (crates.io metadata refresh — description and keywords broadened to cover spend / risk / audit, no behavioral changes)
418
- **Spec:** `cycles-protocol-v0.yaml` v0.1.25 (OpenAPI 3.1.0)
519
- **Client:** Rust 1.88+ (MSRV), reqwest 0.12, serde 1, tokio 1, bon 3
@@ -8,6 +22,20 @@
822

923
---
1024

25+
## 2026-07-28 — durable settlement replay (v0.3.2)
26+
27+
The Rust SDK now claims the durable recovery profile rather than its prior
28+
inline-only model. `src/journal.rs` implements the fleet's version-1 record
29+
shape and PBKDF2 partition fingerprint. `ReservationGuard::commit` writes the
30+
record before its first request, persists `Retry-After` before sleeping,
31+
switches to event mode before expired-reservation fallback, and returns
32+
`CommitPending` only when the unresolved outcome is durably queued. Replay is
33+
automatic inside Tokio and explicitly drainable with a bounded timeout; the
34+
blocking client owns a replay worker without blocking construction. Tests
35+
cover cross-language fingerprint pins, atomic record/load/discard, corrupt
36+
quarantine, restart replay under API-key rotation, event-mode replay, 429
37+
timing, and strict schema-valid settlement success predicates.
38+
1139
## 2026-07-27/28 — heartbeat extend-drift fix: `remaining_ttl_ms` normative scheduling + grant-ledger fallback (v0.3.1)
1240

1341
Final self-review correction: the strict success predicate is uniform in both

CHANGELOG.md

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,55 @@ All notable changes to this project will be documented in this file.
44

55
The format is based on [Keep a Changelog](https://keepachangelog.com/).
66

7+
## [0.3.2] - 2026-07-28
8+
9+
### Added
10+
11+
- **Durable pending-commit journal.** Once actual usage is known,
12+
`ReservationGuard::commit` atomically records the commit and its
13+
expiry-to-event fallback before the first network attempt. Pending records
14+
survive retry exhaustion, authentication failure, ambiguous 4xx responses,
15+
cancellation, and process restart; successful or genuinely terminal
16+
outcomes remove them.
17+
- Journal records use the Python/TypeScript/Java version-1 wire shape and
18+
PBKDF2 identity fingerprint. Tenant partitions remain stable across API-key
19+
rotation, no credential is stored, Unix directories/files are restricted to
20+
`0700`/`0600`, malformed records are quarantined, and concurrent replay is
21+
safe through the stored idempotency key.
22+
- Journal filenames use `v2-<sha256(exact UTF-8 reservation id)>.json`, safely
23+
migrate matching legacy records, and preserve collision-free cross-SDK
24+
replay.
25+
- Automatic replay starts when an async client is constructed inside Tokio;
26+
the blocking client owns a worker that starts replay without blocking its
27+
constructor. Both clients expose `flush_pending_commits` and bounded
28+
`flush_pending_commits_with_timeout` operations.
29+
- `CyclesConfig::{journal_enabled,journal_dir}`, matching builder setters and
30+
`CYCLES_JOURNAL_ENABLED` / `CYCLES_JOURNAL_DIR` environment variables.
31+
- `Error::CommitPending` distinguishes a durably queued, unresolved settlement
32+
from a genuine terminal failure. Callers must not compensate it with a new
33+
key.
34+
35+
### Fixed
36+
37+
- Commit accepts only a schema-valid HTTP 200 `COMMITTED` response and event
38+
fallback accepts only a schema-valid HTTP 201 `APPLIED` response. Other 2xx
39+
outcomes remain ambiguous, reuse the original key, and keep the durable
40+
record instead of being mistaken for terminal success.
41+
- Corrupt, semantically invalid, and unsupported-version journal records are
42+
quarantined without blocking valid replay, and the shared
43+
recovery-conformance adapter reports the exact native test it executed.
44+
- A 429 retry floor is persisted before the retry sleep and as an absolute
45+
`not_before_ms`, so restarting mid-wait cannot violate `Retry-After`.
46+
- Expired commits persist the journal's event mode before
47+
`POST /v1/events`; a crash cannot make replay return to a doomed commit.
48+
- Pull-request and release CI run every shared durable-recovery and
49+
guarantee-boundary scenario, and publishing is gated on conformance.
50+
51+
### Compatibility
52+
53+
- Adding `Error::CommitPending` and two public `CyclesConfig` fields requires
54+
downstream exhaustive matches/struct literals to add the new members.
55+
756
## [0.3.1] - 2026-07-27
857

958
Heartbeat extend-drift fix (P1 liveness, fleet-wide — same bug in all four SDKs), refined under five rounds of adversarial + spec review and a final spec-alignment pass: alternate-beat → lead-estimate → grant-ledger → grant-ledger with immediate first beat and lead-clamp regime (v2.3) → **server-authoritative `remaining_ttl_ms` scheduling per the spec's HEARTBEAT GUIDANCE (spec PR #148, head `dd60c27`), with the v2.3 heuristic as fallback**.

0 commit comments

Comments
 (0)