Skip to content

Commit 7df3a08

Browse files
committed
fix: align heartbeat to settled spec PR #148 HEARTBEAT GUIDANCE
Spec PR #148 settled at head dd60c27 with a hardened primary algorithm; the yaml's HEARTBEAT GUIDANCE on extendReservation is the authority. Rework of the interim round-5 implementation, both heartbeats (lifecycle + streaming), version stays 0.4.1: - SUCCESS PREDICATE: only a schema-valid HTTP 200 ReservationExtendResponse (status ACTIVE, integer expires_at_ms >= 0, optional integer remaining_ttl_ms >= 0) is an observed success on the primary path. Any other/malformed 2xx is AMBIGUOUS -> same-key recovery, never scheduled from. Fieldless fallback keeps lenient 2xx-as-applied. - SCHEDULING: per-attempt monotonic rtt (max tracked); lead_floor = max(0, remaining_ttl_ms - rtt); request_timeout_budget = enforced connectTimeout + readTimeout (the client applies AbortSignal.timeout on every request); attempt_budget = max(timeout, 1s, 2*maxRtt); safety_margin = max(1s, 2*maxRtt); next_delay = max(0, lead_floor - (2*attempt_budget + safety_margin)) from response receipt, recomputed from every schema-valid response. Budgets/margins round up, leads/delays round down; saturating arithmetic (unbounded timeout -> infinite budget -> next_delay 0). - ZERO-DELAY GUARD: one immediate fresh attempt (new key) per zero-delay success; two consecutive zeros stop + surface (lease shorter than the retry-safety budget). Unreliable timing forces lead_floor = 0 into the same guard; never a silent fallback downgrade. - RECOVERY: retry_window = current_lead - attempt_budget - safety_margin, UNclamped; negative -> stop + surface; non-429 same-key retry after min(30s, lead/4, window); 429 honors Retry-After (delta-seconds * 1000, overflow-safe) exactly and only within the window, else stop; repeated recovery recomputes from the same last schema-valid response each failure; zero window -> one immediate retry; progress guard stops zero-time loops; any other 4xx stops without rotating the key. Permanent stops unchanged. - CREATE: first beat from the create response's remaining_ttl_ms with the create call's own measured rtt (no first-attempt-rtt widening — replays recompute remaining_ttl_ms server-side). - FALLBACK path (fieldless servers) unchanged. Tests reworked to the settled formulas (T=7000 test config -> reserve 15000: 45000ms cadence at remaining 60000; 15000ms under a 30000 lead cap) and extended: ambiguous-2xx same-key recovery; zero-delay guard (1s lease -> one immediate fresh attempt -> stop + surfaced warning); recovery loop 3750/2812/438/0 then stop when no retry+margin fits, all same key; 429 honored at exactly 2000ms within window / stop at 8000ms exceeding it; 4xx stop without key rotation; unbounded-timeout stop (no silent fallback). All legacy fieldless tests pass unchanged. 456 tests pass; line coverage 97.65%, branch 91.96% (gates 95/85); eslint, tsc, build clean. CHANGELOG/AUDIT/README aligned in place.
1 parent 504c48b commit 7df3a08

5 files changed

Lines changed: 1092 additions & 237 deletions

File tree

AUDIT.md

Lines changed: 54 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -48,19 +48,37 @@ the band forever, −2 s per cycle) — so the protocol gained a
4848
server-authoritative field (spec PR #148): **`remaining_ttl_ms`** on both
4949
`ReservationCreateResponse` and `ExtendResponse` (int64 ≥ 0, remaining
5050
lifetime at response evaluation, same clock snapshot as `expires_at_ms`,
51-
optional in the client models for back-compat). When present, scheduling is
52-
**normative**: `lead_floor = max(0, remaining − rtt)` (rtt = monotonic
53-
send→receive elapsed, unknown → 0; max observed rtt tracked per heartbeat),
54-
`retry_reserve = min(lead_floor/2, max(1 s, 2·max_rtt))`, next beat at
55-
`lead_floor − retry_reserve` after receipt — recomputed from every
56-
field-carrying response, never from accumulated expiry differences; the
57-
`lead_min` skip check is bypassed (exact schedule; a heuristic skip could
58-
overshoot the real lease) while the ledger keeps running for seamless
59-
fallback if the field disappears; transient failures retry same-key after
60-
`clamp(lead_estimate/4, 1 s, 30 s)`; a field-carrying create derives the
61-
first beat from the same formula instead of the immediate prime (no wasted
62-
extension under max-lead clamping). The grant-ledger heuristic below is
63-
retained verbatim as the fallback for servers without the field.
51+
optional in the client models for back-compat). The final spec-alignment
52+
pass conformed the client to the settled HEARTBEAT GUIDANCE (spec PR #148
53+
head `dd60c27`). When present, scheduling is **normative**: only a
54+
schema-valid HTTP 200 counts as success (other/malformed 2xx = ambiguous →
55+
same-key transient recovery; new `extend_reservation_strict`);
56+
`lead_floor = max(0, remaining − rtt)` (rtt = the individual attempt's
57+
monotonic send→receive elapsed — replays recompute `remaining_ttl_ms`
58+
server-side, so no earlier attempt's timing is ever substituted),
59+
`attempt_budget = max(request_timeout_budget, 1 s, 2·max_rtt)` (the
60+
enforced reqwest per-attempt bound `connect_timeout + read_timeout`;
61+
unknown/unbounded → infinity → delay 0), `safety_margin = max(1 s,
62+
2·max_rtt)`, `retry_reserve = 2·attempt_budget + safety_margin`,
63+
`next_delay = max(0, lead_floor − retry_reserve)` after receipt —
64+
recomputed from every field-carrying response, never from accumulated
65+
expiry differences; overflow-safe saturating ms, budgets/margins rounded
66+
up, leads/delays rounded down (`ceil_ms`); the `lead_min` skip check is
67+
bypassed while the ledger keeps running for seamless fallback if the field
68+
disappears. Zero-delay guard: one immediate fresh-key extension after a
69+
zero-delay success, stop + warn on the second in a row (lease shorter than
70+
the retry-safety budget). Recovery (timeout/conn/5xx/429/ambiguous 2xx):
71+
`retry_window = lead_estimate − attempt_budget − safety_margin` (signed);
72+
negative → stop and surface; else same-key retry after
73+
`min(30 s, lead_estimate/4, window)`; 429 retries after exactly
74+
`Retry-After` (delta-seconds × 1000, checked) only when it fits the window,
75+
else stop; repeated recovery recomputes lead/window from the same last
76+
schema-valid response after every failure with a progress guard (window
77+
must decrease); any other 4xx stops without key rotation. A field-carrying
78+
create derives the first beat from the same formula instead of the
79+
immediate prime (no wasted extension under max-lead clamping). The
80+
grant-ledger heuristic below is retained verbatim as the fallback for
81+
servers without the field.
6482
**Grant-ledger fallback (v2.3)** design: correctness rests on
6583
`lead_min = grants_sum − elapsed` (signed, starts 0), a rigorous lower bound
6684
built only from same-frame arithmetic — each grant is the difference of
@@ -81,25 +99,36 @@ equivalent, hand-rolled for variable delays). Any 2xx counts as applied
8199
(`expires_at_ms` authoritative, warn on odd status); permanent codes
82100
(`RESERVATION_EXPIRED`/`RESERVATION_FINALIZED`/`MAX_EXTENSIONS_EXCEEDED`/
83101
`TENANT_CLOSED`/`NOT_FOUND` or HTTP 410/404) stop the heartbeat.
84-
Cancellation unchanged. Sixteen wiremock tests with dynamic expiry
102+
Cancellation unchanged. Twenty-four wiremock tests with dynamic expiry
85103
responders — fallback: immediate first beat + extend@0/1000/2000,
86104
skip@3000, extend@4000; capped grant keeping `extend_by_ms` at the request
87105
while the immediate beat discovers the cap; 503 on the immediate beat →
88106
single held-cadence retry with the same key; permanent stops; small-ttl
89107
liveness; per-extend grant clamp still tightening to grant/2; lead-clamp
90108
echo responder holding cadence instead of collapsing; zero-grant immediate
91-
prime holding cadence; unknown-status 200 as applied — normative: no prime
92-
+ `lead_floor − retry_reserve` first beat; skip bypass under accumulating
93-
grants; capped 1 s lease first-beating at ~500 ms inside the lease;
94-
field-carrying max-lead clamp at ~cap − reserve with no collapse; field
95-
disappearing mid-flight → heuristic resuming (with its skip) on the
96-
ledger maintained through the normative phase; same-key normative retry at
97-
`clamp(lead/4, 1 s, 30 s)`. Extracted pure scheduling functions
98-
unit-tested (30 s held-cadence cap, lead-clamp band boundaries, 60 s → 59 s
99-
normative delay pin, rtt widening/saturation, retry-clamp bounds) in
100-
`src/heartbeat.rs`; `remaining_ttl_ms` wire-format serde tests in
109+
prime holding cadence; unknown-status 200 as applied; non-conformant
110+
reserve without expires_at_ms still heartbeating on the requested-amount
111+
grant fallback — normative (small enforced timeout → minimal reserve
112+
3000 ms): no prime + `lead_floor − retry_reserve` first beat with the
113+
`lead_min` skip bypassed; zero-delay guard (sub-reserve lease → one
114+
immediate fresh-key extension → stop + warn) and its single-dip recovery
115+
twin (one zero-delay success → one fresh immediate extension → healthy
116+
lease resumes the schedule); field-carrying max-lead clamp at
117+
~cap − reserve with no collapse; field disappearing mid-flight → heuristic
118+
resuming at grant/2; 503, ambiguous 200 (non-schema body), and ambiguous
119+
204 (non-200 2xx) recovering same-key inside the window then resuming
120+
normatively with a fresh key; persistent 500 retrying same-key until the
121+
recomputed window goes negative, then stopping for good; 429 in-window
122+
`Retry-After: 0` retried after exactly that (same key) vs. 429 exceeding
123+
the window stopping without an early retry; 400 stopping with no retry and
124+
no key rotation. Extracted pure scheduling functions unit-tested (30 s
125+
held-cadence cap, lead-clamp band boundaries, the spec's worked examples
126+
60 000/10 000/1 500 → 37 000 and 30 s timeout → 0, budget/margin floors,
127+
unbounded-budget infinity, signed windows, `min(30 s, lead/4, window)`,
128+
`ceil_ms` rounding, the recovery progress guard, field-mode
129+
recoverable/stop classification) in `src/heartbeat.rs`; `remaining_ttl_ms` wire-format serde tests in
101130
`tests/models_test.rs`; `Date`-parsing unit tests in `src/response.rs`.
102-
Coverage 95.26%; tests, clippy `-D warnings`, fmt green.
131+
Coverage 95.05%; tests, clippy `-D warnings`, fmt green.
103132

104133
## 2026-07-27 — v0.3.0 self-review hardening
105134

CHANGELOG.md

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,17 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/).
66

77
## [0.3.1] - 2026-07-27
88

9-
Heartbeat extend-drift fix (P1 liveness, fleet-wide — same bug in all four SDKs), refined under five rounds of adversarial + spec review: alternate-beat → lead-estimate → grant-ledger → grant-ledger with immediate first beat and lead-clamp regime (v2.3) → **server-authoritative `remaining_ttl_ms` scheduling with the v2.3 heuristic as fallback (round 5)**.
9+
Heartbeat extend-drift fix (P1 liveness, fleet-wide — same bug in all four SDKs), refined under five rounds of adversarial + spec review and a final spec-alignment pass: alternate-beat → lead-estimate → grant-ledger → grant-ledger with immediate first beat and lead-clamp regime (v2.3) → **server-authoritative `remaining_ttl_ms` scheduling per the spec's HEARTBEAT GUIDANCE (spec PR #148, head `dd60c27`), with the v2.3 heuristic as fallback**.
1010

1111
### Added
1212

13-
- **`remaining_ttl_ms` support (spec PR #148) — normative heartbeat scheduling.** Round 5 of the spec review proved regime detection from `(grant, elapsed)` samples **undecidable in general**: any real per-extend grant in the sticky window `[0.75·min(ttl/2, 30 s), 0.9·ttl)` tracks the held cadence closely enough to stay classified lead-clamp while the lease erodes to a lapse (e.g. ttl 24 s with real +10 s grants → held cadence 12 s → post-skip ratio 10/12 sits inside the `[0.75, 1.25]` band forever, losing 2 s per cycle). The protocol therefore gained a server-authoritative field: `remaining_ttl_ms` (int64 ≥ 0) on **both** `ReservationCreateResponse` and `ExtendResponse` — the remaining reservation lifetime in ms at response evaluation, same clock snapshot as `expires_at_ms`, present on successful live-reservation responses (absent on dry-run/DENY and on older servers; the field is optional in the client models for back-compat).
13+
- **`remaining_ttl_ms` support (spec PR #148, settled at head `dd60c27`) — normative heartbeat scheduling.** Round 5 of the spec review proved regime detection from `(grant, elapsed)` samples **undecidable in general**: any real per-extend grant in the sticky window `[0.75·min(ttl/2, 30 s), 0.9·ttl)` tracks the held cadence closely enough to stay classified lead-clamp while the lease erodes to a lapse (e.g. ttl 24 s with real +10 s grants → held cadence 12 s → post-skip ratio 10/12 sits inside the `[0.75, 1.25]` band forever, losing 2 s per cycle). The protocol therefore gained a server-authoritative field: `remaining_ttl_ms` (int64 ≥ 0) on **both** `ReservationCreateResponse` and `ExtendResponse` — the remaining reservation lifetime in ms at response evaluation, same clock snapshot as `expires_at_ms`, present on successful live-reservation responses (absent on dry-run/DENY and on older servers; the field is optional in the client models for back-compat).
1414

15-
When a successful response carries the field, scheduling is **normative**: `lead_floor = max(0, remaining_ttl_ms − rtt)` (rtt = monotonic elapsed between sending the call and receiving the response; unknown → 0), `retry_reserve = min(lead_floor/2, max(1 s, 2·max_observed_rtt))`, and the next beat lands `lead_floor − retry_reserve` after response receipt — recomputed from **every** field-carrying response, never from accumulated expiry differences. The `lead_min` skip heuristic is **bypassed** in this mode (the schedule is exact; a heuristic skip could overshoot the real lease), but the grant ledger keeps running in the background so the heuristic resumes seamlessly if a later response omits the field (mixed fleets, rollbacks). A transient failure retries with the **same idempotency key** after `clamp(lead_estimate/4, 1 s, 30 s)`, where `lead_estimate` is the last `lead_floor` minus the monotonic time since that response (saturating at 0). When the **create** response carries the field, the first beat is derived from the same formula instead of the immediate prime — a 60 s remaining lease first beats at 59 s, a tenant-capped 1 s lease at 500 ms (inside the lease), and no primed extension is spent even under a maximum-lead clamp. Extend amount (requested ttl), permanent stops, and 2xx-as-applied are unchanged.
15+
When a successful response carries the field, scheduling follows the spec's **HEARTBEAT GUIDANCE primary algorithm** verbatim. Only a **schema-valid HTTP 200** `ReservationExtendResponse` (or the create response, for the first beat) counts as an observed success — a different or malformed 2xx is *ambiguous* and handled as a transient failure with same-key recovery (new `extend_reservation_strict` internal call; the fallback keeps its any-2xx-as-applied semantics). On every success the schedule is recomputed from that response alone, never from accumulated expiry differences: `lead_floor = max(0, remaining_ttl_ms − rtt)` (rtt = the *individual attempt's* monotonic send→receive elapsed; same-key replays are safe to schedule from because the server recomputes `remaining_ttl_ms` at replay-response construction), `attempt_budget = max(request_timeout_budget, 1 s, 2·max_observed_rtt)` where `request_timeout_budget` is the client's **enforced** finite per-attempt bound (the reqwest client's `connect_timeout + read_timeout`; an unknown/unbounded budget is positive infinity → `next_delay = 0`), `safety_margin = max(1 s, 2·max_observed_rtt)`, `retry_reserve = 2·attempt_budget + safety_margin`, `next_delay = max(0, lead_floor − retry_reserve)` — scheduled after response receipt, overflow-safe saturating ms arithmetic, budgets/margins rounded up and leads/delays rounded down. The `lead_min` skip heuristic is **bypassed** in this mode (the schedule is exact; a heuristic skip could overshoot the real lease), but the grant ledger keeps running in the background so the heuristic resumes seamlessly if a later response omits the field (mixed fleets, rollbacks).
16+
17+
**Zero-delay guard:** a schema-valid success producing `next_delay = 0` permits one immediate fresh-key extension; a second consecutive zero-delay success stops the heartbeat and surfaces (warn) that the lease is shorter than the retry-safety budget — an additive-delta server gets its one immediate extension to establish a larger lead, while a maximum-lead server's extension budget is never burned in a tight loop.
18+
19+
**Recovery** (timeout, connection error, 5xx, 429, ambiguous 2xx): `current_lead_estimate = max(0, last lead_floor − elapsed since the schema-valid response that established it)`; `retry_window = current_lead_estimate − attempt_budget − safety_margin` (signed, unclamped). A negative window means no complete retry plus margin is provably safe: the heartbeat stops and surfaces. Otherwise non-429 failures retry with the **same idempotency key** after `min(30 s, lead_estimate/4, retry_window)`; a 429 retries after exactly `Retry-After` (delta-seconds × 1000, overflow-checked) and only when it fits the window — a missing/invalid/oversized `Retry-After` stops rather than inventing an earlier retry that violates throttling. Recovery repeats while the freshly recomputed window stays non-negative (recomputed from the same last schema-valid response after **every** failure); a zero window permits one immediate retry, and a progress guard stops the run when the window fails to decrease between consecutive failures. Any **other 4xx** stops and surfaces without rotating the idempotency key. Extend amount (requested ttl) and the permanent stop set are unchanged.
1620

1721
### Fixed
1822

@@ -36,7 +40,7 @@ Heartbeat extend-drift fix (P1 liveness, fleet-wide — same bug in all four SDK
3640

3741
Cancellation semantics are unchanged (`CancellationToken`; the guard cancels on commit/release/drop). Regression tests (`tests/heartbeat_test.rs`, wiremock with dynamic `expires_at_ms` responders) pin: the immediate first beat (an extend arrives well before ttl/2) and the v2.3 full-grant cadence extend@0/1000/2000 ms / skip@3000 (bound exactly `1.5·grant`, inclusive) / extend@4000; the capped scenario (requested 8000 / granted 2000 → the immediate beat discovers the cap that a requested/2 schedule would have found 2 s after expiry, cadence tracking the *observed* grant at 1000 ms, wire `extend_by_ms` staying the requested 8000); a 503 on the immediate first beat retrying at the held cadence — exactly one attempt inside the first margin, never a zero-delay hot loop — with the **same** idempotency key, then a fresh key after success (asserted from received request bodies); permanent-failure stop for 409 `MAX_EXTENSIONS_EXCEEDED`, 409 `TENANT_CLOSED`, and 404 `NOT_FOUND` (no further requests); ttl 1200 staying alive across 600 ms beats with the skip landing exactly at the threshold beat; a per-extend grant clamp (+ttl/4) still *tightening* the cadence to grant/2 = 500 ms; a **lead-clamp responder** (echoing `reserve_expiry + elapsed-at-receipt`) holding the cadence at requested/2 instead of collapsing to the 500 ms floor; a zero-grant immediate prime holding the cadence likewise; and a 200 with an unknown status counting as applied (fresh key next beat + the steady-state skip still occurring — pinning both the resolution and the ledger update). The pure cadence/regime computations are extracted as functions and unit-tested in `src/heartbeat.rs` — including the 30 s held-cadence cap for huge TTLs, which would be impractical to wait out in wall-clock tests — alongside skip-threshold, grant-cadence, lead-clamp-band boundary, and permanent-classification tests; `Date`-parsing unit tests remain in `src/response.rs` (the parsing is now a general utility, exercised end-to-end in `tests/response_test.rs`).
3842

39-
**Normative-mode tests** (round 5): wiremock responders now optionally emit `remaining_ttl_ms` (constant, or only on the first *n* responses) and pin — no immediate prime and a `lead_floor − retry_reserve` first beat when the create carries the field; ~1 s steady normative cadence with the `lead_min` skip **bypassed** even when accumulated grants would trip it; a tenant-capped 1 s lease first-beating at ~500 ms (inside the lease); a max-lead-clamping field-carrying server scheduling at ~cap − reserve with no cadence collapse and no primed extension; the field disappearing mid-flight with the heuristic resuming at the observed cadence **and** skipping on the ledger maintained through the normative phase; and a transient failure in normative mode retrying with the same idempotency key after `clamp(lead/4, 1 s, 30 s)`. Pure-function pins in `src/heartbeat.rs` cover the 59 s delay for a 60 s remaining lease, rtt/max-rtt widening, the lead-floor saturation, and the retry clamp bounds; wire-format serde tests in `tests/models_test.rs` cover the optional field on both responses.
43+
**Normative-mode tests** (round 5, reworked to the settled spec): wiremock responders optionally emit `remaining_ttl_ms` (constant, or only on the first *n* responses), and the field-mode clients enforce a small per-attempt timeout so the minimal recovery reserve is `2·1000 + 1000 = 3000 ms`. Pinned: no immediate prime and a `lead_floor − retry_reserve` first beat when the create carries the field, with the `lead_min` skip **bypassed** even when accumulated grants would trip it; the zero-delay guard (a lease below the reserve → exactly one immediate fresh-key extension, then stop + warn); a max-lead-clamping field-carrying server scheduling at ~cap − reserve with no cadence collapse and no primed extension; the field disappearing mid-flight with the heuristic resuming at the observed grant/2 cadence; a 503 and an ambiguous 200 (non-schema body) both recovering with the **same** idempotency key inside the window and resuming the normative schedule with a fresh key after success; repeated recovery against a persistent 500 retrying same-key until the recomputed window goes negative, then stopping for good; a 429 with in-window `Retry-After: 0` retried after exactly that delay (same key) and a 429 whose `Retry-After` exceeds the window stopping without an early retry; a 400 stopping immediately with no retry and no key rotation; an ambiguous 204 (non-200 2xx) recovered same-key like the malformed 200; a single zero-delay success recovering (one fresh immediate extension, then a healthy lease resumes the schedule — no stop); and a non-conformant reserve without `expires_at_ms` still heartbeating in the fallback on the requested-amount grant. Pure-function pins in `src/heartbeat.rs` cover the spec's worked examples (60 s lead + 10 s timeout + 1.5 s max rtt → 37 s delay; 30 s timeout → 0), the 1 s floors, unbounded-budget infinity, lead-floor saturation, signed window arithmetic, `min(30 s, lead/4, window)`, `ceil_ms` rounding, and the field-mode recoverable/stop classification; wire-format serde tests in `tests/models_test.rs` cover the optional field on both responses.
4044

4145
## [0.3.0] - 2026-07-27
4246

0 commit comments

Comments
 (0)