Skip to content

Commit f0c5b19

Browse files
committed
docs(wave2): apply codex review round
Apply 9 / modify 0 / skip 0: - security-hardening ACL gains ~idempotency:* (admin fund keys use idempotency:fund:v2:*) - prometheus-reference: release counter counts every outcome (RELEASED or DENY); UNKNOWN normalisation scoped to operation counters with the evidence artifact_type exception - no-budget vs insufficient-budget split fixed where wave-2 protocol corrections created contradictions: glossary three-way-decision entry and two full-stack quickstart spots now say 404 NOT_FOUND for budgetless scopes, 409 BUDGET_EXCEEDED for insufficient existing budgets - retry-storms snippet: import os - api-key-management: "6 defaults" wording fixed; cross-tenant admin advice corrected (admin key required; tenant-key wildcards never grant AdminKeyAuth access) - changelog current-feature summary: BLMOVE (BRPOP historical) - multi-tenant-saas: live denial is 409, DENY is decide/dry-run
1 parent e831017 commit f0c5b19

8 files changed

Lines changed: 11 additions & 10 deletions

changelog.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -201,7 +201,7 @@ New event-driven observability system spanning all three services.
201201
- `reservation.commit_overage` on commit with actual > estimated
202202

203203
**Events delivery service (`cycles-server-events`, internal app port 7980):**
204-
- Async webhook delivery via BRPOP from shared Redis dispatch queue
204+
- Async webhook delivery via BLMOVE claim/ack from the shared Redis dispatch queue (BRPOP before v0.1.25.18)
205205
- HMAC-SHA256 payload signing (`X-Cycles-Signature: sha256=<hex>`)
206206
- Exponential backoff retry, auto-disable after consecutive failures
207207
- Stale delivery protection (>24h deliveries auto-fail on pickup)

glossary.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ A constraint applied to execution when budget is running low but not yet exhaust
7777

7878
### Three-Way Decision
7979

80-
The three possible outcomes of a budget evaluation: **ALLOW** (proceed normally), **ALLOW_WITH_CAPS** (proceed with reduced limits), or **DENY** (reject the request). `decision=DENY` appears only on `/decide` responses and `dry_run` reservations; a live (non-dry-run) reservation that lacks budget is rejected with HTTP `409 BUDGET_EXCEEDED` instead of a `DENY` decision. This model enables graceful degradation instead of hard pass/fail. See [Caps and the Three-Way Decision Model](/protocol/caps-and-the-three-way-decision-model-in-cycles).
80+
The three possible outcomes of a budget evaluation: **ALLOW** (proceed normally), **ALLOW_WITH_CAPS** (proceed with reduced limits), or **DENY** (reject the request). `decision=DENY` appears only on `/decide` responses and `dry_run` reservations; a live (non-dry-run) reservation against an existing but insufficient budget is rejected with HTTP `409 BUDGET_EXCEEDED`; if no derived scope has a budget at all, the server returns `404 NOT_FOUND` (dry-run and `/decide` return `DENY` with `reason_code=BUDGET_NOT_FOUND`). This model enables graceful degradation instead of hard pass/fail. See [Caps and the Three-Way Decision Model](/protocol/caps-and-the-three-way-decision-model-in-cycles).
8181

8282
### Overage Policy
8383

how-to/api-key-management-in-cycles.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,7 +106,7 @@ curl -X POST http://localhost:7979/v1/admin/api-keys \
106106

107107
> **Defaults:** When no permissions are specified at key creation, the key receives 10 default permissions: the 6 runtime permissions (`reservations:create`, `reservations:commit`, `reservations:release`, `reservations:extend`, `reservations:list`, `balances:read`) plus `budgets:read`, `budgets:write`, `policies:read`, `policies:write`. Webhook, event, and admin permissions must be explicitly requested.
108108
109-
A typical runtime key needs only the 6 defaults. Add `budgets:write` and `budgets:read` if tenants manage their own budgets. Add `webhooks:write` and `webhooks:read` for [webhook subscriptions](/how-to/managing-webhooks#tenant-self-service). Add `admin:read`/`admin:write` (or granular equivalents) only if the key is used for cross-tenant admin operations via the admin server (port 7979).
109+
A typical runtime key needs only the 6 runtime permissions (`reservations:*` and `balances:read`) — remember that an explicit `permissions` array replaces the 10-value default set, so include `budgets:read`/`budgets:write` if the key also manages budgets. Add `webhooks:write` and `webhooks:read` for [webhook subscriptions](/how-to/managing-webhooks#tenant-self-service). Cross-tenant admin operations require the admin server's `X-Admin-API-Key` — no tenant-key permission (including the legacy `admin:read`/`admin:write` wildcards) grants access to those endpoints.
110110

111111
::: warning Admin permissions on tenant keys (v0.1.25.7)
112112
`admin:read` and `admin:write` are accepted on tenant keys for backward compatibility, but **SHOULD NOT be assigned to new tenant keys**. Use the specific permissions (`budgets:write`, `policies:read`, etc.) instead. The admin key (`X-Admin-API-Key`) is server-configured and is not provisioned through the API key creation endpoint.

how-to/multi-tenant-saas-with-cycles.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ Map your pricing tiers to budget allocations:
4242
| Pro | $50 (5,000,000,000 microcents) | $5 overdraft | 5 |
4343
| Enterprise | $500 (50,000,000,000 microcents) | $50 overdraft | Unlimited |
4444

45-
When a customer hits their budget limit, Cycles returns `DENY` on the next reservation. Your application decides what happens: show an upgrade prompt, queue the request, or degrade to a cheaper model. See [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) for patterns.
45+
When a customer hits their budget limit, the next live reservation fails with `409 BUDGET_EXCEEDED` (a `/v1/decide` or dry-run check returns `DENY`). Your application decides what happens: show an upgrade prompt, queue the request, or degrade to a cheaper model. See [Degradation Paths](/how-to/how-to-think-about-degradation-paths-in-cycles-deny-downgrade-disable-or-defer) for patterns.
4646

4747
## Customer onboarding workflow
4848

how-to/prometheus-metrics-reference.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ Cycles' Micrometer instrumentation uses dotted source names (`cycles.*`) which P
1212
::: info Tenant-tag cardinality flag
1313
Every counter tagged with `tenant` respects a per-service toggle, `cycles.metrics.tenant-tag.enabled` (env `CYCLES_METRICS_TENANT_TAG_ENABLED`) — but the **defaults differ by service**: `cycles-server` (runtime) defaults to `true`, while `cycles-server-events` defaults to `false`. The admin server's `cycles_admin_*` counters carry no `tenant` tag at all, so the flag doesn't apply there. Deployments with many thousands of tenants can flip the runtime flag to `false` to drop the per-tenant series and keep Prometheus cardinality bounded; deployments that want per-tenant webhook drill-downs must explicitly enable it on the events service. Set the flag consistently across the two services so dashboards can share the same tag schema.
1414

15-
Null or blank tag values are normalised to the sentinel `UNKNOWN`. Missing tags would otherwise collapse series — making it look like traffic moved when the upstream data actually just got sparse.
15+
For the operation counters below, null or blank tag values are normalised to the sentinel `UNKNOWN` (exception: `cycles.evidence.emit_failed` uses lowercase `unknown` for a null `artifact_type` and does not normalise blanks). Missing tags would otherwise collapse series — making it look like traffic moved when the upstream data actually just got sparse.
1616
:::
1717

1818
## Scrape targets
@@ -33,7 +33,7 @@ Introduced in v0.1.25.10. All counters live under the `cycles.*` namespace.
3333
|---|---|---|---|---|
3434
| `cycles.reservations.reserve` | `cycles_reservations_reserve_total` | Counter | `tenant`, `decision`, `reason`, `overage_policy` | Every `POST /v1/reservations` outcome. |
3535
| `cycles.reservations.commit` | `cycles_reservations_commit_total` | Counter | `tenant`, `decision`, `reason`, `overage_policy` | Every `POST /v1/reservations/{id}/commit` outcome. |
36-
| `cycles.reservations.release` | `cycles_reservations_release_total` | Counter | `tenant`, `actor_type`, `decision`, `reason` | Every successful release. `actor_type` distinguishes tenant-driven releases from v0.1.25.8 admin-on-behalf-of releases. |
36+
| `cycles.reservations.release` | `cycles_reservations_release_total` | Counter | `tenant`, `actor_type`, `decision`, `reason` | Every release outcome (`decision` is `RELEASED` or `DENY`). `actor_type` distinguishes tenant-driven releases from v0.1.25.8 admin-on-behalf-of releases. |
3737
| `cycles.reservations.extend` | `cycles_reservations_extend_total` | Counter | `tenant`, `decision`, `reason` | Every `POST /v1/reservations/{id}/extend` outcome. |
3838
| `cycles.reservations.expired` | `cycles_reservations_expired_total` | Counter | `tenant` | Each reservation the expiry sweep actually marks EXPIRED. Skipped reservations (still in grace, already finalised) do not increment. |
3939
| `cycles.events` | `cycles_events_total` | Counter | `tenant`, `decision`, `reason`, `overage_policy` | Every `POST /v1/events` outcome. |

how-to/security-hardening.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ Restrict the Cycles service account to the key patterns the services actually us
8888

8989
```conf
9090
# redis.conf
91-
user cycles on >${REDIS_PASSWORD} ~tenant:* ~budget:* ~budgets:* ~reservation:* ~reserve:* ~idem:* ~apikey:* ~apikeys:* ~policy:* ~policies:* ~audit:* ~event:* ~events:* ~delivery:* ~deliveries:* ~dispatch:* ~webhook:* ~webhooks:* ~evidence:* ~config:* ~replay:* +@all
91+
user cycles on >${REDIS_PASSWORD} ~tenant:* ~budget:* ~budgets:* ~reservation:* ~reserve:* ~idem:* ~idempotency:* ~apikey:* ~apikeys:* ~policy:* ~policies:* ~audit:* ~event:* ~events:* ~delivery:* ~deliveries:* ~dispatch:* ~webhook:* ~webhooks:* ~evidence:* ~config:* ~replay:* +@all
9292
user default off
9393
```
9494

incidents/retry-storms-and-idempotency-failures.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -244,6 +244,7 @@ This limits total retry spend per document to the workflow budget, regardless of
244244
Before each retry, use `decide` to check if budget is available without creating a reservation. This avoids creating reservations you'll immediately release:
245245

246246
```python
247+
import os
247248
import uuid
248249
249250
from runcycles import (

quickstart/deploying-the-full-cycles-stack.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -403,7 +403,7 @@ export CYCLES_API_KEY="cyc_live_..." # paste the key from the response
403403

404404
## Step 4: Create a budget
405405

406-
Create a budget ledger for the tenant. Without a budget, all reservations will be denied with `BUDGET_EXCEEDED`:
406+
Create a budget ledger for the tenant. Without a budget at any derived scope, reservations fail with `404 NOT_FOUND` ("Budget not found for provided scope"):
407407

408408
```bash
409409
curl -s -X POST http://localhost:7979/v1/admin/budgets \
@@ -683,9 +683,9 @@ requests.post(f"{CYCLES_URL}/v1/reservations/{reservation_id}/commit", json={
683683
684684
## Troubleshooting
685685
686-
### "BUDGET_EXCEEDED" on first reservation
686+
### "NOT_FOUND" (no budget) or "BUDGET_EXCEEDED" on first reservation
687687
688-
No budget exists for the scope. Create a budget ledger via the admin API (Step 4). Every scope in the subject hierarchy needs an allocated budget.
688+
`404 NOT_FOUND` means no budget exists at any derived scope — create a budget ledger via the admin API (Step 4); at least one scope in the subject hierarchy needs an allocated budget. `409 BUDGET_EXCEEDED` means a budget exists but the estimate exceeds what remains.
689689
690690
### "UNAUTHORIZED" or 401
691691

0 commit comments

Comments
 (0)