Skip to content

Commit 25a6950

Browse files
docs: preserve circuit breaker guide
1 parent ba8e69f commit 25a6950

1 file changed

Lines changed: 143 additions & 10 deletions

File tree

docs/starknet/circuit-breaker.md

Lines changed: 143 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,147 @@
1-
# Circuit-breaker metrics
1+
# Starknet Circuit Breaker
2+
3+
## Overview
4+
5+
The circuit breaker pattern prevents cascading failures and reduces latency by failing fast when an RPC endpoint is unhealthy. Each Starknet RPC endpoint is protected by its own circuit breaker instance that tracks failures and manages state transitions.
6+
7+
## States
8+
9+
### CLOSED (Normal Operation)
10+
- All calls pass through normally
11+
- Recent failures are tracked in a rolling time window
12+
- Transitions to OPEN when failures reach the configured threshold
13+
14+
### OPEN (Fail-Fast)
15+
- All calls are immediately rejected with `CircuitOpenError`
16+
- No actual RPC calls are made, saving timeout costs
17+
- After a cooldown period, transitions to HALF_OPEN to test recovery
18+
19+
### HALF_OPEN (Recovery Probe)
20+
- A single probe call is allowed through to test endpoint health
21+
- Success: Accumulates toward `successThreshold` to close the circuit
22+
- Failure: Immediately reopens the circuit and resets the cooldown
23+
24+
## Configuration
25+
26+
Circuit breaker behavior is controlled via environment variables:
27+
28+
```bash
29+
# Number of failures before circuit opens (default: 5)
30+
CIRCUIT_BREAKER_FAILURE_THRESHOLD=5
31+
32+
# Number of successes needed to close from HALF_OPEN (default: 2)
33+
CIRCUIT_BREAKER_SUCCESS_THRESHOLD=2
34+
35+
# Cooldown period before attempting recovery (default: 30000ms = 30s)
36+
CIRCUIT_BREAKER_COOLDOWN_MS=30000
37+
38+
# Rolling time window for counting failures (default: 60000ms = 60s)
39+
CIRCUIT_BREAKER_WINDOW_MS=60000
40+
```
41+
42+
## Integration with Failover
43+
44+
The circuit breaker composes with the existing RPC failover logic:
45+
46+
1. **Circuit Open**: When a circuit is open, `invokeWithFailover` immediately skips that endpoint and tries the next one in the failover order
47+
2. **Failover Triggered**: If the primary endpoint's circuit opens, subsequent calls automatically use the healthy secondary endpoint
48+
3. **Recovery**: After the cooldown, the circuit attempts recovery with a probe call
49+
50+
This combination provides both fast failure detection (circuit breaker) and automatic endpoint switching (failover).
51+
52+
## Monitoring
53+
54+
Circuit breaker state is exposed in the diagnostics endpoint (`GET /diagnostics/events`):
55+
56+
```json
57+
{
58+
"circuitBreakers": [
59+
{
60+
"endpointUrl": "https://rpc.example.com",
61+
"state": "CLOSED",
62+
"recentFailureCount": 0,
63+
"openedAt": null
64+
}
65+
]
66+
}
67+
```
68+
69+
### State Fields
70+
71+
- **endpointUrl**: The RPC endpoint URL
72+
- **state**: Current circuit state (`CLOSED`, `OPEN`, or `HALF_OPEN`)
73+
- **recentFailureCount**: Number of failures in the current rolling window
74+
- **openedAt**: Timestamp when circuit opened (milliseconds since epoch), or `null` if closed
75+
76+
### Metrics
277

378
The Starknet client exposes two process-local metrics for each RPC endpoint:
479

5-
- `starknet_circuit_breaker_state`, a gauge where `0=CLOSED`, `1=OPEN`, and
6-
`2=HALF_OPEN`.
7-
- `starknet_circuit_breaker_transitions_total`, a counter labeled with the
8-
endpoint and transition name, such as `CLOSED_to_OPEN`.
80+
- `starknet_circuit_breaker_state`, a gauge where `0=CLOSED`, `1=OPEN`, and `2=HALF_OPEN`.
81+
- `starknet_circuit_breaker_transitions_total`, a counter labeled with the endpoint and transition name, such as `CLOSED_to_OPEN`.
82+
83+
Alert when an endpoint remains `OPEN` (state `1`) for longer than the normal RPC recovery window, or when `CLOSED_to_OPEN` transitions repeat rapidly. A useful starting threshold is state `1` for more than five minutes, adjusted for the configured circuit-breaker cooldown and the number of healthy fallback endpoints.
84+
85+
## Example Scenarios
86+
87+
### Scenario 1: Endpoint Degrades
88+
1. RPC endpoint starts returning timeouts
89+
2. After 5 failures in 60 seconds, circuit opens
90+
3. Subsequent calls skip this endpoint immediately
91+
4. Failover to secondary endpoint succeeds
92+
5. After 30 seconds, circuit attempts recovery
93+
6. 2 successful probe calls close the circuit
94+
7. Endpoint returns to normal rotation
95+
96+
### Scenario 2: Temporary Network Blip
97+
1. 2 failures occur due to temporary network issue
98+
2. Circuit remains CLOSED (below threshold of 5)
99+
3. After 60 seconds, these failures age out of the window
100+
4. No circuit opening occurs
101+
102+
### Scenario 3: All Endpoints Down
103+
1. Primary circuit opens after 5 failures
104+
2. Failover to secondary
105+
3. Secondary also fails 5 times and circuit opens
106+
4. All circuits open, final request throws error
107+
5. After cooldown, both circuits attempt recovery
108+
6. First endpoint to recover closes its circuit and handles requests
109+
110+
## Implementation Details
111+
112+
### Failure Tracking
113+
114+
Failures are tracked with timestamps in a rolling window. Old failures outside the window are automatically pruned, ensuring the circuit only reacts to recent patterns.
115+
116+
```typescript
117+
// Example: 3 failures at t=0, t=30s, t=120s with 60s window
118+
// At t=120s: Only the t=30s and t=120s failures count (2 total)
119+
// The t=0 failure is outside the 60s window and is pruned
120+
```
121+
122+
### Half-Open Behavior
123+
124+
In HALF_OPEN state, only one probe call is allowed through at a time. This prevents a “thundering herd” of simultaneous recovery attempts that could overwhelm a recovering endpoint.
125+
126+
### Success Threshold Rationale
127+
128+
The `successThreshold` of 2 (default) prevents premature circuit closure from a single lucky success. Two consecutive successes provide more confidence that the endpoint has genuinely recovered.
129+
130+
## Testing
131+
132+
The circuit breaker is tested at two levels:
133+
134+
1. **Unit Tests** (`circuit-breaker.test.ts`): Test state transitions, threshold logic, and time windows
135+
2. **Integration Tests** (`client.test.ts`): Test interaction with failover, proxy behavior, and end-to-end flows
136+
137+
All tests achieve >95% coverage as required by project guidelines.
138+
139+
## Security Considerations
140+
141+
The circuit breaker improves security posture by:
142+
143+
1. **Reducing Attack Surface**: Open circuits prevent repeated calls to potentially compromised endpoints
144+
2. **DoS Mitigation**: Fail-fast behavior prevents resource exhaustion from hanging calls
145+
3. **Observable State**: Circuit state is exposed in diagnostics for operator visibility
9146

10-
Alert when an endpoint remains `OPEN` (state `1`) for longer than the normal
11-
RPC recovery window, or when `CLOSED_to_OPEN` transitions repeat rapidly. A
12-
useful starting threshold is state `1` for more than five minutes, adjusted
13-
for the configured circuit-breaker cooldown and the number of healthy fallback
14-
endpoints.
147+
Circuit state does not leak sensitive information (e.g., request contents or user data) and is safe to expose in admin-only diagnostics endpoints.

0 commit comments

Comments
 (0)