incidents.md

Atlas API -- Incident Log

(Fixture for /stability and /postmortem. Stands in for an incident tracker export, e.g. PagerDuty/Statuspage, or a pasted incident log.)

Incident 1 -- 2026-04-22, connection pool exhaustion (first occurrence)

• Detected: 09:41 (latency alert)
• Resolved: 09:58 (connection pool size increased)
• Severity: Sev-2
• Postmortem: None written. Fixed and closed same day.

Incident 2 -- 2026-05-30, export job stuck queue

• Detected: 16:02 (customer-reported)
• Resolved: 17:10 (manually requeued stuck jobs)
• Severity: Sev-2
• Postmortem: None written. Fixed and closed same day.

Incident 3 -- 2026-06-18, auth token expiry mass logout

• Detected: 08:15 (spike in support tickets)
• Resolved: 09:00 (token TTL config rolled back)
• Severity: Sev-2
• Postmortem: None written. Fixed and closed same day.

Incident 4 -- 2026-07-14, connection pool exhaustion (recurrence)

• Detected: 14:02 -- alert fires: p99 latency > 5s
• Acknowledged: 14:06 -- on-call engineer acks
• Root cause identified: 14:19 -- billing webhook retry loop (PR #491) has no backoff/cap, exhausts the connection pool under a single slow downstream call
• Mitigated: 14:24 -- pool size increased, retry loop disabled
• Fully resolved: 15:10 -- fix deployed (PR #508 follow-up), retry loop rewritten with exponential backoff
• Severity: Sev-2
• Customer impact: ~12 minutes of degraded API latency (p99 5-8s), all customers, no data loss, no failed payments (webhook is idempotent)
• Postmortem: Not yet written -- this is the incident to feed /postmortem for a worked example. Note this is the second connection-pool incident in 90 days (see Incident 1).

Load / breaking-point testing

None on record. Current ~500 req/s capacity estimate is based on Incident 4 traffic levels at the time, not a deliberate load test.