fix(gateway): stop reconnect storms — per-socket state, stability-keyed backoff, identify budget

[christianlappin]

Problem

Two related failure modes, one root cause:

1. Reconnect storm. On 2026-07-22 a single deployment made 276,379 gateway connections in one day (baseline ~1,700). Discord force-reset the bot token after the identify limit was exceeded. Log signature: bursts of identical "Connecting to Discord Gateway" lines, 85× close code 4005 "Already authenticated", 1,107× 4008 "Rate limited".
2. Never-stable baseline. Even in "healthy" months, every connection was closed by Discord with code 1000 (no reason) ~50 seconds after connect, around the clock — that's where the ~1,700/day baseline came from. Supporting evidence from months of logs: ~993k "Heartbeat ACK not received" warnings, ~89.7k RESUMED vs ~1.8k READY.

Root cause

connectGateway() kept ws, heartbeatInterval, and heartbeatAckTimeout as single closure variables shared by every socket generation, and the jittered first-heartbeat setTimeout in startHeartbeat() was never tracked:

• After any op 7 / 1001 reconnect, the old socket's onclose still fired and its cleanup() cleared the new socket's heartbeat timers; stale generations kept clobbering the live socket's heartbeat pipeline, so Discord zombie-closed it ~one heartbeat interval later (the ~50s cycle).
• Each stale onclose / ack-timeout scheduled its own untracked setTimeout(connect) — under stress these parallel loops multiplied exponentially (the storm).
• Backoff was ineffective: consecutiveFailures reset on any READY, and storm sockets did reach READY before dying, so every reconnect ran at the 5s floor.
• Storm reconnects identified with resume:false every time; Discord resets the token at 1000 identifies/24h.

Fix

• Per-socket state (GatewaySocket): each connection owns its WebSocket, heartbeat timers (including the previously untracked first-beat jitter timeout), and awaiting-ack flag. Teardown detaches handlers, so a superseded socket can never log, heartbeat, touch session state, or schedule reconnects.
• Single pending reconnect timer — first scheduler wins; parallel loops are structurally impossible.
• Stability-keyed exponential backoff: 5s doubling to a 15-min cap with 50–100% jitter; resets only after a connection stays READY/RESUMED for 60s (not on READY itself).
• Close-code handling: 4008 → 30-min cooldown and backoff pinned to the cap; 4007/4009 → drop session, re-identify; fatal codes (4004, 4010–4014) → stop permanently.
• Identify budget (500 per rolling 24h, half of Discord's token-reset threshold): when exhausted the gateway stops reconnecting, logs an error, and reports through plugin health via a new optional onPermanentFailure hook (wired to runtimeHealth in worker.ts) instead of burning the token.

Testing

• 12 new regression tests in tests/gateway-reconnect.test.ts (fake WebSocket + fake timers): loop-multiplication guards, single pending reconnect, heartbeat ownership across generations, backoff growth/cap/stability-reset, 4008 cooldown, fatal codes, identify budget, rolling-window policy.
• Full suite: 488/488 passing on this branch; tsc build clean.
• Deployed to the affected instance: single "Gateway ready" (generation 1), zero closes/reconnects/heartbeat warnings in the observation window — first time the connection survived past ~50s in months.

🤖 Generated with Claude Code

[christianlappin]

Pushed a follow-up commit after ~22h in production: one handshake got a WebSocket error event while still CONNECTING and no close event ever followed, so onclose-driven recovery never ran and the gateway hung silently. The new commit adds a 60s connect watchdog (socket must reach READY/RESUMED or it is torn down and retried with normal backoff) and treats onerror on a non-OPEN socket as terminal. Both paths route through the same single-pending-reconnect scheduler; two regression tests added.

🤖 Generated with Claude Code