Multi-instance job control: route reconnects to the owner, cancel across machines

[georgi]

What

Makes the job-run resilience from #4713 work across multiple server instances. Today the replay buffer, cancel hooks, and adoption all live in the one process executing a run — behind Fly's load balancer with 2+ machines, a reconnecting client lands on a random instance, resume silently degrades to the persisted-row fallback, and cancel misses a live run entirely.

Both mechanisms are inert when no instance id is configured: identity comes from NODETOOL_INSTANCE_ID, falling back to Fly's FLY_MACHINE_ID, else null (single-machine mode — current prod — is behaviorally unchanged, confirmed by the full Ring 1 suite).

Owner-aware reconnect routing (fly-replay)

• The executing instance stamps its id on the Job row — new nullable runner_instance column in both schemas plus a guarded additive migration.
• A reconnecting client appends resume_job=<job_id> to the WebSocket handshake URL (via a new per-connect urlProvider on WebSocketManager; GlobalWebSocketManager computes the hint from the running workflow stores).
• An instance that doesn't own that still-running job answers the upgrade with a raw 409 + fly-replay: instance=<owner> (hijack + write + end, per the ws-upgrade.ts discipline; a 5s unref'd timer destroys the socket if the peer never closes), and Fly's proxy re-issues the whole handshake at the owner.
• Bounded and hardened: a request already replayed (fly-replay-src) is never replayed again; the stored owner id is validated (/^[A-Za-z0-9_-]+$/) before touching response bytes; job lookups are user-scoped (no cross-user existence oracle); and the client retires the hint after two consecutive failed connects, so a row stamped with a machine a deploy retired can't lock the tab's shared socket out — the third attempt connects bare and falls back to reconnect_job's persisted-row answer.

Cross-instance cancel (row + poll)

• cancel_job/stop for a run stamped by a different instance marks the row cancelled with a conditional update (Job.markCancelledIfActive — only while non-terminal, so it can't clobber the owner's completed outcome and cost accounting in a race).
• The row is the only transport: each instance polls its own running sessions' rows (NODETOOL_JOB_CANCEL_POLL_MS, default 15s, 0 disables — one indexed IN query per tick, skipped when idle) and cancels any that read cancelled. Worst-case cross-instance cancel latency is one poll interval; a cancel on the machine that holds the run reaches the session hooks directly and stays immediate. (An earlier revision also carried a Postgres LISTEN/NOTIFY bus for immediacy; it was dropped — it required a direct/session-pooled connection that Supabase's transaction pooler can't provide, and the row poll alone is sufficient and unconditionally reliable.)
• Rows with no runner_instance (HTTP POST /run, trigger, MCP runs — nothing holds a session for those anywhere) keep the old "Job not found or already completed" answer instead of a false "cancellation requested".

Also: the urlProvider await is guarded by a connect-generation counter so a teardown during URL resolution can't orphan a live socket.

Testing

• New suites: job-cancel.test.ts (conditional cancel incl. losing the race to a completed write), job-multi-instance.test.ts (11 cases — stamping, foreign-owned cancel via the row, poller pickup, unstamped/terminal rows keep the not-found answer), fly-replay-upgrade.test.ts (8 cases incl. auth ordering, fly-replay-src, terminal/same-instance fallthrough), GlobalWebSocketManager.resumeJob.test.ts, plus WebSocketManager teardown-during-resolve cases.
• An adversarial review pass ran on the diff; all 7 findings (dead-owner lockout, orphan socket, cancel over-claim/row clobber, half-open fd, header injection, and two more) were fixed and re-verified.
• Full runs: models 50 files / 798 tests, websocket 188 files / 2162 tests, web 134 suites / 1838 tests, npm run reliability:ring1 all 15 green, typecheck (models/websocket/web) and lint clean.

Deployment notes

Scaling to 2+ machines needs only fly scale count 2+ — no database configuration changes. Docs: docs/websocket-api.md § Multi-Instance Deployments.

🤖 Generated with Claude Code

https://claude.ai/code/session_01P6LtxKGYqd7tnK9MdopkHb