Keep workflow runs alive across WebSocket disconnects - #4713
Merged
Conversation
Chat turns already survive a dropped socket via the process-wide ChatTurnRegistry (seq-stamped replay buffer, detach grace, resume_chat). Workflow jobs did not: activeJobs is per-connection, disconnect() cancelled every running job, and reconnect_job from a fresh connection misreported a completed run as failed with "replay unavailable". Mirror the chat design for job runs: - New JobRunRegistry (packages/websocket/src/job-run-registry.ts): process-wide sessions keyed by user+job, frames stamped with job_seq into a bounded buffer, detach grace (NODETOOL_JOB_DETACH_GRACE_MS, 10 min -> cancel) and post-finish retention (NODETOOL_JOB_REPLAY_RETENTION_MS, 5 min), plus execution hooks so a later connection can cancel, push stream input, or update node properties on an adopted run. - disconnect() detaches running jobs instead of cancelling them; queued runs are still drained and cancelled. - reconnect_job/resume_job accept last_seq, adopt the session, and reply with a job_resumed header followed by the missed tail. The persisted-row fallback now echoes only settled statuses; anything non-terminal with no session reports failed instead of hanging the client. - Concurrency caps count runs process-wide through the registry, so detached runs cannot bypass MAX_CONCURRENT_JOBS across reconnects. - Web WorkflowRunner tracks a job_seq cursor, resends reconnect_job with last_seq on the socket's open event for running jobs, and handles job_resumed (including settling a store whose replay is already complete). - Reliability journey client-reconnect-mid-run now asserts the resumed stream and real terminal state instead of pinning the old gap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P6LtxKGYqd7tnK9MdopkHb
The reliability Ring 1 gate (user-journeys.yml, added with the journeys in #4660) has been red since it landed, blocking every Fly deploy — including the rejected-upgrade fd-leak fix (#4673) whose absence in prod exhausts the machine's 10240 file descriptors roughly daily. Three of the four failures are real ws-server bugs; the fourth is the gate mis-invoking a journey. - cancelJob no longer sends an eager terminal job_update for an active run. That out-of-band frame told clients the job was over while nodes still read "running" (lifecycle.running-after-job-terminal — the mid-run-cancel-node and mid-run-cancel-streaming journeys), which is what left canvas nodes stuck spinning after Stop. The DB row is still marked cancelled immediately; the kernel's own terminal frame relays through the drain loop after the node-level terminals. - The generation autosave dedupe read (Asset.paginate) in the drain loop is best-effort now. On a DB-free run it threw "Database not initialized", killed the drain loop, and failed a run that had actually completed (provider-failure-mid-stream journey). - reliability-ring1.mjs gives ws-transport-faults its own per-fault block on its declared ws-server surface, like python-node-workflow: the ws faults all wrap one proxy, so the bare invocation applied all five at once, and the three black-hole faults legitimately end client-side as "timeout" — a verdict the CLI cannot express as success — and stay covered by the harness's own ws-transport-faults.test.ts. All 15 Ring 1 runs pass locally; websocket suite 185 files / 2119 tests green; harness vitest suite green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P6LtxKGYqd7tnK9MdopkHb
georgi
pushed a commit
that referenced
this pull request
Aug 5, 2026
…oss machines The job-run resilience shipped in #4713 is per-process: a run's replay buffer and control hooks live in the one server executing it. Behind a load balancer a reconnecting client lands on a random instance, so resume silently degrades and cancel misses. Two mechanisms fix that, both inert when no instance id is configured (NODETOOL_INSTANCE_ID, falling back to Fly's FLY_MACHINE_ID): - Owner-aware routing. The executing instance stamps its id on the Job row (new nullable runner_instance column, both dialects + migration). A reconnecting client appends resume_job=<id> to the handshake URL; an instance that does not own that still-running job answers the upgrade with `fly-replay: instance=<owner>` so Fly's proxy re-issues the handshake at the owner. A replayed request (fly-replay-src) is never replayed again, the stored owner id is validated before touching response bytes, the refused socket is destroyed on a timer if the peer never closes, and the client retires the hint after two consecutive failed connects so a dead owner cannot lock the shared socket out. - Cross-instance cancel. cancel_job/stop for a run stamped by another instance marks the row cancelled with a conditional update (only while non-terminal — it cannot clobber the owner's completed outcome) and publishes on a control bus: Postgres LISTEN/NOTIFY on nodetool_job_control, in-process emitter under SQLite. LISTEN needs a session-pooled/direct connection, so the listener opens its own client from NODETOOL_JOB_CONTROL_DATABASE_URL / DIRECT_URL / DATABASE_DIRECT_URL; and because a pooler can eat notifications silently, each instance also polls its own running sessions' rows (NODETOOL_JOB_CANCEL_POLL_MS, default 15s) and cancels any that read cancelled — the bus makes cancels immediate, the poll makes them certain. Rows with no runner_instance (HTTP/trigger/MCP runs, which hold no session anywhere) keep the old not-found answer. The web WebSocketManager gains a per-connect urlProvider (with a generation guard so teardown during URL resolution cannot orphan a socket); GlobalWebSocketManager builds the resume hint from the running workflow stores. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P6LtxKGYqd7tnK9MdopkHb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two related changes to the WebSocket run lifecycle:
user-journeys.yml→fly-deploy.yml) has been red since it landed in fix(docker): unbreak the image build, and block PRs on it #4660, blocking every Fly deploy, including the rejected-upgrade fd-leak fix (fix(websocket): close the socket of a rejected WebSocket upgrade #4673) whose absence in prod exhausts the machine's file descriptors roughly daily (CLOSE_WAIT pile-up → connection resets → outage).Run resilience (commit 1)
Previously, each connection's
UnifiedWebSocketRunnercancelled every running job indisconnect(), andreconnect_jobfrom a fresh connection misreported a completed run asfailedwith "Job event replay is unavailable".Mirrors the
ChatTurnRegistrydesign (resume_chat) for job runs:Server (
packages/websocket,packages/protocol)JobRunRegistry/JobRunSession(job-run-registry.ts): process-wide sessions keyed by user+job. Every job frame is stamped with a monotonicjob_seqand appended to a bounded buffer (NODETOOL_JOB_REPLAY_BUFFER_EVENTS, default 2000). A running session nobody is attached to is cancelled after a detach grace (NODETOOL_JOB_DETACH_GRACE_MS, default 10 min); a finished session is kept for replay (NODETOOL_JOB_REPLAY_RETENTION_MS, default 5 min).disconnect()detaches running jobs instead of cancelling them. Queued (never-started) runs are still drained and cancelled.reconnect_job/resume_jobacceptlast_seq, adopt the session onto the new connection, and reply with ajob_resumedheader followed by the missed tail. Cross-connection control (cancel_job,stop,stream_input,end_input_stream,update_node_properties) routes through the session's execution hooks.completed/failed/cancelled); any non-terminal row with no session reportsfailedinstead of leaving the client spinning — and a genuinely completed job is no longer flipped tofailed.MAX_CONCURRENT_JOBS, per-workflow cap) count runs process-wide through the registry, so detached runs can't be bypassed by reconnecting.Web (
web/src/stores)WorkflowRunnertracks a per-jobjob_seqcursor, resendsreconnect_jobwithlast_seqon the socket'sopenevent for in-flight server runs (skipping browser-local runs), and handlesjob_resumed— including settling a store whose replay is already complete so it can't hang in "running".Ring 1 deploy-gate fixes (commit 2)
cancelJobno longer sends an eager terminaljob_updatefor an active run. That out-of-band frame told clients the job was over while nodes still read "running" (lifecycle.running-after-job-terminal, pinned by themid-run-cancel-node/mid-run-cancel-streamingjourneys) — the bug that left canvas nodes stuck spinning after Stop. The DB row is still marked cancelled immediately; the kernel's own terminal frame relays through the drain loop after node-level terminals.Asset.paginate) in the drain loop is best-effort. On a DB-free run it threw "Database not initialized", killed the drain loop, and failed a run that had actually completed (provider-failure-mid-streamjourney).reliability-ring1.mjsgivesws-transport-faultsits own per-fault block on its declared ws-server surface (likepython-node-workflow): its ws faults all wrap one proxy, so the bare invocation applied all five simultaneously, and the three black-hole faults legitimately end client-side as "timeout" — covered by the harness's ownws-transport-faults.test.tsinstead.Testing
npm run reliability:ring1), including the four that fail on main today.job-run-registry.test.ts(14 tests) andjob-run-resilience.test.ts(8 runner-level tests: detach + cross-connection replay incl. terminal frame, cross-connection cancel/control, slot accounting across connections, fallback statuses).workflowUpdates.jobResume.test.tsplus extendedWorkflowRunner.test.tson the web side (cursor tracking, open-event resubscribe,job_resumedpaths,isBrowserRunreset).client-reconnect-mid-runreliability journey now asserts the fixed behavior: replayed node stream,job_resumed{status:"finished"}, realcompletedterminal state.packages/websocket185 files / 2119 tests, reliability harness vitest 153 tests, web suites green; typecheck and lint clean (mobile typecheck fails pre-existing in this sandbox — its Expo tree isn't installed).An adversarial review pass was run on the diff; the surviving accepted limitations are documented in code: one delivery target per session (a second tab takes the stream, same as chat), and the delivery chain relies on the bounded buffer for memory, not socket backpressure.
🤖 Generated with Claude Code
https://claude.ai/code/session_01P6LtxKGYqd7tnK9MdopkHb