fix(core): Wait for detached sub-workflows before the worker stops its task runner (backport to release-candidate/2.37.x)

[n8n-assistant]

Description

Backport of #37737 to release-candidate/2.37.x.

Checklist for the author (@sovietspaceship) to go through.

• Review the backport changes
• Fix possible conflicts
• Merge to target branch

After this PR has been merged, it will be picked up in the next patch release for release track.

Original description

Summary

A worker that receives SIGTERM loses a sub-workflow started with "wait for sub-workflow completion" turned off. The child execution fails at its next Code node with Task request timed out. The parent is recorded as success, so only the child carries the error.

The worker runs its @OnShutdown hooks in priority order, and only calls stopProcess() after all of them finish. ScalingService.stopWorker() runs first, at the highest priority, but waits only on getRunningJobsCount(). A detached child has no queue job of its own, so that wait returns at once. TaskRunnerModule.stop() then shuts the task runner down at the default priority. ActiveExecutions.shutdown() does wait for the child, but it runs from stopProcess(), after every hook, so it waits for an execution whose task runner is already gone. The child asks the draining task broker for a task, the broker refuses it immediately, and the requester reports that refusal as a timeout.

This PR waits for the running in-process executions in the same hook that waits for queued jobs, so the task runner stays available until they finish.

1. ActiveExecutions gains getRunningExecutionIds() and cancelRunningExecutions(), and ExecutionRepository gains cancelManyRunning().
2. ScalingService.stopWorker() waits on both sources. The queued-job wait keeps its previous behaviour and stays unbounded. Only the new wait carries a budget, 80% of the graceful shutdown timeout, the same heuristic the task-broker drain uses.
3. Past the budget, the executions that are still running are cancelled, so they report an accurate status instead of failing later against a task runner that is gone. Queued jobs hold the loop open, so this happens only once the queue is empty.
4. The executions are recorded as cancelled before they are cancelled, and that write is awaited. The engine writes the final status of a cancelled execution from a hook it does not await, which raced the database connection close and lost, leaving the execution running with no stoppedAt. The engine's later write is conditional on the execution not being cancelled, so it becomes a no-op. The write matches on status as well as ID, so an execution that reached a terminal status after the ID snapshot is left alone. Its deadline is half of what the shutdown window still has, capped at three seconds, so it cannot outlive the force-exit timer.
5. QUEUE_WORKER_TIMEOUT now sets both the field that arms the force-exit timer and the field the drains budget against. Before, it moved one and not the other, so a drain could plan a window longer than the timer that ends the process. It also rejects a malformed value rather than reading a leading number out of it, matching how the config layer parses N8N_GRACEFUL_SHUTDOWN_TIMEOUT.

Net effect: a detached sub-workflow that finishes inside the shutdown window now completes, and the worker exits cleanly.

Notes for reviewers:

• This widens the window; it does not close it. A detached child may legitimately run for minutes, because resolveSubworkflowDeadline gives it a deadline independent of its parent. A child that outlives the budget still fails.
• The worker drain and the task-broker drain each budget 80% of the same window, against one force-exit timer. Left as is, and worth a follow-up: the shutdown window wants one absolute deadline, computed once.
• Cancellation cannot tell a detached child from an awaited one. Cancelling an awaited child leaves its parent waiting until the force-exit timer, after which its queue job is re-delivered.
• Main and webhook instances have the same ordering defect and are not addressed here.

How to test

Unit tests: pnpm --filter=n8n test src/scaling/__tests__/scaling.service.test.ts src/__tests__/active-executions.test.ts src/commands/__tests__/worker.test.ts.

Integration test for the new repository method: pnpm --filter=n8n test:integration test/integration/database/repositories/execution.repository.test.ts.

Manual, in queue mode with a worker:

1. Import a parent with a Webhook and an Execute Sub-workflow node with "wait for sub-workflow completion" turned off, and a child of two slow HTTP nodes followed by a Code node. Activate the parent.
2. Fire the webhook. The parent finishes in well under a second; the child keeps running on the worker.
3. Wait about 6 seconds, so the child is inside its first HTTP node.
4. Send SIGTERM to the worker.

Expected: the worker logs Waiting for 1 in-process executions to finish..., the task runner stops only after the child finishes, and the child completes with success.

Verified on a real worker. Before: the child failed at the Code node with Task request timed out and elapsedSeconds 0, and Failed to get Redis context followed. After: the child finished success in 21.0s, and the runner stopped 141ms after it. An ordinary queued execution drains as it did before.

The expiry path was verified separately, with N8N_GRACEFUL_SHUTDOWN_TIMEOUT=10 against a child needing about 20s. The drain waits 8s, logs the timeout and the cancelled execution IDs, and the child is recorded canceled with a stoppedAt. The worker exits 0 without reaching the force-exit path.

Related Linear tickets, Github issues, and Community forum posts

https://linear.app/n8n/issue/CAT-4349

Review / Merge checklist

• I have seen this code, I have run this code, and I take responsibility for this code.
• PR title and summary are descriptive. (conventions)
• Docs updated or follow-up ticket created.
• Tests included.
• PR Labeled with Backport to Beta, Backport to Stable, or Backport to v1 (if the PR is an urgent fix that needs to be backported)

[n8n-assistant]

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-37737-to-release-candidate/2.37.x
git worktree add --checkout .worktree/backport-37737-to-release-candidate/2.37.x backport-37737-to-release-candidate/2.37.x
cd .worktree/backport-37737-to-release-candidate/2.37.x
git reset --hard HEAD^
git cherry-pick -x afc4a54798bbed080dad6b68231655a34d4ed14d
git push --force-with-lease