fix(server): eagerly cancel stale queued runs on issue reassignment (RENA-55610)

[rendedennis6-byte]

Thinking Path

• Paperclip is the open source app people use to manage AI agents for work.
• The heartbeat scheduler queues a run for an agent, then claims it later when a concurrency slot opens.
• When an issue changes assignee, the old assignee's queued run stays queued. The lazy check in claimQueuedRun only looks at it once the run reaches the front of the queue with a free slot. Behind a busy queue, that can take hours. Until then, the stale run still holds the issue's execution lock and blocks the new assignee.
• This gap needed an eager cancel path that runs right after reassignment, plus a sweep that also runs when the agent has zero free slots.
• The first version of the eager cancel path read the run's state, then wrote "cancelled" with a plain UPDATE ... WHERE id = runId. A concurrent claimQueuedRun could flip the same run to "running" in between. The plain write would then overwrite a live run's status to "cancelled" and strand active work on a terminal record.
• This pull request closes that window. It guards the cancel write with a compare-and-set on status = 'queued', and it skips every cancel side effect when the guard matches zero rows. It also adds a targeted regression test for exactly this window.
• The benefit: reassignment still eagerly frees the previous assignee's queue slot and execution lock, but a run that wins the claim race in the same instant is never marked cancelled while it is actually executing.

Linked Issues or Issue Description

No matching public GitHub issue was found (see "Duplicate/related PR search" below), so this section follows the bug report template fields directly.

What happened?
Reassigning an issue away from an agent left that agent's queued heartbeat run alive. The run kept the issue's executionRunId lock until the lazy staleness check in claimQueuedRun happened to reach it, which can take hours behind a busy queue.

Expected behavior
A queued run that is made stale by reassignment (or by the issue reaching a terminal status, or by another change covered by the existing staleness rules) should be cancelled promptly, and the issue's execution lock should be released, without waiting for the run to reach the front of the queue.

Steps to reproduce

1. Assign an issue to agent A, so a heartbeat run for agent A is queued and holds the issue's executionRunId.
2. Reassign the issue to agent B before agent A's run starts (for example, while agent A's queue is backed up or its concurrency slots are full).
3. Observe that agent A's queued run, and the execution lock it holds, do not clear until the queue eventually reaches that run.

This PR adds cancelStaleQueuedRunsForIssue (called from the reassignment path) and a zero-free-slot sweep in startNextQueuedRunForAgent, so both cases are handled eagerly instead of lazily. This update on top of that also closes a claim-race window a reviewer found in the eager cancel write itself (see "What Changed").

Duplicate/related PR search
Searched open PRs in this repo for related work on queued-run staleness and cancellation. No exact duplicate of this change. Related but distinct open PRs in the same area, from other contributors:

• fix(server): reap stale queued execution locks safely #10449 — reap stale queued execution locks safely
• fix(heartbeat): expire runs stuck in queue beyond a bounded age #10211 — expire runs stuck in queue beyond a bounded age
• fix(heartbeat): pre-enqueue terminal-issue skip + bulk stale queued-run cancel API #6163 — pre-enqueue terminal-issue skip + bulk stale queued-run cancel API
• fix(heartbeat): cancel stale queued runs and suppress toast for internal cancels #3533 — cancel stale queued runs and suppress toast for internal cancels
• fix(heartbeat): cross-agent slot handoff + stale-row starvation guard for company concurrency ceiling #9757 — cross-agent slot handoff + stale-row starvation guard for company concurrency ceiling

What Changed

• Guarded the stale-queued-run cancel write in cancelQueuedRunForStaleIssue (server/src/services/heartbeat.ts) with a compare-and-set on status = 'queued', using the same atomic pattern as setRunStatusIfRunning/setRunStatusFromLive.
• When the guard matches zero rows (the run already left "queued", most likely because a concurrent claimQueuedRun promoted it to "running"), the function now returns without touching the wakeup request, without clearing the issue's execution lock, and without appending a run event. The run that won the claim race is left untouched.
• Added HeartbeatServiceOptions.beforeQueuedRunCancelWrite, a test-only hook invoked immediately before the guarded write. Production call sites do not set it, so behavior is unchanged in production. Tests use it to land a concurrent status flip deterministically inside the exact window the guard protects.
• Added a regression test, heartbeat-stale-queue-invalidation.test.ts → "does not cancel a queued run that wins the claim race between staleness evaluation and the cancel write", covering this exact window.
• Fixed a separate, pre-existing test-double gap the CI run for this PR exposed: three route test files (issue-agent-mutation-ownership-routes.test.ts, issue-assignee-invokability-routes.test.ts, issue-comment-reopen-routes.test.ts) hoist their own mockHeartbeatService double and did not define cancelStaleQueuedRunsForIssue. The reassignment branch in routes/issues.ts (added earlier on this PR, guarded by its own .catch) calls that method on every PATCH that changes assigneeAgentId; against the incomplete mock, the call threw synchronously before the .catch could run, turning any reassignment-path test into a 500. Added the missing mock (vi.fn(async () => [])) to every hoisted mockHeartbeatService in server/src/__tests__ that mounts routes/issues.ts (14 files total — 3 that were actually exercising the broken path, plus 11 more with the same latent gap that had not yet hit a reassignment test case).

Verification

• corepack pnpm exec tsc --noEmit in server/ — clean, no errors (confirmed by the "Typecheck + Release Registry" CI check on this branch; local tsc in this sandbox reports unrelated @paperclipai/plugin-sdk resolution errors from an intentionally partial pnpm install --filter @paperclipai/server..., not from this diff).
• corepack pnpm exec vitest run src/__tests__/heartbeat-stale-queue-invalidation.test.ts in server/ (embedded Postgres) — 27/27 passing (26 pre-existing + 1 new); also confirmed green in CI's "General tests (server)" shards for this head commit.
• After the mock fix: corepack pnpm exec vitest run --no-file-parallelism --maxWorkers=1 src/__tests__/issue-comment-reopen-routes.test.ts src/__tests__/issue-agent-mutation-ownership-routes.test.ts src/__tests__/issue-assignee-invokability-routes.test.ts (the CI "Verify serialized server suites" flags) — 174/174 passing, versus 3 files / 8+ tests failing with 500 before the fix.
• Manual trace of the new test: seeds a queued run whose issue was reassigned (so evaluateQueuedRunStaleness reports it stale), injects a concurrent status flip to "running" right before the cancel write via the new test hook, then asserts the run stays "running", the wakeup request stays "queued" (not "skipped"), the issue keeps its executionRunId, and no run event is appended.

Risks

• Low risk. The change only adds a WHERE status = 'queued' guard to a write that was previously unconditional, and only skips side effects when that guard already tells us the run is no longer queued. It cannot cause an eligible stale run to stay queued longer than before; it only stops a live run from being wrongly marked cancelled.
• No schema or migration changes.
• The new beforeQueuedRunCancelWrite option is optional and unset by every production call site (heartbeatService(db) is constructed without it outside tests), so there is no behavior change outside the test suite.
• This PR's branch name and title predate this update and include an internal ticket-style identifier, which CONTRIBUTING.md → "No Internal Issue References" / "Branch Naming" asks contributors to avoid. Renaming the branch would require opening a new PR against a renamed head, which conflicts with keeping this update on the existing open PR. This description itself avoids internal ticket references and only links public GitHub PR numbers above.

Model Used

Anthropic Claude, model id claude-sonnet-5, running inside Claude Code (Claude Agent SDK) with tool use enabled (shell/git commands, file read/edit, GitHub REST API calls). The runtime does not expose an explicit extended-thinking-mode flag or a context-window figure to this session, so neither is stated as a specific number.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above
• I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
• I have not referenced internal/instance-local Paperclip issues or links (only public GitHub #NNN / github.com/paperclipai/paperclip URLs)
• My branch name describes the change (e.g. docs/..., fix/...) and contains no internal Paperclip ticket id or instance-derived details — see "Risks" above for why this one is not fixed here
• I have run tests locally and they pass
• I have added or updated tests where applicable
• I have updated relevant documentation to reflect my changes (none applicable — no user-facing or API-surface change)
• I have considered and documented any risks above
• All Paperclip CI gates are green
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups
• I will address all Greptile and reviewer comments before requesting merge

[commitperclip]

✅ All checks passing — ready for Greptile review and maintainer approval.

— commitperclip

[greptile-apps]

Greptile Summary

The PR eagerly cancels queued heartbeat runs made stale by issue reassignment and adds a scheduler sweep that performs stale-run cleanup even when an agent has no free concurrency slots.

• Adds issue-scoped stale queued-run cancellation after reassignment.
• Sweeps obvious stale candidates before checking available execution slots.
• Adds embedded-Postgres tests for reassignment cleanup and zero-slot queues.

Confidence Score: 4/5

The concurrent claim-and-cancel race should be fixed before merging because eager cleanup can mark an already executing run as cancelled.

The new post-commit cleanup evaluates a queued snapshot and later performs an unconditional status update, while the normal scheduler can transition that same row to running in between.

Files Needing Attention: server/src/services/heartbeat.ts

Important Files Changed

Filename	Overview
server/src/services/heartbeat.ts	Adds both eager and scheduler-driven stale-run cancellation, but the eager path can race a concurrent claim and overwrite a running status.
server/src/routes/issues.ts	Invokes best-effort stale-run cleanup after a committed assignee change.
server/src/tests/heartbeat-stale-queue-invalidation.test.ts	Covers reassignment and zero-slot cleanup behavior, but not concurrent claiming during eager cancellation.

Prompt To Fix All With AI

### Issue 1
server/src/services/heartbeat.ts:12835
**Prevent claimed-run cancellation**

If reassignment cleanup overlaps a scheduler claim, `claimQueuedRun` can transition the selected row to `running` before this call reaches the unconditional status update. The cleanup then marks an executing run as cancelled and its wakeup as skipped, leaving active work attached to a terminal run record; cancellation must require that the run is still queued.

### Issue 2
server/src/routes/issues.ts:9002-9008
**Complete the required PR description**

The description explains the change and verification, but it omits the required Thinking Path and Risks sections, and the model note lacks the exact model version, context window, and capability details. Please update it to follow the PR template so maintainers have the required rationale and risk assessment for this concurrency-sensitive change.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "fix(server): eagerly cancel stale queued..." | Re-trigger Greptile}

[greptile-apps]

Prevent claimed-run cancellation

If reassignment cleanup overlaps a scheduler claim, claimQueuedRun can transition the selected row to running before this call reaches the unconditional status update. The cleanup then marks an executing run as cancelled and its wakeup as skipped, leaving active work attached to a terminal run record; cancellation must require that the run is still queued.

Knowledge Base Used: Agent Execution Runtime

Prompt To Fix With AI

This is a comment left during a code review.
Path: server/src/services/heartbeat.ts
Line: 12835

Comment:
**Prevent claimed-run cancellation**

If reassignment cleanup overlaps a scheduler claim, `claimQueuedRun` can transition the selected row to `running` before this call reaches the unconditional status update. The cleanup then marks an executing run as cancelled and its wakeup as skipped, leaving active work attached to a terminal run record; cancellation must require that the run is still queued.

**Knowledge Base Used:** [Agent Execution Runtime](https://app.greptile.com/paperclip-org-3/-/custom-context/knowledge-base/paperclipai/paperclip/-/docs/agent-execution-runtime.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Complete the required PR description

The description explains the change and verification, but it omits the required Thinking Path and Risks sections, and the model note lacks the exact model version, context window, and capability details. Please update it to follow the PR template so maintainers have the required rationale and risk assessment for this concurrency-sensitive change.

Context Used: CONTRIBUTING.md has a guide for a good PR message ... (source)

Prompt To Fix With AI

This is a comment left during a code review.
Path: server/src/routes/issues.ts
Line: 9002-9008

Comment:
**Complete the required PR description**

The description explains the change and verification, but it omits the required Thinking Path and Risks sections, and the model note lacks the exact model version, context window, and capability details. Please update it to follow the PR template so maintainers have the required rationale and risk assessment for this concurrency-sensitive change.

**Context Used:** CONTRIBUTING.md has a guide for a good PR message ... ([source](https://app.greptile.com/review/custom-context?memory=instruction-0))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!