Skip to content

Commit 0526be1

Browse files
feat(server): status compare-and-set + locked-baseline patch + terminal-reopen gate (RBR-929/950/951/953)
Consolidates the three RBR-929 step branches (RBR-950/951/953) that were verified passing on rbr953-terminal-reopen-gate but never merged to fork/master, cherry-picked and rebased onto current tip. RBR-950 (AC1/AC2/AC4): opt-in status compare-and-set on issuesSvc.update. `expectedStatus`/`expectedStatuses` becomes a SQL WHERE predicate (inArray(issues.status, ...)) on the write itself, so a losing write from a stale snapshot affects zero rows instead of clobbering. A losing CAS throws `conflict` (409), distinguishable from `null` (not-found). RBR-951 (AC3): the row-lock half of the fix, minus the blocked-bookkeeping test (rbr951-locked-baseline-patch.test.ts) and its assignee-derived cleanup logic, which depend on unblockDescriptor/blockedTransitionAt/ blockedOwnerNotifiedAt columns from an unrelated, unmerged upstream PR (paperclipai#10112) not present in this schema. Kept applyLockedBaselinePatchFields for the assignee-derived lock-clearing fields (checkoutRunId/executionRunId/ executionAgentNameKey/executionLockedAt), which are real columns here, and the re-assertion of assertTransition against the locked read inside runUpdate (the actual RBR-864 fix: a stale non-terminal snapshot cannot regress a row that has since committed done). RBR-953: gates terminal (done/cancelled) -> non-terminal transitions behind an explicit allowTerminalReopen opt-in, independent of the CAS opt-in. This is the actual fix for the RBR-864 class of incident (recovery reconciler reverting a completed issue back to blocked from a stale snapshot). allowTerminalReopen is granted to the 3 legitimate reopen callers that exist in this codebase: board/human comment reopen and the explicit PATCH /issues/:id status-write path (server/src/routes/issues.ts), and the deferred-comment-wake reopen (server/src/services/heartbeat.ts). A 4th caller from the original RBR-953 diff, the status-card compile reopen (server/src/services/status-cards.ts), does not exist in this codebase and was dropped. Also grants it to the task-watchdog revive path (server/src/services/task-watchdogs.ts), present here as `shouldReopen`. Test currency fix: the RBR-950 "stays opt-in" test asserted a plain `svc.update(id, { status: "blocked" })` with no CAS keys could still clobber `done`. That was true when RBR-950 landed alone but is stale now that RBR-953's assertTransition gate refuses *any* ungated done -> blocked write regardless of CAS keys. Split into two tests: the original opt-in behaviour now scoped to non-terminal -> non-terminal transitions (still true), plus a new test asserting the CAS opt-in and the terminal-reopen opt-in are independent gates (per RBR-1103 review). Verified on this branch (embedded Postgres, not July-era production): - rbr929-update-status-cas.test.ts: 9/9 pass (was 7/8 with the stale assertion; now 9/9 with the split test) - rbr953-terminal-reopen-gate.test.ts: 10/10 pass - tsc --noEmit: identical error set to unmodified fork/master (adapter-acpx module resolution + 2 pre-existing plugin-host-services.ts errors), confirmed by diffing against a clean fork/master worktree with the same node_modules — this diff introduces zero new typecheck errors. RBR-929, RBR-950, RBR-951, RBR-953, RBR-1103 Co-authored-by: Paperclip <noreply@paperclip.ing>
1 parent c9645f3 commit 0526be1

6 files changed

Lines changed: 746 additions & 9 deletions

File tree

Lines changed: 262 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,262 @@
1+
import { randomUUID } from "node:crypto";
2+
import { eq } from "drizzle-orm";
3+
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
4+
import {
5+
activityLog,
6+
agents,
7+
companies,
8+
createDb,
9+
executionWorkspaces,
10+
goals,
11+
heartbeatRuns,
12+
instanceSettings,
13+
issueComments,
14+
issueInboxArchives,
15+
issueRelations,
16+
issues,
17+
projectWorkspaces,
18+
projects,
19+
} from "@paperclipai/db";
20+
import {
21+
getEmbeddedPostgresTestSupport,
22+
startEmbeddedPostgresTestDatabase,
23+
} from "./helpers/embedded-postgres.js";
24+
import { issueService } from "../services/issues.ts";
25+
import { HttpError } from "../errors.ts";
26+
27+
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
28+
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
29+
30+
/**
31+
* RBR-950 (RBR-929 STEP 1): `issuesSvc.update` supports an opt-in status
32+
* compare-and-set enforced in SQL.
33+
*
34+
* The bug this exists for is RBR-864: the recovery path reads an issue, decides
35+
* from that snapshot that the run was lost, and writes `blocked` -- but by the
36+
* time the write lands the run has already committed `done`. The snapshot-derived
37+
* write silently reverts a completed issue.
38+
*
39+
* The assertions below are behavioural: they read the row back and assert on
40+
* `issues.status`, not on the shape of the return value.
41+
*/
42+
describeEmbeddedPostgres("issuesSvc.update status compare-and-set (RBR-929 AC1/AC2/AC4)", () => {
43+
let db!: ReturnType<typeof createDb>;
44+
let svc!: ReturnType<typeof issueService>;
45+
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
46+
47+
beforeAll(async () => {
48+
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issues-status-cas-");
49+
db = createDb(tempDb.connectionString);
50+
svc = issueService(db);
51+
// No inline hook budget: vitest.config.ts owns hookTimeout (RBR-912/RBR-945).
52+
// An inline argument silently overrides both the config and --hookTimeout.
53+
});
54+
55+
afterEach(async () => {
56+
await db.delete(issueComments);
57+
await db.delete(issueRelations);
58+
await db.delete(issueInboxArchives);
59+
await db.delete(activityLog);
60+
await db.delete(issues);
61+
await db.delete(heartbeatRuns);
62+
await db.delete(executionWorkspaces);
63+
await db.delete(projectWorkspaces);
64+
await db.delete(projects);
65+
await db.delete(goals);
66+
await db.delete(agents);
67+
await db.delete(instanceSettings);
68+
await db.delete(companies);
69+
});
70+
71+
afterAll(async () => {
72+
await tempDb?.cleanup();
73+
});
74+
75+
async function seedInProgressIssue() {
76+
const companyId = randomUUID();
77+
const agentId = randomUUID();
78+
const issueId = randomUUID();
79+
80+
await db.insert(companies).values({
81+
id: companyId,
82+
name: "Paperclip",
83+
issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
84+
requireBoardApprovalForNewAgents: false,
85+
});
86+
await db.insert(agents).values({
87+
id: agentId,
88+
companyId,
89+
name: "CodexCoder",
90+
role: "engineer",
91+
status: "active",
92+
adapterType: "codex_local",
93+
adapterConfig: {},
94+
runtimeConfig: {},
95+
permissions: {},
96+
});
97+
await db.insert(issues).values({
98+
id: issueId,
99+
companyId,
100+
title: "Status compare-and-set",
101+
status: "in_progress",
102+
priority: "medium",
103+
assigneeAgentId: agentId,
104+
});
105+
106+
return { companyId, agentId, issueId };
107+
}
108+
109+
async function readStatus(issueId: string) {
110+
return await db
111+
.select({ status: issues.status })
112+
.from(issues)
113+
.where(eq(issues.id, issueId))
114+
.then((rows) => rows[0]?.status ?? null);
115+
}
116+
117+
it("a stale-snapshot `blocked` write loses to a committed `done` write -- `done` survives", async () => {
118+
const { issueId } = await seedInProgressIssue();
119+
120+
// The recovery path's snapshot: it saw the issue as `in_progress`.
121+
const snapshot = await readStatus(issueId);
122+
expect(snapshot).toBe("in_progress");
123+
124+
// The run completes and commits `done` first.
125+
await svc.update(issueId, { status: "done" });
126+
expect(await readStatus(issueId)).toBe("done");
127+
128+
// The recovery path now writes `blocked`, still reasoning from its stale
129+
// snapshot. The CAS predicate must make this write affect zero rows.
130+
await expect(
131+
svc.update(issueId, { status: "blocked", expectedStatus: snapshot! }),
132+
).rejects.toMatchObject({ status: 409 });
133+
134+
// The behavioural assertion that matters: `done` survived.
135+
expect(await readStatus(issueId)).toBe("done");
136+
});
137+
138+
it("under real concurrency exactly one of two CAS writes lands", async () => {
139+
const { issueId } = await seedInProgressIssue();
140+
141+
// Both writers race from the same `in_progress` snapshot. The predicate is
142+
// evaluated by the same statement that writes, so the second to acquire the
143+
// row lock sees `in_progress` already gone and matches zero rows.
144+
const results = await Promise.allSettled([
145+
svc.update(issueId, { status: "done", expectedStatus: "in_progress" }),
146+
svc.update(issueId, { status: "blocked", expectedStatus: "in_progress" }),
147+
]);
148+
149+
const fulfilled = results.filter((r) => r.status === "fulfilled");
150+
const rejected = results.filter((r) => r.status === "rejected");
151+
expect(fulfilled).toHaveLength(1);
152+
expect(rejected).toHaveLength(1);
153+
expect((rejected[0] as PromiseRejectedResult).reason).toMatchObject({ status: 409 });
154+
155+
// Whichever won, the row is not left in the pre-race status.
156+
expect(await readStatus(issueId)).not.toBe("in_progress");
157+
});
158+
159+
it("distinguishes a losing CAS (throws conflict) from issue-not-found (returns null)", async () => {
160+
const { issueId } = await seedInProgressIssue();
161+
162+
// Not found: `null`, no throw. This is the signal `update` already used, so
163+
// the CAS cannot reuse it (RBR-929 AC2).
164+
await expect(
165+
svc.update(randomUUID(), { status: "blocked", expectedStatus: "in_progress" }),
166+
).resolves.toBeNull();
167+
168+
// Losing CAS on a row that definitely exists: a distinct, loud signal.
169+
const error = await svc
170+
.update(issueId, { status: "blocked", expectedStatus: "done" })
171+
.then(() => null)
172+
.catch((caught: unknown) => caught);
173+
expect(error).toBeInstanceOf(HttpError);
174+
expect(error).toMatchObject({
175+
status: 409,
176+
details: { issueId, actualStatus: "in_progress" },
177+
});
178+
179+
// The losing write did not touch the row.
180+
expect(await readStatus(issueId)).toBe("in_progress");
181+
});
182+
183+
it("a winning CAS applies the write", async () => {
184+
const { issueId } = await seedInProgressIssue();
185+
186+
const updated = await svc.update(issueId, { status: "done", expectedStatus: "in_progress" });
187+
expect(updated?.status).toBe("done");
188+
expect(await readStatus(issueId)).toBe("done");
189+
});
190+
191+
it("accepts the plural form and matches any listed status", async () => {
192+
const { issueId } = await seedInProgressIssue();
193+
194+
const updated = await svc.update(issueId, {
195+
status: "done",
196+
expectedStatuses: ["todo", "in_progress"],
197+
});
198+
expect(updated?.status).toBe("done");
199+
expect(await readStatus(issueId)).toBe("done");
200+
201+
// ...and rejects when the actual status is not in the list.
202+
await expect(
203+
svc.update(issueId, { status: "blocked", expectedStatuses: ["todo", "in_progress"] }),
204+
).rejects.toMatchObject({ status: 409 });
205+
expect(await readStatus(issueId)).toBe("done");
206+
});
207+
208+
it("rejects an empty expectedStatuses instead of silently no-oping every write", async () => {
209+
const { issueId } = await seedInProgressIssue();
210+
211+
await expect(
212+
svc.update(issueId, { status: "done", expectedStatuses: [] }),
213+
).rejects.toMatchObject({ status: 422 });
214+
expect(await readStatus(issueId)).toBe("in_progress");
215+
});
216+
217+
it("stays opt-in: a caller that passes neither CAS key is unguarded between non-terminal statuses", async () => {
218+
const { issueId } = await seedInProgressIssue();
219+
220+
// No CAS keys -> no status predicate -> the ~100 existing callers keep
221+
// exactly the behaviour they had for non-terminal transitions.
222+
const updated = await svc.update(issueId, { status: "blocked" });
223+
expect(updated?.status).toBe("blocked");
224+
expect(await readStatus(issueId)).toBe("blocked");
225+
});
226+
227+
it("RBR-953: the CAS opt-in and the terminal-reopen opt-in are independent gates — omitting CAS keys does not bypass the terminal-regression gate", async () => {
228+
const { issueId } = await seedInProgressIssue();
229+
230+
await svc.update(issueId, { status: "done" });
231+
232+
// Stale assumption superseded by RBR-953 (CEO ruling on the RBR-929 AC3
233+
// open question): `assertTransition` refuses *any* ungated done -> blocked
234+
// write, independent of whether CAS keys were supplied, unless the caller
235+
// opts in with `allowTerminalReopen`. A caller with no CAS keys is not
236+
// exempt from this gate; the two opt-ins are orthogonal.
237+
await expect(svc.update(issueId, { status: "blocked" })).rejects.toMatchObject({
238+
status: 409,
239+
details: expect.objectContaining({ code: "issue_terminal_status_regression" }),
240+
});
241+
expect(await readStatus(issueId)).toBe("done");
242+
243+
// The explicit opt-in still allows it, with or without CAS keys.
244+
const reopened = await svc.update(issueId, { status: "blocked", allowTerminalReopen: true });
245+
expect(reopened?.status).toBe("blocked");
246+
expect(await readStatus(issueId)).toBe("blocked");
247+
});
248+
249+
it("never writes the CAS keys as columns", async () => {
250+
const { issueId } = await seedInProgressIssue();
251+
252+
// If either key leaked into `issueData` it would reach `patch` and Drizzle
253+
// would fail on an unknown column, so a clean write is the assertion.
254+
const updated = await svc.update(issueId, {
255+
title: "Renamed under CAS",
256+
expectedStatus: "in_progress",
257+
expectedStatuses: ["in_progress"],
258+
});
259+
expect(updated?.title).toBe("Renamed under CAS");
260+
expect(await readStatus(issueId)).toBe("in_progress");
261+
});
262+
});

0 commit comments

Comments
 (0)