Skip to content

Commit cd91603

Browse files
committed
chore(release): gate the Artur sync on green source CI and a recorded rehearsal
1 parent 435151f commit cd91603

7 files changed

Lines changed: 441 additions & 2 deletions

File tree

.github/workflows/sync-artur-release.yml

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,10 @@ on:
1616
permissions:
1717
contents: read
1818
pull-requests: read
19+
# Reading the CI verdict of the released commit needs the Actions API. The
20+
# Artur App token minted later in this job is scoped to ai-workflow-arthur
21+
# and cannot read this repository, so the source CI gate uses the job token.
22+
actions: read
1923

2024
concurrency:
2125
group: artur-release-sync
@@ -76,6 +80,42 @@ jobs:
7680
run: |
7781
echo "target_sha=$(jq -r '.targetSourceCommit' .release-notes/approved-source.json)" >> "$GITHUB_OUTPUT"
7882
echo "notes_pr=$(jq -r '.releaseNotesPullRequest' .release-notes/approved-source.json)" >> "$GITHUB_OUTPUT"
83+
- name: Require green source CI at the released commit
84+
env:
85+
GH_TOKEN: ${{ github.token }}
86+
TARGET_SHA: ${{ steps.source.outputs.target_sha }}
87+
run: |
88+
# Only push runs judge main. A manual dispatch or a merge-group run
89+
# can share this SHA, and dispatches cancel each other through the CI
90+
# concurrency group, so counting them would let a stray cancelled run
91+
# block a release the push run already passed.
92+
runs="$(gh api --paginate --slurp \
93+
"repos/Blazity/ai-workflow/actions/workflows/ci.yml/runs?head_sha=$TARGET_SHA&per_page=100" \
94+
| jq '[.[].workflow_runs[] | select(.event == "push") | {status, conclusion, url: .html_url}]')"
95+
if [[ "$(jq 'length' <<<"$runs")" -eq 0 ]]; then
96+
echo "No CI push run exists for $TARGET_SHA, so the released commit was never verified on main. Missing CI is not a pass: run CI on that commit, and release only once it is green." >&2
97+
exit 1
98+
fi
99+
pending="$(jq -r '[.[] | select(.status != "completed") | .url] | join(", ")' <<<"$runs")"
100+
if [[ -n "$pending" ]]; then
101+
echo "CI has not finished at $TARGET_SHA: $pending. Wait for the run to complete, then rerun this synchronization from Actions." >&2
102+
exit 1
103+
fi
104+
# A cancelled, skipped, failed, timed-out or action_required push
105+
# run all land here. Only an explicit success releases.
106+
failed="$(jq -r '[.[] | select(.conclusion != "success") | "\(.conclusion // "no conclusion") \(.url)"] | join("; ")' <<<"$runs")"
107+
if [[ -n "$failed" ]]; then
108+
echo "CI did not succeed at $TARGET_SHA: $failed. Fix the source, cut the release from a commit whose CI is green, and do not ship over a red main." >&2
109+
exit 1
110+
fi
111+
- name: Require a recorded green rehearsal at the released commit
112+
env:
113+
VERSION: ${{ steps.release.outputs.version }}
114+
TARGET_SHA: ${{ steps.source.outputs.target_sha }}
115+
run: |
116+
pnpm release-notes validate-rehearsal \
117+
--version "$VERSION" \
118+
--source-commit "$TARGET_SHA"
79119
- name: Check out immutable source snapshot
80120
uses: actions/checkout@v4
81121
with:

docs/releases/artur/README.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,9 @@ its Vercel production deployment and smoke tests succeed.
1414
edit the non-technical English wording where needed, and approve it. Before
1515
merging, run the tenant-database check in
1616
[`upgrade-preflight.md`](upgrade-preflight.md). Merge only after the check
17-
identifies no unrepaired deployed workflows.
17+
identifies no unrepaired deployed workflows. Rehearse the pinned
18+
`targetSourceCommit` and merge its record first; synchronization refuses to
19+
run without one, see [`rehearsals/README.md`](rehearsals/README.md).
1820
3. The merge automatically runs **Sync Approved Artur Release**. It copies the
1921
complete application tree from the pinned `targetSourceCommit` into a new
2022
`release/artur-<version>` branch in `Blazity/ai-workflow-arthur`.
Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# Artur release rehearsals
2+
3+
A rehearsal is one real end-to-end run on our own production instance, at the
4+
exact source commit the release ships, against a repository whose pre-PR checks
5+
have the same shape as the client's: a toolchain setup phase followed by long
6+
running checks. The run must reach the **open pull request** node green.
7+
8+
**Sync Approved Artur Release** refuses to touch `Blazity/ai-workflow-arthur`
9+
until a rehearsal record for the released version exists here and matches the
10+
released commit. Two releases have already gone out over a red source `main`,
11+
one of them broke the client, because snapshot integrity and an approved
12+
release-note pull request were the only gates.
13+
14+
## Order of operations
15+
16+
The order is not free. `targetSourceCommit` is frozen into the release note's
17+
YAML front matter when **Prepare Artur Release** runs, and the synchronization
18+
gate compares the rehearsal against that frozen SHA. A rehearsal recorded
19+
before preparation cannot match it, because merging the rehearsal record itself
20+
moves the tip of `main`. Follow this sequence:
21+
22+
1. Run **Actions → Prepare Artur Release** for the version. It pins
23+
`targetSourceCommit` into `docs/releases/artur/<version>.md`.
24+
2. Read the pinned SHA out of the front matter of that generated note. That is
25+
the SHA to rehearse, and nothing else passes the gate.
26+
3. Deploy that exact SHA to our own production and run the rehearsal (below).
27+
4. Commit `docs/releases/artur/rehearsals/<version>.json` with `sourceCommit`
28+
set to the pinned SHA, and merge it in its own pull request. It cannot ride
29+
along with the release note: `validate-source` requires the release-note
30+
pull request to change exactly one file, the note itself.
31+
5. Merge the release-note pull request. That merge starts the synchronization,
32+
which reads the rehearsal record from `main` and finds both gates satisfied.
33+
34+
**Run preparation once.** Re-running it after other commits land moves the
35+
pinned target, and a rehearsal recorded against the old target no longer
36+
matches, so the synchronization aborts until you rehearse the new SHA.
37+
38+
## Running one
39+
40+
1. Make sure production runs a deployment built from the pinned SHA.
41+
2. Dispatch a workflow on production against a fixture repository with real
42+
check commands, for example `Blazity/aiw-checks-fixture`. Its checks must
43+
install a toolchain and then run the real suite. Stub commands such as
44+
`echo ok` do not count, see the floor below.
45+
3. Watch the run to the **open pull request** node. If it stops earlier, or
46+
stops on a clarification, there is no rehearsal: fix the cause and run again.
47+
4. Read the duration of the checks phase from the run trace and record the run.
48+
49+
## Recording one
50+
51+
Write `docs/releases/artur/rehearsals/<version>.json`, with every field present
52+
and no extra fields:
53+
54+
```json
55+
{
56+
"version": "2026.08.8",
57+
"sourceCommit": "0123456789abcdef0123456789abcdef01234567",
58+
"runId": "wrun_01K5NRQ8ABCDEF",
59+
"runUrl": "https://ai-workflow.blazity.com/runs/wrun_01K5NRQ8ABCDEF",
60+
"repository": "Blazity/aiw-checks-fixture",
61+
"checksDurationSec": 1180,
62+
"outcome": "success",
63+
"recordedAt": "2026-08-19T10:00:00Z",
64+
"recordedBy": "someone@blazity.com"
65+
}
66+
```
67+
68+
| Field | Meaning |
69+
| --- | --- |
70+
| `version` | The version being released, matching the file name. |
71+
| `sourceCommit` | 40 lowercase hex characters, the pinned `targetSourceCommit`. |
72+
| `runId` | The production run id, `wrun_...`. |
73+
| `runUrl` | Absolute link to that run, so a reviewer can reopen it. |
74+
| `repository` | `owner/name` of the rehearsed repository. |
75+
| `checksDurationSec` | Whole seconds the checks phase took, from the trace. |
76+
| `outcome` | Exactly `success`, nothing else passes. |
77+
| `recordedAt` | ISO 8601 UTC timestamp, such as `2026-08-19T10:00:00Z`. |
78+
| `recordedBy` | The person who ran the rehearsal and can answer for it. |
79+
80+
To check a record before merging anything:
81+
82+
```bash
83+
pnpm release-notes validate-rehearsal --version <version> --source-commit <target_sha>
84+
```
85+
86+
## Why checks must run at least 300 seconds
87+
88+
Vercel allows 300 seconds per invocation. The client outage came from a check
89+
batch that outlived a single invocation, so a rehearsal that finishes inside
90+
one invocation never reaches the failure mode that actually bit the client. A
91+
run whose checks finished in 12 seconds because they were `echo ok` stubs
92+
proves the pipeline can open a pull request and proves nothing else.
93+
94+
This is not hypothetical. The client configuration "worked" for weeks precisely
95+
because its heaviest repository had no check commands configured at all, so
96+
nothing ever ran long enough to cross the limit until it did.
97+
98+
A rehearsal below the floor is rejected. Do not lower the floor to make a
99+
release pass: give the fixture repository checks that really take longer than
100+
five minutes.
101+
102+
## Worked example of a run that must NOT be recorded
103+
104+
Run `wrun_01M0CGC9GEMEBC3THA2DNBECNJ` on our production reproduced the client
105+
failure. Its `checks` node lived 358631 ms and then died with:
106+
107+
```
108+
Step "step//./src/workflows/agent//runPrePrChecksStep" failed after 0 retries: terminated
109+
```
110+
111+
358 seconds is above the 300 second floor, so duration alone would have let
112+
this through. The run never reached the **open pull request** node, so its
113+
`outcome` is not `success` and the gate rejects it. Duration proves the checks
114+
were long enough to be worth believing; `outcome` proves they actually passed.
115+
Both fields are required, and neither substitutes for the other. Recording this
116+
run with `"outcome": "success"` would be a false statement about production,
117+
and it is exactly the shape of release that broke the client.

scripts/release-notes/cli.ts

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ import { collectRelease } from "./collect.js";
1010
import { generateReleaseDraft } from "./generate.js";
1111
import { parseVersion } from "./classify.js";
1212
import { validateApprovedSourceRelease } from "./manifest.js";
13+
import { validateRehearsalEvidence } from "./rehearsal.js";
1314
import { extractShareableNotes, renderReleaseNotes } from "./render.js";
1415
import {
1516
findUnbackportedDestinationCommits,
@@ -197,6 +198,22 @@ async function validateSourceCommand(
197198
return validation;
198199
}
199200

201+
async function validateRehearsalCommand(argv: string[]): Promise<unknown> {
202+
const version = parseVersion(requiredArg(argv, "version"));
203+
const rehearsalPath = path.resolve(
204+
arg(
205+
argv,
206+
"rehearsal",
207+
path.join("docs", "releases", "artur", "rehearsals", `${version}.json`),
208+
),
209+
);
210+
return validateRehearsalEvidence({
211+
version,
212+
sourceCommit: requiredArg(argv, "source-commit"),
213+
rehearsalPath,
214+
});
215+
}
216+
200217
async function shareableCommand(argv: string[]): Promise<unknown> {
201218
const version = parseVersion(requiredArg(argv, "version"));
202219
const notesPath = path.resolve(
@@ -397,14 +414,15 @@ export async function runCli(argv: string[], deps: CliDeps = {}): Promise<unknow
397414
"guard-artur": () => guardArturCommand(argv),
398415
"validate-source": () =>
399416
validateSourceCommand(argv, deps.validate ?? validateApprovedSourceRelease),
417+
"validate-rehearsal": () => validateRehearsalCommand(argv),
400418
"sync-artur": () => syncArturCommand(argv, deps),
401419
shareable: () => shareableCommand(argv),
402420
};
403421
const command = argv[0] ?? "";
404422
const execute = commands[command];
405423
if (!execute) {
406424
throw new Error(
407-
"Usage: pnpm release-notes <prepare|guard-artur|validate-source|sync-artur|shareable> [options]",
425+
"Usage: pnpm release-notes <prepare|guard-artur|validate-source|validate-rehearsal|sync-artur|shareable> [options]",
408426
);
409427
}
410428
return execute();
Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
import assert from "node:assert/strict";
2+
import { mkdtemp, writeFile } from "node:fs/promises";
3+
import os from "node:os";
4+
import path from "node:path";
5+
import test from "node:test";
6+
7+
import { runCli } from "./cli.js";
8+
import { validateRehearsalEvidence } from "./rehearsal.js";
9+
10+
const version = "2026.08.8";
11+
const sourceCommit = "b".repeat(40);
12+
13+
function rehearsal(overrides: Record<string, unknown> = {}) {
14+
return {
15+
version,
16+
sourceCommit,
17+
runId: "wrun_01K5NRQ8ABCDEF",
18+
runUrl: "https://ai-workflow.blazity.com/runs/wrun_01K5NRQ8ABCDEF",
19+
repository: "Blazity/aiw-checks-fixture",
20+
checksDurationSec: 1180,
21+
outcome: "success",
22+
recordedAt: "2026-08-19T10:00:00Z",
23+
recordedBy: "someone@blazity.com",
24+
...overrides,
25+
};
26+
}
27+
28+
async function recordPath(contents: string): Promise<string> {
29+
const directory = await mkdtemp(path.join(os.tmpdir(), "artur-rehearsal-"));
30+
const file = path.join(directory, `${version}.json`);
31+
await writeFile(file, contents);
32+
return file;
33+
}
34+
35+
test("accepts a rehearsal of the released commit whose checks outlived one invocation", async () => {
36+
const rehearsalPath = await recordPath(JSON.stringify(rehearsal()));
37+
const record = await validateRehearsalEvidence({ version, sourceCommit, rehearsalPath });
38+
assert.equal(record.runId, "wrun_01K5NRQ8ABCDEF");
39+
assert.equal(record.checksDurationSec, 1180);
40+
});
41+
42+
test("accepts checks that last exactly the one-invocation limit", async () => {
43+
const rehearsalPath = await recordPath(JSON.stringify(rehearsal({ checksDurationSec: 300 })));
44+
await assert.doesNotReject(validateRehearsalEvidence({ version, sourceCommit, rehearsalPath }));
45+
});
46+
47+
test("blocks a release with no recorded rehearsal", async () => {
48+
const directory = await mkdtemp(path.join(os.tmpdir(), "artur-rehearsal-missing-"));
49+
await assert.rejects(
50+
validateRehearsalEvidence({
51+
version,
52+
sourceCommit,
53+
rehearsalPath: path.join(directory, `${version}.json`),
54+
}),
55+
/No rehearsal is recorded for 2026\.08\.8/,
56+
);
57+
});
58+
59+
test("blocks a rehearsal record that is not valid JSON", async () => {
60+
const rehearsalPath = await recordPath("{ not json");
61+
await assert.rejects(
62+
validateRehearsalEvidence({ version, sourceCommit, rehearsalPath }),
63+
/is not valid JSON/,
64+
);
65+
});
66+
67+
test("blocks a rehearsal record with a missing field", async () => {
68+
const { checksDurationSec: _omitted, ...withoutDuration } = rehearsal();
69+
const rehearsalPath = await recordPath(JSON.stringify(withoutDuration));
70+
await assert.rejects(
71+
validateRehearsalEvidence({ version, sourceCommit, rehearsalPath }),
72+
/does not match the required shape.*checksDurationSec/s,
73+
);
74+
});
75+
76+
test("blocks a rehearsal record with a malformed commit, timestamp or URL", async () => {
77+
const rehearsalPath = await recordPath(
78+
JSON.stringify(
79+
rehearsal({ sourceCommit: "B".repeat(40), recordedAt: "19-08-2026", runUrl: "runs/1" }),
80+
),
81+
);
82+
await assert.rejects(
83+
validateRehearsalEvidence({ version, sourceCommit, rehearsalPath }),
84+
/sourceCommit.*runUrl.*recordedAt/s,
85+
);
86+
});
87+
88+
test("blocks a rehearsal of a different source commit", async () => {
89+
const rehearsalPath = await recordPath(
90+
JSON.stringify(rehearsal({ sourceCommit: "c".repeat(40) })),
91+
);
92+
await assert.rejects(
93+
validateRehearsalEvidence({ version, sourceCommit, rehearsalPath }),
94+
/ran at source commit cccc.*ships bbbb/s,
95+
);
96+
});
97+
98+
test("blocks a rehearsal that did not end in success", async () => {
99+
const rehearsalPath = await recordPath(JSON.stringify(rehearsal({ outcome: "failed" })));
100+
await assert.rejects(
101+
validateRehearsalEvidence({ version, sourceCommit, rehearsalPath }),
102+
/ended with outcome "failed"/,
103+
);
104+
});
105+
106+
test("blocks a rehearsal whose checks never crossed one invocation", async () => {
107+
const rehearsalPath = await recordPath(JSON.stringify(rehearsal({ checksDurationSec: 299 })));
108+
await assert.rejects(
109+
validateRehearsalEvidence({ version, sourceCommit, rehearsalPath }),
110+
/299 seconds, below the 300 second floor/,
111+
);
112+
});
113+
114+
test("blocks a rehearsal whose checks were stubs", async () => {
115+
const rehearsalPath = await recordPath(JSON.stringify(rehearsal({ checksDurationSec: 12 })));
116+
await assert.rejects(
117+
validateRehearsalEvidence({ version, sourceCommit, rehearsalPath }),
118+
/12 seconds, below the 300 second floor/,
119+
);
120+
});
121+
122+
test("validate-rehearsal CLI reads the recorded rehearsal for the released version", async () => {
123+
const rehearsalPath = await recordPath(JSON.stringify(rehearsal()));
124+
await assert.doesNotReject(
125+
runCli([
126+
"validate-rehearsal",
127+
"--version",
128+
version,
129+
"--source-commit",
130+
sourceCommit,
131+
"--rehearsal",
132+
rehearsalPath,
133+
]),
134+
);
135+
await assert.rejects(
136+
runCli(["validate-rehearsal", "--version", version, "--rehearsal", rehearsalPath]),
137+
/Missing required argument: --source-commit/,
138+
);
139+
});

0 commit comments

Comments
 (0)