Skip to content

fix(ci): gate the release before the draft exists - #1711

Merged
FelixTJDietrich merged 1 commit into
mainfrom
fix/release-evidence-gate-ordering
Sep 1, 2026
Merged

fix(ci): gate the release before the draft exists#1711
FelixTJDietrich merged 1 commit into
mainfrom
fix/release-evidence-gate-ordering

Conversation

@FelixTJDietrich

Copy link
Copy Markdown
Collaborator

Description

The v0.75.0 evidence gate failed after the draft release already existed, and because release images
are promoted by digest and never rebuilt, that draft could never pass — escaping it took a manual
draft deletion plus an admin-bypassed revert (#1701). This takes the reorder, not the cleanup
job: the gate now runs before anything is created, so a rejected release leaves no state at all.

Fixes #1705

Which approach, and why

Reorder — the preferred option. tag-images already resolved digests, generated and verified the
evidence, wrote the image lock, attested and signed it before touching the release; only
gh release upload needed a release to exist. Creating the draft immediately before that upload was
a local change, so the fallback cleanup job was not needed:

  • release no longer creates anything. It decides whether this commit cuts a version, enforces the
    preconditions, and publishes the same outputs as before.
  • tag-images creates the draft as its second-to-last step, then uploads the evidence to it.
  • Release notes are assembled where the draft is created, from the CHANGELOG.md that job already
    checks out. Only the schema-migration flag still travels as a job output, because deciding it needs
    tags and history that tag-images (depth 1) does not have. A job output, not an artifact: outputs
    survive a partial re-run in which release is not replayed, and the notes themselves would be a
    ~120 KB output at current release sizes, close enough to the per-output limit to be worth avoiding.
  • Both gh release upload calls gained --clobber, so a re-run over a resumed draft replaces assets
    instead of failing on every name that already exists.

The PARENT_VERSION must be published precondition is untouched, including its "draft targets a
different commit" sibling. Both are good invariants; they only bit us because failure left state
behind. scripts/ci-contract.test.ts now asserts both are still there, that the draft is created
after the evidence verifier runs, that assets are uploaded after the draft exists, and that every
release upload clobbers.

Failure modes considered

release.yml cannot be rehearsed, so these are argued rather than tested:

Failure Before After
Evidence gate rejects an image Draft exists at a commit that can never pass; blocks the next version too, needs a manual delete + admin-bypassed revert Nothing was created. No draft, no tag, no assets. The version is not consumed
Same-commit re-run of a gate failure Same digests, same verdict, plus a stale draft Same verdict, no residue
Failure after the draft exists (upgrade test, host smoke, publish) Resume path Unchanged: the release job still resumes a draft that targets this commit, tag-images reuses it instead of recreating, and the uploads clobber. This is the window the resume path was designed for — a flake at the same digests. A draft that genuinely can never pass still needs deleting by hand, but that window is now three jobs wide instead of the whole pipeline, and I deliberately did not add an if: failure() cleanup, because deleting the draft is exactly what would break the resume path for the flake case
Full workflow re-run after success release sees a published release, sets released=false, everything downstream skips
Full re-run after the draft was created gh release upload fails on existing asset names --clobber replaces them
Draft creation itself fails No assets uploaded; the run fails with an empty draft at worst, which the resume path picks up
git diff against the previous tag errors while deciding the migration warning Silently treated as "has migrations" Distinguishes exit 1 (differences) from any other status, which now fails loudly

Reordering does not weaken any gate: nothing between the digest resolution and the draft creation
reads the release, and the draft is still published only after the seeded-upgrade and supported-host
gates pass. Attestations and image promotion are unchanged.

Revert exemption, and its threat model

verify-changesets freeze-guards MIGRATION.md, .migration/*.md and pending changesets. A revert
of a version commit restores all three by construction, so the guard cannot pass on that path — it
has been admin-bypassed three times (#1686, #1691, #1701), and a guard that must be bypassed on a
known-good path teaches people to bypass it.

scripts/verify-revert.ts decides the exemption structurally. The title and branch name are
never read. Every non-merge commit the PR adds over the base must:

  1. record exactly one This reverts commit <sha>. line — the trailer git revert writes;
  2. name a commit that exists and is an ancestor of the base, so it is already reviewed and merged;
  3. name a commit with a single parent — reverting a merge is out of scope; and
  4. have a patch equal to that commit's patch reversed, by git patch-id --stable over a --binary
    diff.

Threat model:

  • Spoofing by title (revert: …, revert/x branch): impossible, neither is an input.
  • Fabricated trailer: the named commit must resolve and be an ancestor of the base.
  • Smuggling a change into the revert commit: any extra hunk changes the patch id.
  • Smuggling a change in a second commit: every commit in the range is checked, and a non-revert
    commit denies the exemption for the whole PR.
  • Binary payload swap under a real trailer: --binary puts the payload in the hashed diff, so a
    different blob is a different patch id. Covered by a test.
  • Reverting a merge, where "the inverse" is ambiguous: refused twice over — git's merge-revert
    trailer does not match the pattern, and the parent count is checked.
  • Residual risk, accepted: the exemption trusts that the reverted commit satisfied policy when it
    landed. It is an ancestor of the base, so it passed this same guard or was bypassed by an admin.
  • Fails closed: a conflict-resolved revert whose patch no longer matches, a commit whose trailer
    git did not write, a shallow clone that cannot see the reverted commit — all leave the guard in
    full force, and the admin bypass remains as the escape hatch.

The check runs with contents: read, no secrets and no network; it only reads git objects from the
already-checked-out clone (fetch-depth: 0).

It is verified against the real case: verifyRevert("8ff1a912a", "27a698954") — the base and head of
#1701 — returns verified, and the squash-merged commit on main (whose body lost the trailer) does
not.

Changeset

None. verify-changesets scopes SHIPPED_PATHS to server, webapp and docker, excluding
**/*.md and tests. This PR touches .github/, scripts/ and docs/ only, so no changeset is
required and none is added.

How to test

CI covers this: pnpm run format && pnpm run check pass locally, scripts/verify-revert.test.ts
adds 12 node:test cases over throwaway git fixtures (release-shaped reverts, reverts taken after
later work landed, riders, unrelated commits, fabricated trailers, binary swaps, merge reverts, empty
ranges, and the GITHUB_OUTPUT contract), and scripts/ci-contract.test.ts pins the new release job
wiring. release.yml itself cannot be exercised without cutting a release — the reasoning above is
the substitute.

Checklist

  • My changeset summary reads as an operator/user-facing note — n/a, no changeset required
  • If operators must act, the changeset and migration entry give actionable upgrade instructions — n/a
  • I did not commit generated-artifact changes that this PR did not cause

The v0.75.0 evidence gate rejected image digests that could never pass at
that commit, and escaping it meant deleting the draft by hand and reverting
the version commit under an administrator bypass. The gate now runs before
anything is created: tag-images resolves digests, generates and verifies the
evidence, and only then creates the draft it uploads the evidence to. A
rejected release leaves no draft and no tag, so the version re-cuts from a
commit that carries the fix with nothing to clean up first.

The parent-version precondition stays exactly as it was — it is a good
invariant, and it only bit us because failure left state behind.

Release notes move to the job that creates the draft, assembled from the
CHANGELOG.md it already checks out; only the schema-migration bit, which
needs tags, still travels as an output. Both release asset uploads clobber,
so re-running over a resumed draft replaces assets instead of failing on
their names.

The changeset freeze rules are lifted for a verified revert, which
scripts/verify-revert.ts decides structurally rather than by title: every
commit the pull request adds must record a revert trailer, name a commit
already reachable from the base, and carry that commit's patch exactly
reversed by patch id. Reverting a release restores MIGRATION.md and the
pending changesets by construction, so the guard could never pass on that
path — it has been bypassed three times for it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

Next included review available in 50 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: f8f0f7a7-06dc-4c33-a303-e47055ab5435

📥 Commits

Reviewing files that changed from the base of the PR and between 33f30c2 and 32b47bc.

📒 Files selected for processing (6)
  • .github/workflows/release.yml
  • .github/workflows/verify-changesets.yml
  • docs/contributor/release-management.mdx
  • scripts/ci-contract.test.ts
  • scripts/verify-revert.test.ts
  • scripts/verify-revert.ts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved automatically: @FelixTJDietrich is listed in the REVIEW_POLICY_MAINTAINERS repository variable, which the repository treats as satisfying the review requirement. See the review policy in docs/contributor/ci-cd.mdx.

@github-project-automation github-project-automation Bot moved this from Backlog to In Review in Hephaestus Sep 1, 2026
@github-actions github-actions Bot added bug Something isn't working documentation Improvements or additions to documentation ci GitHub Actions, workflows, build pipeline changes size:XL labels Sep 1, 2026
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

📚 Documentation Preview

Preview has been removed (PR closed)

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

🧩 Storybook Preview

Preview has been removed (PR closed)

@FelixTJDietrich
FelixTJDietrich added this pull request to the merge queue Sep 1, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 1, 2026
@FelixTJDietrich
FelixTJDietrich added this pull request to the merge queue Sep 1, 2026
Merged via the queue into main with commit af0d813 Sep 1, 2026
36 checks passed
@FelixTJDietrich
FelixTJDietrich deleted the fix/release-evidence-gate-ordering branch September 1, 2026 15:05
@github-project-automation github-project-automation Bot moved this from In Review to Done in Hephaestus Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ci GitHub Actions, workflows, build pipeline changes documentation Improvements or additions to documentation

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

fix(ci): a failed evidence gate must not wedge the version line

1 participant