fix(ci): scan images against the vulnerability policy where they are built - #1710
Merged
Conversation
…built The v0.75.0 release failed at `tag-images` on four HIGH CVEs in the pinned nginx base. Main was green because no container image is scanned anywhere before release: `ci-security-scan.yml` runs Trivy with `scan-type: "fs"` and `continue-on-error: true`, so it never sees an image. A gate whose first execution is at release is a tripwire, not a safety net. Two defects, fixed together because both rewrite `evaluate()`. Unfixable findings no longer block (#1704). `evaluate()` rejected any HIGH or CRITICAL regardless of whether upstream had published a fix, and `security/release-images.json` runs four upstream images through the same gate — alpine, nats, nginx and traefik cannot be patched at all. `Finding` now carries `FixedVersion`, and a finding needs a non-empty one to reach `rejected`. The filter is in the evaluator rather than on Trivy's command line, so unfixable findings stay counted in `highCritical` and visible in the signed evidence bundle. The gate now runs where images are built (#1703). That needed the exception schema fixed first: exceptions matched on the release image digest, which does not exist until `tag-images` runs, so a pull-request-time exception could never match a release-time digest and an exception could only be authored after the gate had already failed. `digest` is dropped from the match key and kept in the recorded exception for auditability; `installedVersion` already gives the "dies when the package moves" property. A new `scan` job in `reusable-docker-build.yml` then blocks on the `linux/amd64` image behind the run tag, calling the same script and the same `security/vulnerability-policy.json` the release and the weekly rescan call. Both architectures are still scanned at release. A failing gate now names the rejected CVEs in `$GITHUB_STEP_SUMMARY` and uploads its `.policy.json`; diagnosing v0.75.0 required rebuilding and rescanning the base image by hand. `public.ecr.aws/aquasecurity/trivy-db` is a `TRIVY_DB_REPOSITORY` fallback, because a blocking gate inherits Trivy's database availability as a hard dependency and GHCR answers TOOMANYREQUESTS. The evaluation result keeps its shape, so a published release's `.policy.json` still re-derives byte for byte in the weekly rescan. Closes #1703 Closes #1704 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Warning Review limit reachedNext included review available in 55 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (6)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
documentation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Approved automatically: @FelixTJDietrich is listed in the REVIEW_POLICY_MAINTAINERS repository variable, which the repository treats as satisfying the review requirement. See the review policy in docs/contributor/ci-cd.mdx.
github-actions
Bot
added
size:L
Contributor
📚 Documentation Preview
|
Contributor
🧩 Storybook Preview
|
This was referenced Sep 1, 2026
This was referenced Sep 1, 2026
FelixTJDietrich
added a commit
that referenced
this pull request
Sep 1, 2026
The policy existed only as `evaluate()` in `scripts/check-release-vulnerabilities.ts`. NIST SSDF RV.1.3 asks for a documented remediation policy and RV.2.1 for analysis leading to "remediation or other risk response"; neither prescribes thresholds, only that ours are written down. The practical payoff is that whoever writes the next exception knows what a justification has to contain. `docs/contributor/vulnerability-remediation.mdx` states the severity floor and the fixable/unfixable split with its reasoning, where the gate runs, the 90-day exception cap and its required evidence, and why VEX is not adopted. It sits under Operations beside the two pages #1710 already amended, and does not restate them. `Exception` gains an optional `justificationCategory` carrying CISA's five `not_affected` justifications, required for `not_affected` and forbidden for `affected`. That vocabulary is the one genuinely valuable part of VEX and it stops "not reachable" standing in for an analysis nobody did. Closes #1707 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The v0.75.0 release failed at
tag-imageswithwebapp does not satisfy vulnerability policy.Main was green because no container image is scanned anywhere before release:
ci-security-scan.ymlruns Trivy withscan-type: "fs"andcontinue-on-error: true, so itnever sees an image. This fixes both defects behind that, in one PR because both rewrite
evaluate()inscripts/check-release-vulnerabilities.ts.#1704 — unfixable findings no longer block.
evaluate()rejected any HIGH or CRITICALwhether or not upstream had published a fix, and
security/release-images.jsonruns fourupstream images — alpine, nats, nginx, traefik — through the same gate. Those cannot be
patched at all, so failing on them removes no risk and leaves nobody an action.
Findingnowcarries
FixedVersion, and a finding needs a non-empty one to reachrejected. The filter isin the evaluator, not on Trivy's command line (
--ignore-unfixed), so unfixable findingsstay counted in
highCriticaland visible in the signed evidence bundle. On this run thatdistinction is doing most of the work:
agent-pireports 59 HIGH/CRITICAL and rejects 1,postgresreports 99 and rejects 23.#1703 — the gate runs where images are built. That needed the exception schema fixed first.
Exceptions matched on
image | platform | digest | vulnerability | package | installedVersion,including the release image digest — which does not exist until
tag-imagesruns. Apull-request-time exception could therefore never match a release-time digest, and an exception
could only be authored after the gate had already failed a release.
digestis dropped fromthe match key (and from the duplicate key) and kept in the recorded exception for auditability;
installedVersionalready gives the "dies when the package moves" property.A new
scanjob inreusable-docker-build.ymlthen blocks after the push, on thelinux/amd64image behind the
run-<run_id>-<run_attempt>tag the workflow already produces. It calls thesame script and the same
security/vulnerability-policy.jsonthe release and the weeklyrescan call;
ci-contract.test.tsnow asserts there is no second copy. Both architectures arestill scanned at release, where the evidence bundle has to be complete.
Also: a failing gate names the rejected CVEs in
$GITHUB_STEP_SUMMARYand uploads its.policy.json— diagnosing v0.75.0 required rebuilding and rescanning the base image by hand.And
public.ecr.aws/aquasecurity/trivy-dbis added as aTRIVY_DB_REPOSITORYfallback, becausea blocking gate inherits Trivy's database availability as a hard dependency and GHCR returns
TOOMANYREQUESTSoften enough that Harbor, Rancher and New Relic all override it. The existing3× retry stays.
Closes #1703
Closes #1704
The gate is blocking on a policy that has never run at build time, so this PR fails on the
images it is meant to protect. The mechanism itself works end to end — subject resolution,
database download, scan, evaluation, step summary and artifact upload all succeeded on every
image; the four
Scan linux/amd64 Imagejobs failed on the policy, which is the point.Measured on this run's images:
webapplibcrypto3/libssl33.5.7-r0 (CVE-2026-14456),libexpat2.8.2-r0 (CVE-2026-66046, CVE-2026-76641)agent-pilibexpat12.5.0-1+deb12u2 (CVE-2026-56408)postgreslibexpat1(same CVE) — and 22 GostdlibCVEs inusr/local/bin/gosu, built with Go 1.24.6libexpat1one, and only if it includesdocker/postgres/application-serverorg.bouncycastle:bcprov-jdk18on1.82 — CVE-2025-14813 (CRITICAL) and CVE-2026-5598 (HIGH), both fixed in 1.84So the dependency is: #1702 first (it clears
webappandagent-pioutright), and then twothings no issue covers today:
application-server: bump Bouncy Castle 1.82 → 1.84. This is an ordinary dependencybump in the server's tree, not an image concern.
postgres:gosuis a Go binary baked into the upstreampostgres:*-bookwormimage andis not reachable by
apt-get upgrade; the fix has to come from a docker-library rebuild, abase-image bump that carries one, or dropping/replacing
gosu. If none of those lands intime, this is exactly the case the exception mechanism exists for — and because of this
PR's match-key fix, such an exception can now be authored before a release instead of only
after one has already failed.
I have deliberately not weakened the gate, added
--ignore-unfixed, or written exceptionson someone else's behalf to get this green.
Worth noting what this measurement already bought:
application-serverandpostgreswereboth going to block the very next release attempt, behind
webappin the manifest order, withno warning anywhere. That is the invisibility this PR removes.
How to test
CI covers this once the blockers above clear. Locally:
pnpm run format && pnpm run check— both pass.scripts/check-release-vulnerabilities.test.tsgoes 4 → 7 and coversthe fixable/unfixable split (including a HIGH with no
FixedVersionthat appears inhighCriticalbut not inrejected, a whitespace-onlyFixedVersion, an absent field, and amixed report), the exception match key with and without
digest(every other field stillbinds, and two exceptions differing only by digest are now a duplicate), and the run summary.
scripts/ci-contract.test.tsgains an assertion that the scan job is blocking, amd64-only,scans the run tag, uploads its result, and that every workflow call site evaluates
security/vulnerability-policy.json— there is no second copy.scripts/verify-release-evidence.tsstill consumesevaluate()unchanged, and its own testspass. The result keeps its exact shape, which matters: the weekly rescan re-derives a
published release's
.policy.jsonand fails on any difference.rejectedcan only shrink anda published release had
rejected: [], so older evidence still re-derives byte for byte.Checklist
verify-changesets.ymlscopesSHIPPED_PATHStoserver,webappanddocker; this PR touches.github/,scripts/anddocs/only, soshipped_changedisempty and the release-note check does not fire. Confirmed by reading the workflow, and by
the
Verify changesetsjob passing on this PR.instructions — n/a, no operator-visible change.
Deliberately not changed
release.ymlandrescan-release-images.ymldid not get the ECR fallback. In both, theTrivy database download shares one
run:block — and therefore oneenv:— with the imagescan, which sets
TRIVY_USERNAME/TRIVY_PASSWORDfor GHCR. Trivy applies those credentials toevery registry, so a fallback to
public.ecr.awswould present a GHCR token and fail authinstead of pulling anonymously. Giving those two workflows a real fallback means splitting the
database download into its own credential-free step, a separate change to a release-critical
path. The new job here is already structured that way.
🤖 Generated with Claude Code