Commit c4c6a9b
TT-17641: Own next-generation plugin compiler workflows and image attestations (#517)
* feat: own plugin compiler workflows and image attestations
* fix: use Buildx attestation-capable builder
* Harden NG compiler self-test exclusion
* fix(plugin-compiler): harden NG image and module handling
* fix(plugin-compiler): prune unsupported Kerberos sysroot files
* feat: add DHI VEX-aware Trivy scanning
* feat(plugin-compiler): manage DHI toolchain base
* fix(plugin-compiler): validate native Go FIPS setting
* revert: drop the gromit dhi-vex scan command
The DHI VEX work does not belong in gromit. Re-keying Docker Hardened
Images decisions onto installed binary package identities is a
publishing pipeline that produces a Trivy VEX repository, not a
scan-time wrapper: consumers point Trivy at the published repository
and need no gromit-side tooling at all.
Reverts eec1c2b. The pipeline lives in tyk-vex-records-live.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(plugin-compiler): require the DHI-provisioned toolchain
The NG compiler toolchain is no longer installable at build time. It
must arrive pre-provisioned in the Docker Hardened Images customization
named by BASE_IMAGE, and the Dockerfile fails closed if any expected
component is missing from the base.
This is a security requirement, not a convenience. Only packages Docker
provisions into the customization are covered by Docker's maintenance
obligation, carry +dhi versions, and have published DHI vulnerability
decisions. A toolchain package apt-installed at build time would be an
ordinary upstream build that nobody is obliged to patch and that no DHI
advisory covers, silently widening the image's unmaintained surface.
Removed with it: PREPROVISIONED_TOOLCHAIN, the pinned debian:bookworm-slim
pull-request base, NG_PR_BASE_SOURCE, and the PullRequestBaseImage policy
field. Pull requests now build the same base the release builds, so the
compile/load gate validates the artifact that actually ships instead of a
Debian stand-in.
The trade-off is deliberate: changing the toolchain package list now
requires Docker to rebuild the customization first.
Also folded in:
- consolidate plugin-compiler-ng-base.yml into the build workflow and
drop the unused rewrite-imports.go, declaring both under the tyk
repo's deletedfiles so generated checkouts are cleaned up
- extract removeDeletedFiles, and stop continuing past a failed stat
into a nil-FileInfo dereference
- extend the loadtest gate and build.sh
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(plugin-compiler): let pull requests pull the private DHI base
Requiring the DHI-provisioned toolchain broke pull-request builds. The
customization lives in a private Docker Hub repository, but the Docker Hub
login step was gated `github.event_name != 'pull_request'` -- a condition that
was correct only while pull requests used the public debian:bookworm-slim
fallback. Verified: an anonymous manifest fetch for
tykio/dhi-busybox-plugin-compiler returns HTTP 401, while public repos on the
same probe return 200. "Resolve source base digest" runs unconditionally, so
every pull-request run would have failed there.
Two changes:
- The Docker Hub login now runs on pull requests too. The ECR steps stay gated,
because only release builds push.
- docker-build is skipped for pull requests raised from a fork. GitHub
withholds secrets from fork runs, so such a build cannot pull the base at
all; skipping it beats failing every community pull request with an opaque
401. There is still no public fallback base.
The fork guard is composed as a bare expression. Splicing a ${{ }} fragment
into an `if` that is already bare string-concatenates instead of evaluating,
which silently always passes -- a test now asserts the rendered condition
contains no ${{.
Note this widens secret exposure: DOCKER_USERNAME/DOCKER_PASSWORD are now
readable by same-repo pull-request runs, where previously they were not. A
pull-only Docker Hub token would narrow that back and is worth a follow-up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(plugin-compiler): refresh the DHI customization before each release
Docker does not rebuild a customization when packages it ADDS receive security
patches. Only a base-image patch or an OCI artifact change triggers a rebuild,
and this base is a minimal BusyBox runtime that Docker's own catalog reports as
0 CRITICAL / 0 HIGH / 0 MEDIUM / 0 LOW. That trigger therefore effectively never
fires, while every vulnerability in the compiler image lives in an added package.
Observed: the customization sat unbuilt from 2026-07-29 to 2026-08-10 while
Docker published expat 2.8.2-1~deb13u1+dhi0 on 2026-07-30. The image kept
shipping libexpat1 2.7.1-2+dhi5 with four HIGH findings that already had a fix.
A manual rebuild moved libexpat1 to 2.8.2-1~deb13u1+dhi0 AND linux-libc-dev from
6.12.96-1+dhi0 to 6.12.101-1+dhi0, and raised +dhi coverage from 112/162 to
118/162. linux-libc-dev was never declared, which proves rebuilds resolve
packages freshly rather than replaying a snapshot -- the mechanism works, nothing
was triggering it.
Release builds now refresh the customization before resolving the base digest.
There is no supported way to ask for a rebuild: `customization build` exposes
only get/list/logs, and `edit` is a no-op unless the manifest actually differs --
while still returning "Successfully updated". Confirmed deliberately: an
unmodified re-apply queued no build after seven minutes. The step therefore
stamps the run id into an annotation solely to make the manifest unequal. The
annotation means nothing; the step should be deleted once Docker ships a real
trigger. Reported as docker-hardened-images/advisories#2017.
Because `edit` reports success either way, the step does not trust its exit
code. It records the newest build id first and polls for a genuinely different
one, so a silently discarded request is detected rather than assumed to have
worked.
Constraints:
- Release events only. A pull request must never rewrite the shared hardened
base, so the step is gated off for them and fork PRs cannot reach it at all.
- continue-on-error. A refresh that fails leaves the base as fresh as it was
yesterday, which is worse than current but far better than blocking a release
on Docker-side behaviour we do not control. Failures raise a ::warning::, write
to the step summary, and mark the step red so a persistently stale base is
visible rather than silent.
- Configurable, not hardcoded. `customizationid` is a policy field and the org
defaults to the baseimage prefix; omitting the id removes the step entirely.
The customization id is stable across rebuilds (only build ids change), so
pinning it in config is safe.
Also declares expat in the customization. It previously arrived only as a
transitive dependency, which is exactly why it drifted unnoticed while every
explicitly declared package stayed level with Docker's newest +dhi builds.
* fix(plugin-compiler): degrade cleanly when the DHI CLI is absent
The customization refresh failed on its first release run with:
unknown flag: --org
Usage: docker [OPTIONS] COMMAND [ARG...]
Process completed with exit code 125
`docker dhi` is a Docker Desktop CLI plugin and is not present on hosted
runners, so the first call fell through to plain `docker`, which rejected
`--org`. Because that happened on the very first command, none of the step's
warning or step-summary reporting ran -- the operator saw a raw error that says
nothing about the actual problem.
continue-on-error worked as intended: v5.15.0-alpha11 published normally, and
its base was current because the customization had been rebuilt manually
beforehand. But a missing dependency should announce itself, not fail obscurely.
The step now checks for the CLI up front and, when it is missing, emits a
::warning:: naming the real cause and records it in the step summary before
exiting. The exit is still non-zero so continue-on-error marks the step red and
a persistently unrefreshed base stays visible.
This does not make the refresh work in CI. Installing the plugin on the runner
is a separate question, and if no standalone Linux distribution exists then
automating customization refresh is not possible with the supported tooling at
all -- which is worth adding to docker-hardened-images/advisories#2017.
* chore: drop files committed by accident from this branch
An earlier commit on this branch used `git add -A`, which swept in 42 files
that were sitting untracked in the working tree and have nothing to do with the
plugin compiler: an AI-metadata migration from 2026-05-21 (its scripts, CSV
reports, rsync excludes and ~4000 lines of sync logs), a stray root-level
config.yaml, a zellij session layout, and goal.md, a 1661-line experiment ledger
from the session that preceded this work.
Also drops plugin-compiler-ng-jira-ticket.md and
plugin-compiler-ng-security-scanning.md. Those are about this project, but they
are drafts of the ticket text rather than repository documentation, and their
content now lives in TT-17641.
Removed with `git rm --cached`, so the files stay on disk for whoever was using
them; only the tracking goes away. Since the additions and removals are both on
this branch, the files no longer appear in the pull request diff at all.
.gitignore gains entries for them so the next `git add -A` cannot repeat this.
Nothing under dhi/, docs/, policy/ or config/ is touched -- those are the actual
subject of this branch.
* fix(dhi): stop the refresh stamp from causing endless rebuilds
Review of this branch found that the release-time customization refresh and the
apply script undo each other.
The refresh stamps `io.tyk.rebuild-trigger` into the remote manifest purely to
create a difference, because Docker offers no way to request a rebuild. But
EqualCustomizations only strips `id`, so after any release the apply script sees
the stamped remote as different from the repository manifest, edits it to remove
the stamp -- and that edit is itself a difference, which queues another rebuild.
The script's "already up to date" path was dead from the first release onwards,
which also contradicts this branch's own claim of idempotent apply tooling.
The stamp describes nothing about the desired image, so manifest comparison now
ignores it. Two tests cover it: a stamped remote compares equal, and a genuine
annotation change is still a difference.
Four further fixes in the refresh step, all from the same review:
- An empty build list left `latest` empty, which is not equal to `before`, so
the "no rebuild was queued" guard was skipped and the step reported success
while having done nothing. It now treats an empty result as failure.
- The annotation stamper matched the first `annotations:` anywhere in the file.
A nested block, or a key like `image-annotations:`, would take the match and
the new entry would be dedented out of its mapping, corrupting the manifest
about to be sent to Docker. It now anchors to a top-level block, refuses to
guess when several exist, and appends one when there is none. Verified against
nested, decoy, absent and already-stamped inputs; all now yield valid YAML with
exactly one stamp.
- The failure check looked at a fixed two rows, so a failure on any further
platform was missed. It now checks every row of the batch just queued, matched
on their shared start timestamp.
- CustomizationOrgValue read the raw BaseImage field rather than BaseImageValue,
returning "" whenever only the default was configured, which rendered
`--org ""`. It now uses the defaulted value and rejects a registry host such as
`dhi.io`, since that is not an organization. The step also refuses to run with
an empty org instead of calling the CLI with one.
Also restores gofmt on policy/policy.go and policy/diff_test.go, which this
branch had broken, and extends .gitignore to the two working documents removed
in 6c68840 so a `git add -A` cannot bring them back.
---------
Co-authored-by: Tit Petric <tit@tyk.io>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>1 parent 0367dc1 commit c4c6a9b
55 files changed
Lines changed: 8712 additions & 247 deletions
File tree
- config
- dhi
- cmd/customization-equal
- customizations
- docs
- policy
- templates
- plugin-compiler-ng
- .github/workflows
- ci/images/plugin-compiler-ng
- data
- scripts
- plugin-compiler
- .github/workflows
- ci/images/plugin-compiler
- data
- releng/.github/workflows/release.yml.d
- testdata/golden/tyk
- master
- .github/workflows
- ci/images
- plugin-compiler-ng
- data
- scripts
- plugin-compiler
- data
- release-5.13.1/.github/workflows
- release-5.13/.github/workflows
- release-5.14.0/.github/workflows
- release-5.14/.github/workflows
- release-5.3/.github/workflows
- release-5.8.15/.github/workflows
- release-5.8/.github/workflows
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
47 | | - | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
669 | 669 | | |
670 | 670 | | |
671 | 671 | | |
672 | | - | |
| 672 | + | |
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
53 | 53 | | |
54 | 54 | | |
55 | 55 | | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
56 | 59 | | |
57 | 60 | | |
58 | 61 | | |
| |||
173 | 176 | | |
174 | 177 | | |
175 | 178 | | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
176 | 224 | | |
177 | 225 | | |
178 | 226 | | |
| |||
191 | 239 | | |
192 | 240 | | |
193 | 241 | | |
| 242 | + | |
| 243 | + | |
194 | 244 | | |
195 | 245 | | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
196 | 258 | | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
197 | 281 | | |
198 | 282 | | |
199 | 283 | | |
200 | 284 | | |
201 | 285 | | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
202 | 308 | | |
203 | 309 | | |
204 | 310 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
14 | | - | |
15 | | - | |
16 | | - | |
17 | | - | |
18 | | - | |
19 | | - | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
0 commit comments