Skip to content

Commit 2e0bc38

Browse files
authored
fix(ci): defer release aliases until security gates pass (#1763)
Signed-off-by: Mark Chmarny <mark@chmarny.com>
1 parent ee10a60 commit 2e0bc38

17 files changed

Lines changed: 4452 additions & 225 deletions

File tree

.github/actions/README.md

Lines changed: 44 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -92,16 +92,25 @@ This action runs `tools/setup-tools --skip-go --skip-docker` in auto mode, which
9292
```
9393

9494
#### `go-build-release/`
95-
**Purpose**: Complete build and release pipeline (tools + auth + make release)
95+
**Purpose**: Validate the exact-tag release target, then run the complete build
96+
and release pipeline (tools + auth + make release)
9697
**When to use**: Release workflows that build and publish artifacts
9798
**Inputs**:
9899
- `registry` (optional): Container registry (default: "ghcr.io")
100+
- `ko_version` (optional): Ko version (default: "v0.18.0")
99101
- `goreleaser_version` (required): GoReleaser version from `load-versions`
102+
- `go_licenses_version` (required): go-licenses version from `load-versions`
103+
- `candidate_tag` (required): Validated `candidate-<run-id>-<run-attempt>` image tag
100104

101105
**Outputs**:
102106
- `release_outcome`: Release step outcome (success/failure)
103107

104-
**Note**: Image repository paths are fully specified in `.goreleaser.yaml` under `kos.repositories`.
108+
**Note**: A partial draft is reused only when its name and tag both equal the
109+
release tag, its pre-release state matches, and its existing assets are a safe
110+
subset of the fixed release asset set. Unexpected draft assets and
111+
already-public releases are rejected before GoReleaser runs, and reused draft
112+
notes are replaced with notes generated from the current tag. Image repository
113+
paths are fully specified in `.goreleaser.yaml` under `kos.repositories`.
105114

106115
**Example**:
107116
```yaml
@@ -110,7 +119,10 @@ This action runs `tools/setup-tools --skip-go --skip-docker` in auto mode, which
110119
- uses: ./.github/actions/go-build-release
111120
id: release
112121
with:
122+
ko_version: ${{ steps.versions.outputs.ko }}
113123
goreleaser_version: ${{ steps.versions.outputs.goreleaser }}
124+
go_licenses_version: ${{ steps.versions.outputs.go_licenses }}
125+
candidate_tag: ${{ needs.detect.outputs.candidate_tag }}
114126
- if: steps.release.outputs.release_outcome == 'success'
115127
run: echo "Release succeeded"
116128
```
@@ -130,11 +142,13 @@ This action runs `tools/setup-tools --skip-go --skip-docker` in auto mode, which
130142
```
131143

132144
#### `attest-image-from-tag/`
133-
**Purpose**: Resolve digest from tag and generate SBOM + provenance
134-
**When to use**: Attesting images by tag (typical release workflow)
145+
**Purpose**: Bind a fixed AICR candidate image to an authoritative digest and
146+
generate SBOM + provenance
147+
**When to use**: Attesting candidate images in the AICR release workflow
135148
**Inputs**:
136-
- `image_name` (required): Full image name without tag (e.g., "ghcr.io/org/image")
137-
- `tag` (required): Image tag (e.g., "v1.2.3")
149+
- `image_name` (required): One of the seven fixed AICR release image names
150+
- `candidate_tag` (required): Validated `candidate-<run-id>-<run-attempt>` tag
151+
- `expected_digest` (required): Authoritative `sha256:<64 lowercase hex>` digest
138152
- `crane_version` (optional): crane version (default: "v0.20.6")
139153

140154
**Outputs**:
@@ -144,23 +158,26 @@ This action runs `tools/setup-tools --skip-go --skip-docker` in auto mode, which
144158
```yaml
145159
- uses: ./.github/actions/attest-image-from-tag
146160
with:
147-
image_name: ghcr.io/${{ github.repository_owner }}/my-app
148-
tag: ${{ github.ref_name }}
161+
image_name: ghcr.io/nvidia/aicrd
162+
candidate_tag: candidate-${{ github.run_id }}-${{ github.run_attempt }}
163+
expected_digest: sha256:0000000000000000000000000000000000000000000000000000000000000000
149164
```
150165

151166
#### `sbom-and-attest/`
167+
152168
**Purpose**: Generate SBOM and attestations for image with known digest
153169
**When to use**: When you already have the digest (e.g., from build output)
154170
**Inputs**:
155-
- `image_name` (required): Full image name
171+
- `image_name` (required): One of the seven fixed AICR release image names
156172
- `image_digest` (required): sha256 digest
157173

158174
**Example**:
175+
159176
```yaml
160177
- uses: ./.github/actions/sbom-and-attest
161178
with:
162-
image_name: ghcr.io/org/image
163-
image_digest: sha256:abc123...
179+
image_name: ghcr.io/nvidia/aicrd
180+
image_digest: sha256:0000000000000000000000000000000000000000000000000000000000000000
164181
```
165182

166183
### KWOK Testing Actions
@@ -236,12 +253,17 @@ This action runs `tools/setup-tools --skip-go --skip-docker` in auto mode, which
236253
**Trigger**: Semantic version tags (v*.*.*)
237254
**Purpose**: Build, release, attest, deploy
238255
**Jobs**:
239-
1. **Unit Tests** (parallel): Go CI + security scan
240-
2. **Integration Tests** (parallel): CLI integration tests
241-
3. **E2E Tests** (parallel): Full end-to-end tests
242-
4. **Build and Release** (after tests): GoReleaser builds binaries and images to GHCR
243-
5. **Attest Images** (after build): SBOM and provenance for aicr and aicrd images
244-
6. **Deploy Demo API Server** (after attest): Copy image to Artifact Registry and deploy demo to Cloud Run (example deployment)
256+
1. **Qualification**: Reusable test, lint, E2E, and source-security gates
257+
2. **Candidate Builds**: Draft release artifacts and all seven images under one
258+
run-unique candidate tag
259+
3. **Digest Resolution**: One authoritative seven-image digest map
260+
4. **Image Security**: Both platforms of every resolved digest are scanned
261+
5. **Attestation**: Platform SBOMs and reusable-workflow provenance for the same digests
262+
6. **Promotion**: Read-only preflight, all version aliases, then stable `latest`
263+
aliases only after every version alias is verified
264+
7. **Publication**: Require the exact release asset set, then publish the
265+
validated numeric GitHub release ID
266+
8. **Stable Distribution**: Publish Homebrew and deploy the demo after publication
245267

246268
### `test-deploy.yaml`
247269
**Trigger**: Manual (workflow_dispatch)
@@ -336,11 +358,15 @@ jobs:
336358
- uses: ./.github/actions/go-build-release
337359
id: release
338360
with:
361+
ko_version: ${{ steps.versions.outputs.ko }}
339362
goreleaser_version: ${{ steps.versions.outputs.goreleaser }}
363+
go_licenses_version: ${{ steps.versions.outputs.go_licenses }}
364+
candidate_tag: candidate-${{ github.run_id }}-${{ github.run_attempt }}
340365
- uses: ./.github/actions/attest-image-from-tag
341366
with:
342367
image_name: ghcr.io/nvidia/aicrd
343-
tag: ${{ github.ref_name }}
368+
candidate_tag: candidate-${{ github.run_id }}-${{ github.run_attempt }}
369+
expected_digest: sha256:0000000000000000000000000000000000000000000000000000000000000000
344370
crane_version: ${{ steps.versions.outputs.crane }}
345371
```
346372

.github/actions/attest-image-from-tag/action.yml

Lines changed: 44 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,11 @@ inputs:
1919
image_name:
2020
description: 'Full image name (without tag or digest), e.g. ghcr.io/org/image'
2121
required: true
22-
tag:
23-
description: 'Image tag to resolve (e.g. v1.2.3)'
22+
candidate_tag:
23+
description: 'Run-unique candidate tag to resolve'
24+
required: true
25+
expected_digest:
26+
description: 'Authoritative digest that the candidate must resolve to'
2427
required: true
2528
crane_version:
2629
description: 'crane version to install (e.g. v0.20.6)'
@@ -36,17 +39,40 @@ runs:
3639
using: 'composite'
3740
steps:
3841
- name: Validate inputs
42+
id: validate
3943
shell: bash
44+
env:
45+
INPUT_IMAGE_NAME: ${{ inputs.image_name }}
46+
INPUT_CANDIDATE_TAG: ${{ inputs.candidate_tag }}
47+
INPUT_EXPECTED_DIGEST: ${{ inputs.expected_digest }}
48+
INPUT_CRANE_VERSION: ${{ inputs.crane_version }}
4049
run: |
4150
set -euo pipefail
42-
if [[ -z "${{ inputs.image_name }}" ]]; then
43-
echo "::error::image_name is required"
51+
# shellcheck source=.github/actions/release-input-validation.sh
52+
source "${GITHUB_ACTION_PATH}/../release-input-validation.sh"
53+
reject_newline image_name "${INPUT_IMAGE_NAME}"
54+
reject_newline candidate_tag "${INPUT_CANDIDATE_TAG}"
55+
reject_newline expected_digest "${INPUT_EXPECTED_DIGEST}"
56+
reject_newline crane_version "${INPUT_CRANE_VERSION}"
57+
require_release_image "${INPUT_IMAGE_NAME}"
58+
if [[ ! "${INPUT_CANDIDATE_TAG}" =~ ^candidate-[0-9]+-[0-9]+$ ]]; then
59+
echo "::error::candidate_tag must match candidate-RUN_ID-RUN_ATTEMPT"
60+
exit 1
61+
fi
62+
if [[ ! "${INPUT_EXPECTED_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]]; then
63+
echo "::error::expected_digest is malformed"
4464
exit 1
4565
fi
46-
if [[ -z "${{ inputs.tag }}" ]]; then
47-
echo "::error::tag is required"
66+
if [[ ! "${INPUT_CRANE_VERSION}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
67+
echo "::error::crane_version must use vMAJOR.MINOR.PATCH"
4868
exit 1
4969
fi
70+
{
71+
echo "image_name=${INPUT_IMAGE_NAME}"
72+
echo "candidate_tag=${INPUT_CANDIDATE_TAG}"
73+
echo "expected_digest=${INPUT_EXPECTED_DIGEST}"
74+
echo "crane_version=${INPUT_CRANE_VERSION}"
75+
} >> "$GITHUB_OUTPUT"
5076
5177
- name: Authenticate to registry
5278
uses: ./.github/actions/ghcr-login
@@ -55,22 +81,30 @@ runs:
5581
uses: ./.github/actions/setup-build-tools
5682
with:
5783
install_crane: 'true'
58-
crane_version: ${{ inputs.crane_version }}
84+
crane_version: ${{ steps.validate.outputs.crane_version }}
5985

6086
- name: Resolve image digest
6187
id: digest
6288
shell: bash
89+
env:
90+
IMAGE_NAME: ${{ steps.validate.outputs.image_name }}
91+
CANDIDATE_TAG: ${{ steps.validate.outputs.candidate_tag }}
92+
EXPECTED_DIGEST: ${{ steps.validate.outputs.expected_digest }}
6393
run: |
6494
set -euo pipefail
65-
DIGEST="$(crane digest "${{ inputs.image_name }}:${{ inputs.tag }}")"
95+
DIGEST="$(timeout --foreground 120s crane digest "${IMAGE_NAME}:${CANDIDATE_TAG}")"
6696
if [[ ! "$DIGEST" =~ ^sha256:[a-f0-9]{64}$ ]]; then
6797
echo "::error::unexpected digest format: $DIGEST"
6898
exit 1
6999
fi
100+
if [[ "${DIGEST}" != "${EXPECTED_DIGEST}" ]]; then
101+
echo "::error::resolved digest does not match expected digest"
102+
exit 1
103+
fi
70104
echo "image_digest=$DIGEST" >> "$GITHUB_OUTPUT"
71105
72106
- name: Generate SBOM and attestations
73107
uses: ./.github/actions/sbom-and-attest
74108
with:
75-
image_name: ${{ inputs.image_name }}
76-
image_digest: ${{ steps.digest.outputs.image_digest }}
109+
image_name: ${{ steps.validate.outputs.image_name }}
110+
image_digest: ${{ steps.validate.outputs.expected_digest }}

.github/actions/go-build-release/action.yml

Lines changed: 61 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -30,10 +30,9 @@ inputs:
3030
go_licenses_version:
3131
description: 'go-licenses version (from .settings.yaml)'
3232
required: true
33-
homebrew_deploy_key:
34-
description: 'SSH deploy key for pushing Homebrew formula'
35-
required: false
36-
default: ''
33+
candidate_tag:
34+
description: 'Run-unique candidate image tag'
35+
required: true
3736

3837
outputs:
3938
release_outcome:
@@ -43,19 +42,62 @@ outputs:
4342
runs:
4443
using: 'composite'
4544
steps:
45+
- name: Validate inputs
46+
id: validate
47+
shell: bash
48+
env:
49+
INPUT_REGISTRY: ${{ inputs.registry }}
50+
INPUT_KO_VERSION: ${{ inputs.ko_version }}
51+
INPUT_GORELEASER_VERSION: ${{ inputs.goreleaser_version }}
52+
INPUT_GO_LICENSES_VERSION: ${{ inputs.go_licenses_version }}
53+
INPUT_CANDIDATE_TAG: ${{ inputs.candidate_tag }}
54+
run: |
55+
set -euo pipefail
56+
# shellcheck source=.github/actions/release-input-validation.sh
57+
source "${GITHUB_ACTION_PATH}/../release-input-validation.sh"
58+
for pair in \
59+
"registry:${INPUT_REGISTRY}" \
60+
"ko_version:${INPUT_KO_VERSION}" \
61+
"goreleaser_version:${INPUT_GORELEASER_VERSION}" \
62+
"go_licenses_version:${INPUT_GO_LICENSES_VERSION}" \
63+
"candidate_tag:${INPUT_CANDIDATE_TAG}"; do
64+
reject_newline "${pair%%:*}" "${pair#*:}"
65+
done
66+
if [[ "${INPUT_REGISTRY}" != "ghcr.io" ]]; then
67+
echo "::error::registry must be ghcr.io"
68+
exit 1
69+
fi
70+
for version in "${INPUT_KO_VERSION}" "${INPUT_GORELEASER_VERSION}" "${INPUT_GO_LICENSES_VERSION}"; do
71+
if [[ ! "${version}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
72+
echo "::error::tool versions must use vMAJOR.MINOR.PATCH"
73+
exit 1
74+
fi
75+
done
76+
if [[ ! "${INPUT_CANDIDATE_TAG}" =~ ^candidate-[0-9]+-[0-9]+$ ]]; then
77+
echo "::error::candidate_tag must match candidate-RUN_ID-RUN_ATTEMPT"
78+
exit 1
79+
fi
80+
{
81+
echo "registry=${INPUT_REGISTRY}"
82+
echo "ko_version=${INPUT_KO_VERSION}"
83+
echo "goreleaser_version=${INPUT_GORELEASER_VERSION}"
84+
echo "go_licenses_version=${INPUT_GO_LICENSES_VERSION}"
85+
echo "candidate_tag=${INPUT_CANDIDATE_TAG}"
86+
} >> "$GITHUB_OUTPUT"
87+
4688
- name: Setup build tools
4789
uses: ./.github/actions/setup-build-tools
4890
with:
4991
install_ko: 'true'
50-
ko_version: ${{ inputs.ko_version }}
92+
ko_version: ${{ steps.validate.outputs.ko_version }}
5193
install_syft: 'true'
5294
install_goreleaser: 'true'
53-
goreleaser_version: ${{ inputs.goreleaser_version }}
95+
goreleaser_version: ${{ steps.validate.outputs.goreleaser_version }}
5496

5597
- name: Install go-licenses
5698
shell: bash
5799
env:
58-
GO_LICENSES_VERSION: ${{ inputs.go_licenses_version }}
100+
GO_LICENSES_VERSION: ${{ steps.validate.outputs.go_licenses_version }}
59101
run: |
60102
set -euo pipefail
61103
if [[ -z "${GO_LICENSES_VERSION}" ]]; then
@@ -73,19 +115,29 @@ runs:
73115
- name: Authenticate to registry
74116
uses: ./.github/actions/ghcr-login
75117
with:
76-
registry: ${{ inputs.registry }}
118+
registry: ${{ steps.validate.outputs.registry }}
77119

78120
# Note: the Rekor v2 signing config (AICR_SIGNING_CONFIG) is fetched and
79121
# exported by the generate-slsa-predicate action, co-located with
80122
# SLSA_PREDICATE, so it is present here without a separate step. See #1650.
81123

124+
- name: Verify release target
125+
shell: bash
126+
env:
127+
GH_TOKEN: ${{ github.token }}
128+
RELEASE_TAG: ${{ github.ref_name }}
129+
GITHUB_SHA: ${{ github.sha }}
130+
run: |
131+
set -euo pipefail
132+
.github/scripts/release-images.sh release-target
133+
82134
- name: Build and Release
83135
id: release
84136
shell: bash
85137
env:
86138
GITHUB_TOKEN: ${{ github.token }}
87139
GOFLAGS: -mod=vendor
88-
HOMEBREW_DEPLOY_KEY: ${{ inputs.homebrew_deploy_key }}
140+
AICR_CANDIDATE_TAG: ${{ steps.validate.outputs.candidate_tag }}
89141
# Repository paths are fully specified in .goreleaser.yaml kos.repositories
90142
run: |
91143
set -euo pipefail
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
#!/usr/bin/env bash
2+
3+
# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
reject_newline() {
18+
local name="$1" value="$2"
19+
if [[ -z "${value}" || "${value}" == *$'\n'* || "${value}" == *$'\r'* ]]; then
20+
echo "::error::${name} must be a non-empty single-line value"
21+
exit 1
22+
fi
23+
}
24+
25+
require_release_image() {
26+
case "$1" in
27+
ghcr.io/nvidia/aicr | \
28+
ghcr.io/nvidia/aicrd | \
29+
ghcr.io/nvidia/aicr-validators/deployment | \
30+
ghcr.io/nvidia/aicr-validators/performance | \
31+
ghcr.io/nvidia/aicr-validators/conformance | \
32+
ghcr.io/nvidia/aicr-validators/aiperf-bench | \
33+
ghcr.io/nvidia/aicr-gate) ;;
34+
*)
35+
echo "::error::image_name is not a fixed AICR release image"
36+
exit 1
37+
;;
38+
esac
39+
}

0 commit comments

Comments
 (0)