Skip to content

Commit 4fe9dc8

Browse files
lockwobrmchmarny
lockwobr
and
mchmarny
authored
feat(ci): Rekor v2 identity monitoring for the release signer (#1727)
Signed-off-by: Brian Lockwood <lockwobr@gmail.com> Co-authored-by: Mark Chmarny <mchmarny@users.noreply.github.com>
1 parent 8068cd4 commit 4fe9dc8

2 files changed

Lines changed: 122 additions & 12 deletions

File tree

‎.github/workflows/rekor-monitor.yaml‎

Lines changed: 64 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -12,17 +12,39 @@
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
1414

15-
# Transparency-log monitoring for AICR's release supply chain.
15+
# Transparency-log monitoring for AICR's release supply chain, on Rekor v2.
1616
#
17-
# Hourly, this calls the upstream sigstore/rekor-monitor reusable workflow to:
18-
# - Consistency: verify the public-good Rekor log stays append-only between
19-
# runs (checkpoint persisted as the `checkpoint` artifact).
20-
# - Identity: scan entries added since the last run for AICR's release signing
21-
# identity. An entry under that identity that a release did not produce
22-
# signals OIDC/key compromise.
17+
# Hourly, this calls the upstream sigstore/rekor-monitor reusable workflow to do
18+
# both checks in a single job (see internal/cmd MonitorLoop upstream):
19+
# - Consistency: prove the Rekor v2 log stays append-only between runs (Merkle
20+
# consistency from the last checkpoint to the current tree head). O(log n),
21+
# always finishes in seconds. Checkpoint persisted as the `checkpoint`
22+
# artifact.
23+
# - Identity: scan entries added since the last checkpoint for AICR's release
24+
# signing identity. An entry under that identity that a release did not
25+
# produce signals OIDC/key compromise.
2326
# On any failure (consistency break or matched identity), it files an issue
2427
# (`file_issue: true`).
2528
#
29+
# Why v2 (NVIDIA/aicr#1623). Identity monitoring is a linear scan of every entry
30+
# added since the last checkpoint (Rekor's index cannot be queried by
31+
# certificate SAN, and our keyless release identity has no email or fixed key).
32+
# On the Rekor **v1** firehose that scan runs ~50x slower than the log grows, so
33+
# it can never keep up in a bounded CI job: the earlier v1 config timed out
34+
# every run and never completed a single scan. Rekor **v2** is tile-based: bulk
35+
# 256-entry reads make a single-worker scan outpace the log, so identity
36+
# monitoring becomes one cheap job. This rides on release signing having moved to
37+
# v2 in NVIDIA/aicr#1650 (only entries actually in v2 can be watched there).
38+
#
39+
# Selecting v2. The monitor picks its Rekor API version by matching its `url`
40+
# against the services in the TUF-distributed Sigstore SigningConfig; a match on
41+
# a v2 service switches it to v2. The resolve-v2-shard job computes that URL at
42+
# run time from the same signing config release signing uses (via `aicr trust
43+
# update --emit-signing-config`), so we never hardcode a shard and yearly shard
44+
# rotation (log2025-1 -> log2026-1 -> ...) needs no change here. Once v2 is
45+
# selected the *full* shard set is auto-discovered from the SigningConfig and
46+
# refreshed each run.
47+
#
2648
# The monitored identity is AICR's release signer (see .goreleaser.yaml and
2749
# .github/workflows/on-tag.yaml): the GitHub Actions OIDC SAN for on-tag.yaml,
2850
# issued by token.actions.githubusercontent.com.
@@ -52,8 +74,38 @@ concurrency:
5274
cancel-in-progress: false
5375

5476
jobs:
77+
# Resolve the current Rekor v2 shard from the Sigstore signing config, so the
78+
# monitor always selects v2 against a live shard without a hardcoded URL. We
79+
# read the same TUF-distributed signing config that release signing resolves
80+
# (`aicr trust update --emit-signing-config`, see pkg/trust), so the monitor
81+
# provably watches where releases actually write, and yearly shard rotation
82+
# (log2025-1 -> log2026-1 -> ...) needs no change here.
83+
resolve-v2-shard:
84+
name: Resolve current Rekor v2 shard
85+
runs-on: ubuntu-latest
86+
permissions:
87+
contents: read # checkout to build the aicr CLI
88+
outputs:
89+
url: ${{ steps.resolve.outputs.url }}
90+
steps:
91+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
92+
- uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6.5.0
93+
with:
94+
go-version-file: go.mod
95+
- id: resolve
96+
name: Resolve v2 shard URL from the signing config
97+
run: |
98+
set -euo pipefail
99+
go run ./cmd/aicr trust update --emit-signing-config signing-config.json
100+
# Newest currently-listed v2 shard (majorApiVersion 2). Any live v2
101+
# shard selects v2; the monitor then discovers the full set from TUF.
102+
url="$(jq -er '[.rekorTlogUrls[] | select(.majorApiVersion == 2)] | sort_by(.validFor.start) | last | .url' signing-config.json)"
103+
echo "Resolved Rekor v2 shard: $url"
104+
echo "url=$url" >> "$GITHUB_OUTPUT"
105+
55106
monitor:
56-
name: AICR release identity + log consistency
107+
name: AICR release identity + log consistency (Rekor v2)
108+
needs: resolve-v2-shard
57109
permissions:
58110
contents: read # checkout performed by the reusable workflow
59111
issues: write # file an issue on monitoring failure
@@ -64,6 +116,10 @@ jobs:
64116
# Must exceed the hourly cron so the consistency checkpoint never expires
65117
# between runs; wide enough to survive a brief scheduling pause.
66118
artifact_retention_days: 30
119+
# Rekor v2 shard resolved at run time from the signing config (see the
120+
# resolve-v2-shard job). Selecting a v2 service switches the monitor to v2;
121+
# it then auto-discovers the full shard set from TUF.
122+
url: ${{ needs.resolve-v2-shard.outputs.url }}
67123
config: |
68124
monitoredValues:
69125
certIdentities:

‎docs/contributor/maintaining.md‎

Lines changed: 58 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -53,17 +53,71 @@ failure during the first 60s after tag publish. Re-run the workflow.
5353

5454
The `Rekor Monitor` workflow (`.github/workflows/rekor-monitor.yaml`) runs
5555
hourly and calls the upstream `sigstore/rekor-monitor` reusable workflow. It
56-
watches the public-good Rekor transparency log for two things: that the log
57-
stays append-only (consistency), and that no entry appears under AICR's release
58-
signing identity that a release did not produce (identity). On either failure it
59-
opens an issue.
56+
watches the **Rekor v2** transparency log (where AICR release signing writes
57+
since [#1650](https://github.com/NVIDIA/aicr/issues/1650)) for two things, both
58+
in one job: that the log stays append-only (consistency), and that no entry
59+
appears under AICR's release signing identity that a release did not produce
60+
(identity). On either failure it opens an issue.
6061

6162
This protects the trust root every AICR consumer depends on: the release
6263
binaries, the signed recipe catalog, and the container images all chain to that
6364
one identity. When the workflow files an issue, follow the triage steps in the
6465
workflow file's header comment; an unrecognized identity hit should be treated
6566
as potential OIDC/key compromise.
6667

68+
### Why v2, and why identity monitoring is feasible now
69+
70+
Identity monitoring is a linear scan of every entry added to the log since the
71+
last checkpoint, because Rekor's index cannot be queried by certificate SAN and
72+
AICR's keyless release identity has no email or fixed public key to search on.
73+
On the Rekor **v1** firehose that scan runs roughly 50x slower than the log
74+
grows, so it can never keep up inside a bounded CI job: the earlier v1
75+
identity config timed out on every run and never completed a single scan
76+
([#1623](https://github.com/NVIDIA/aicr/issues/1623)). Rekor **v2** is
77+
tile-based: bulk 256-entry reads let a single worker outpace the log, so the
78+
identity scan is a cheap job that always finishes. This is why the whole design
79+
is a single unbounded scan again rather than sharded paging.
80+
81+
### Shard selection (automatic, no manual re-tune)
82+
83+
The monitor selects Rekor v2 by pointing its `url` at a v2 shard listed in the
84+
Sigstore `SigningConfig`; a match on a v2 service switches it to v2, after which
85+
it auto-discovers the **full** shard set from TUF and refreshes it every run.
86+
87+
The shard URL is **not hardcoded**. The `resolve-v2-shard` job computes it at run
88+
time from the same TUF-distributed signing config that release signing resolves,
89+
then feeds it to the monitor:
90+
91+
```bash
92+
# Same command the resolve-v2-shard job runs (works from a fresh checkout).
93+
go run ./cmd/aicr trust update --emit-signing-config signing-config.json
94+
jq -er '[.rekorTlogUrls[] | select(.majorApiVersion == 2)] | sort_by(.validFor.start) | last | .url' signing-config.json
95+
```
96+
97+
Because it reads the signing config directly, the monitor provably watches where
98+
releases actually write, and yearly shard rotation (`log2025-1` -> `log2026-1`
99+
-> ...) needs no change to this workflow. If the resolve job ever fails (for
100+
example the TUF CDN is unreachable), the monitor job is skipped for that run and
101+
retries on the next hourly tick.
102+
103+
### First run after switching from v1: reset the checkpoint
104+
105+
The `checkpoint` artifact persists a **v1** checkpoint from the prior config; a
106+
v2 run cannot parse it and will fail. After merging a change that moves this
107+
workflow to v2, delete the stale artifact once so the first v2 run establishes a
108+
fresh v2 baseline (it saves the current v2 tree head and scans forward from
109+
there):
110+
111+
```bash
112+
gh api "repos/NVIDIA/aicr/actions/artifacts?name=checkpoint" \
113+
--jq '.artifacts[].id' \
114+
| xargs -I{} gh api -X DELETE "repos/NVIDIA/aicr/actions/artifacts/{}"
115+
```
116+
117+
The first v2 run then watches forward from the current head; historical entries
118+
predating the baseline are covered by release-time verification (the `aicr
119+
verify` path), not by this monitor.
120+
67121
## Reviewing Recipe Contributions
68122

69123
A recipe PR touches `recipes/overlays/`, `recipes/mixins/`,

0 commit comments

Comments
 (0)