Skip to content

feat(ci): Rekor v2 identity monitoring for the release signer #1727

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mchmarny merged 2 commits into from
Jul 13, 2026
Merged

feat(ci): Rekor v2 identity monitoring for the release signer #1727

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
150badf
feat(ci): Rekor v2 identity monitoring for the release signer
lockwobr Jul 11, 2026
8f7d710
Merge branch 'main' into feat/rekor-v2-identity-monitor
mchmarny Jul 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 64 additions & 8 deletions .github/workflows/rekor-monitor.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,17 +12,39 @@
# See the License for the specific language governing permissions and
# limitations under the License.

# Transparency-log monitoring for AICR's release supply chain.
# Transparency-log monitoring for AICR's release supply chain, on Rekor v2.
#
# Hourly, this calls the upstream sigstore/rekor-monitor reusable workflow to:
# - Consistency: verify the public-good Rekor log stays append-only between
# runs (checkpoint persisted as the `checkpoint` artifact).
# - Identity: scan entries added since the last run for AICR's release signing
# identity. An entry under that identity that a release did not produce
# signals OIDC/key compromise.
# Hourly, this calls the upstream sigstore/rekor-monitor reusable workflow to do
# both checks in a single job (see internal/cmd MonitorLoop upstream):
# - Consistency: prove the Rekor v2 log stays append-only between runs (Merkle
# consistency from the last checkpoint to the current tree head). O(log n),
# always finishes in seconds. Checkpoint persisted as the `checkpoint`
# artifact.
# - Identity: scan entries added since the last checkpoint for AICR's release
# signing identity. An entry under that identity that a release did not
# produce signals OIDC/key compromise.
# On any failure (consistency break or matched identity), it files an issue
# (`file_issue: true`).
#
# Why v2 (NVIDIA/aicr#1623). Identity monitoring is a linear scan of every entry
# added since the last checkpoint (Rekor's index cannot be queried by
# certificate SAN, and our keyless release identity has no email or fixed key).
# On the Rekor **v1** firehose that scan runs ~50x slower than the log grows, so
# it can never keep up in a bounded CI job: the earlier v1 config timed out
# every run and never completed a single scan. Rekor **v2** is tile-based: bulk
# 256-entry reads make a single-worker scan outpace the log, so identity
# monitoring becomes one cheap job. This rides on release signing having moved to
# v2 in NVIDIA/aicr#1650 (only entries actually in v2 can be watched there).
#
# Selecting v2. The monitor picks its Rekor API version by matching its `url`
# against the services in the TUF-distributed Sigstore SigningConfig; a match on
# a v2 service switches it to v2. The resolve-v2-shard job computes that URL at
# run time from the same signing config release signing uses (via `aicr trust
# update --emit-signing-config`), so we never hardcode a shard and yearly shard
# rotation (log2025-1 -> log2026-1 -> ...) needs no change here. Once v2 is
# selected the *full* shard set is auto-discovered from the SigningConfig and
# refreshed each run.
#
# The monitored identity is AICR's release signer (see .goreleaser.yaml and
# .github/workflows/on-tag.yaml): the GitHub Actions OIDC SAN for on-tag.yaml,
# issued by token.actions.githubusercontent.com.
Expand Down Expand Up @@ -52,8 +74,38 @@ concurrency:
cancel-in-progress: false

jobs:
# Resolve the current Rekor v2 shard from the Sigstore signing config, so the
# monitor always selects v2 against a live shard without a hardcoded URL. We
# read the same TUF-distributed signing config that release signing resolves
# (`aicr trust update --emit-signing-config`, see pkg/trust), so the monitor
# provably watches where releases actually write, and yearly shard rotation
# (log2025-1 -> log2026-1 -> ...) needs no change here.
resolve-v2-shard:
name: Resolve current Rekor v2 shard
runs-on: ubuntu-latest
permissions:
contents: read # checkout to build the aicr CLI
outputs:
url: ${{ steps.resolve.outputs.url }}
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6.5.0
with:
go-version-file: go.mod
- id: resolve
name: Resolve v2 shard URL from the signing config
run: |
set -euo pipefail
go run ./cmd/aicr trust update --emit-signing-config signing-config.json
# Newest currently-listed v2 shard (majorApiVersion 2). Any live v2
# shard selects v2; the monitor then discovers the full set from TUF.
url="$(jq -er '[.rekorTlogUrls[] | select(.majorApiVersion == 2)] | sort_by(.validFor.start) | last | .url' signing-config.json)"
echo "Resolved Rekor v2 shard: $url"
echo "url=$url" >> "$GITHUB_OUTPUT"

monitor:
name: AICR release identity + log consistency
name: AICR release identity + log consistency (Rekor v2)
needs: resolve-v2-shard
permissions:
contents: read # checkout performed by the reusable workflow
issues: write # file an issue on monitoring failure
Expand All @@ -64,6 +116,10 @@ jobs:
# Must exceed the hourly cron so the consistency checkpoint never expires
# between runs; wide enough to survive a brief scheduling pause.
artifact_retention_days: 30
# Rekor v2 shard resolved at run time from the signing config (see the
# resolve-v2-shard job). Selecting a v2 service switches the monitor to v2;
# it then auto-discovers the full shard set from TUF.
url: ${{ needs.resolve-v2-shard.outputs.url }}
config: |
monitoredValues:
certIdentities:
Expand Down
62 changes: 58 additions & 4 deletions docs/contributor/maintaining.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,17 +53,71 @@ failure during the first 60s after tag publish. Re-run the workflow.

The `Rekor Monitor` workflow (`.github/workflows/rekor-monitor.yaml`) runs
hourly and calls the upstream `sigstore/rekor-monitor` reusable workflow. It
watches the public-good Rekor transparency log for two things: that the log
stays append-only (consistency), and that no entry appears under AICR's release
signing identity that a release did not produce (identity). On either failure it
opens an issue.
watches the **Rekor v2** transparency log (where AICR release signing writes
since [#1650](https://github.com/NVIDIA/aicr/issues/1650)) for two things, both
in one job: that the log stays append-only (consistency), and that no entry
appears under AICR's release signing identity that a release did not produce
(identity). On either failure it opens an issue.

This protects the trust root every AICR consumer depends on: the release
binaries, the signed recipe catalog, and the container images all chain to that
one identity. When the workflow files an issue, follow the triage steps in the
workflow file's header comment; an unrecognized identity hit should be treated
as potential OIDC/key compromise.

### Why v2, and why identity monitoring is feasible now

Identity monitoring is a linear scan of every entry added to the log since the
last checkpoint, because Rekor's index cannot be queried by certificate SAN and
AICR's keyless release identity has no email or fixed public key to search on.
On the Rekor **v1** firehose that scan runs roughly 50x slower than the log
grows, so it can never keep up inside a bounded CI job: the earlier v1
identity config timed out on every run and never completed a single scan
([#1623](https://github.com/NVIDIA/aicr/issues/1623)). Rekor **v2** is
tile-based: bulk 256-entry reads let a single worker outpace the log, so the
identity scan is a cheap job that always finishes. This is why the whole design
is a single unbounded scan again rather than sharded paging.

### Shard selection (automatic, no manual re-tune)

The monitor selects Rekor v2 by pointing its `url` at a v2 shard listed in the
Sigstore `SigningConfig`; a match on a v2 service switches it to v2, after which
it auto-discovers the **full** shard set from TUF and refreshes it every run.

The shard URL is **not hardcoded**. The `resolve-v2-shard` job computes it at run
time from the same TUF-distributed signing config that release signing resolves,
then feeds it to the monitor:

```bash
# Same command the resolve-v2-shard job runs (works from a fresh checkout).
go run ./cmd/aicr trust update --emit-signing-config signing-config.json
jq -er '[.rekorTlogUrls[] | select(.majorApiVersion == 2)] | sort_by(.validFor.start) | last | .url' signing-config.json
```
Comment thread
coderabbitai[bot] marked this conversation as resolved.

Because it reads the signing config directly, the monitor provably watches where
releases actually write, and yearly shard rotation (`log2025-1` -> `log2026-1`
-> ...) needs no change to this workflow. If the resolve job ever fails (for
example the TUF CDN is unreachable), the monitor job is skipped for that run and
retries on the next hourly tick.

### First run after switching from v1: reset the checkpoint

The `checkpoint` artifact persists a **v1** checkpoint from the prior config; a
v2 run cannot parse it and will fail. After merging a change that moves this
workflow to v2, delete the stale artifact once so the first v2 run establishes a
fresh v2 baseline (it saves the current v2 tree head and scans forward from
there):

```bash
gh api "repos/NVIDIA/aicr/actions/artifacts?name=checkpoint" \
--jq '.artifacts[].id' \
| xargs -I{} gh api -X DELETE "repos/NVIDIA/aicr/actions/artifacts/{}"
```
Comment on lines +111 to +115

@coderabbitai coderabbitai Bot Jul 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n## docs/contributor/maintaining.md (around lines 111-115)\n'
nl -ba docs/contributor/maintaining.md | sed -n '100,125p'

printf '\n## Search for related artifact cleanup guidance\n'
rg -n "actions/artifacts|checkpoint cleanup|checkpoint" docs .github -g '!**/node_modules/**' || true

Repository: NVIDIA/aicr

Length of output: 248

🌐 Web query:

GitHub REST API actions artifacts list pagination per_page page documentation

💡 Result:

The GitHub REST API endpoints for listing artifacts support pagination via the per_page and page query parameters [1][2]. Key details for these parameters: - per_page: Defines the number of results to return per page. The maximum value allowed is 100, and the default value is 30 [1][2]. - page: Specifies the page number of the results to retrieve, with a default value of 1 [1][2]. These parameters are applicable to endpoints such as List artifacts for a repository and List workflow run artifacts [1][2]. For more advanced pagination needs, the GitHub REST API uses Link headers to provide URLs for accessing subsequent pages of data [3][4]. Users can also leverage client libraries like Octokit.js, which provides helper methods like octokit.paginate to automatically handle fetching all pages of results [3].

Citations:

🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
from pathlib import Path
p = Path("docs/contributor/maintaining.md")
for i, line in enumerate(p.read_text().splitlines(), 1):
    if 100 <= i <= 125:
        print(f"{i:4d}: {line}")
PY

printf '\n## Related mentions\n'
python3 - <<'PY'
from pathlib import Path
import re
for path in [Path("docs/contributor/maintaining.md")]:
    text = path.read_text()
    for m in re.finditer(r'checkpoint|artifact|gh api', text, re.I):
        line = text.count("\n", 0, m.start()) + 1
        print(f"{path}:{line}")
PY

Repository: NVIDIA/aicr

Length of output: 1897

🌐 Web query:

GitHub artifacts list endpoint name filter pagination behavior gh api actions/artifacts?name=checkpoint

💡 Result:

The GitHub Actions artifacts API endpoints (e.g., GET /repos/{owner}/{repo}/actions/artifacts) support both a name filter and standard pagination parameters (per_page and page) [1][2][3]. When you include the name query parameter (e.g., ?name=checkpoint), the API filters the returned list to only include artifacts matching that specific name [1][4]. Because the API continues to support pagination, the name filter applies to the entire set of artifacts, and the paginated results will contain only the artifacts that match the provided name [1][5]. Key behaviors to note: 1. Pagination Parameters: The endpoint uses standard page and per_page parameters [1][3]. You should follow the link header in the API response to reliably navigate through paginated results, rather than manually constructing page numbers, as the latter can be error-prone [6][7]. 2. Combining Filters: The name filter can be used in conjunction with pagination [1][5]. If you request actions/artifacts?name=checkpoint&per_page=100, the API will return a paginated list of artifacts where every item matches the name "checkpoint" [1]. 3. Tool Usage: When using the GitHub CLI (gh api), ensure you are targeting the correct endpoint. If you encounter issues with manual pagination, rely on the link header provided in the API response or use higher-level CLI commands where available to handle the iteration for you [8][7]. In summary, the name filter is a server-side parameter that restricts the dataset before pagination occurs, ensuring that all subsequent pages of the result set adhere to your specified filter [1].

Citations:

Paginate the checkpoint cleanup. gh api only returns the first page here, so once checkpoint artifacts exceed the default page size, older stale entries can be left behind. Add --paginate or an explicit page loop so every checkpoint artifact is deleted.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/contributor/maintaining.md` around lines 111 - 115, Update the
checkpoint cleanup command in the maintenance documentation to paginate the
artifact listing, using gh api’s --paginate option or an equivalent explicit
page loop, while preserving extraction of each artifact ID and deletion of all
checkpoint artifacts.


The first v2 run then watches forward from the current head; historical entries
predating the baseline are covered by release-time verification (the `aicr
verify` path), not by this monitor.
Comment on lines +117 to +119

@coderabbitai coderabbitai Bot Jul 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the referenced doc section with line numbers
sed -n '100,140p' docs/contributor/maintaining.md | cat -n

# Find references to the v2 run / checkpoint / verify wording
rg -n "v2 run|checkpoint|aicr verify|Rekor|identity scan|historical" docs pkg cmd .github -S

# Map likely verification-related files before reading
fd -a "verify" docs pkg cmd tools | sed -n '1,200p'

Repository: NVIDIA/aicr

Length of output: 50369

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Read the user-facing verification docs around the relevant claims
sed -n '160,190p' docs/user/artifact-verification.md | cat -n
sed -n '2550,2635p' docs/user/cli-reference.md | cat -n
sed -n '1,90p' docs/integrator/supply-chain-verification.md | cat -n

# Inspect the verifier implementation for any Rekor history scanning
ast-grep outline pkg/bundler/attestation/verifying.go --view expanded
ast-grep outline pkg/evidence/verifier/verify.go --view expanded

# Search for code paths that query Rekor entries by identity, history, or monitor-style scanning
rg -n "history|historical|identity scan|certificate SAN|Rekor.*search|rekor.*search|entries.*identity|query.*Rekor|scan forward|baseline" pkg docs -S

Repository: NVIDIA/aicr

Length of output: 29801

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Read the surrounding maintenance guidance, including the historical audit section
sed -n '288,325p' docs/contributor/maintaining.md | cat -n

# Inspect the verifier implementation paths for any identity-history scan behavior
sed -n '1,220p' pkg/bundler/attestation/verifying.go | cat -n
sed -n '1,280p' pkg/evidence/verifier/verify.go | cat -n

# Look for any Rekor search-by-identity or history scan code
rg -n "search.*identity|identity.*search|SAN|history scan|historical scan|Rekor.*UUID|rekor-cli search|search --sha|search --email|certificate identity" pkg docs -S

Repository: NVIDIA/aicr

Length of output: 29915

Add a one-time Rekor history audit before dropping the v1 checkpoint. The v2 monitor only advances from the new head, and aicr verify checks individual release artifacts; neither covers an unseen pre-baseline identity entry on its own. If that gap is acceptable, document it explicitly; otherwise keep the v1 checkpoint until a historical audit runs.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/contributor/maintaining.md` around lines 117 - 119, Add a one-time Rekor
historical audit step before removing the v1 checkpoint, covering entries
predating the v2 baseline and detecting unseen identity entries. Alternatively,
explicitly document that this historical gap is accepted; update the v2 monitor,
`aicr verify`, and checkpoint guidance accordingly.


## Reviewing Recipe Contributions

A recipe PR touches `recipes/overlays/`, `recipes/mixins/`,
Expand Down
Loading