feat(ci): own Rekor v2 identity monitoring via tools/rekor-monitor

Replace the merged-but-broken reusable-workflow approach (#1727) with an
owned monitor that can actually watch the public-good Rekor v2.

The upstream sigstore/rekor-monitor reusable workflow resolves its Rekor API
version and shards from Sigstore's default signing config
(signing_config.v0.2.json), which is v1-only and stays that way for the
foreseeable future. AICR opted into Rekor v2 early via the separate
signing_config_rekor_v2.v0.2.json TUF target, which upstream never reads and
cannot be pointed at, so passing it a v2 shard URL falls through to v1 and its
v1 client times out against the v2 tile endpoint. That is why the merged
workflow fails every hour on main.

tools/rekor-monitor closes only the AICR-specific gap: it reads the v2 signing
config AICR signs against (pkg/trust) and reuses the upstream rekor-monitor
library packages (pkg/rekor/v2, pkg/tiles, pkg/identity) for the
security-critical consistency proof and identity search, so we do not
reimplement transparency-log verification. When Sigstore makes v2 the ecosystem
default the reusable workflow can monitor v2 directly and this tool can be
retired.

- tools/rekor-monitor: doc.go (rationale), main.go (run orchestration),
  monitor.go (shards + identity checks + outcome), checkpoint.go (cursor +
  native zip restore), with unit tests and a README.
- .github/workflows/rekor-monitor.yaml: run the tool; fetch the prior
  checkpoint artifact via the API and restore it natively; open/close a
  deduplicated alert issue on failure/success. New artifact name so the stale
  v1 checkpoint is ignored (no migration).
- go.mod/vendor: add sigstore/rekor-monitor + transparency-dev/formats.
- docs/contributor/maintaining.md: document why upstream cannot do this yet.

Signed-off-by: Brian Lockwood <lockwobr@gmail.com>

Author: lockwobr

.github/workflows/rekor-monitor.yaml

@@ -52,12 +52,12 @@ failure during the first 60s after tag publish. Re-run the workflow.
 ## Release Supply-Chain Monitoring
 
 The `Rekor Monitor` workflow (`.github/workflows/rekor-monitor.yaml`) runs
-hourly and calls the upstream `sigstore/rekor-monitor` reusable workflow. It
-watches the **Rekor v2** transparency log (where AICR release signing writes
-since [#1650](https://github.com/NVIDIA/aicr/issues/1650)) for two things, both
-in one job: that the log stays append-only (consistency), and that no entry
-appears under AICR's release signing identity that a release did not produce
-(identity). On either failure it opens an issue.
+hourly and runs our own monitor, `tools/rekor-monitor`, against the **Rekor v2**
+transparency log (where AICR release signing writes since
+[#1650](https://github.com/NVIDIA/aicr/issues/1650)). In one job it checks two
+things: that the log stays append-only (consistency), and that no entry appears
+under AICR's release signing identity that a release did not produce (identity).
+On any failure it opens a tracking issue; a later clean run closes it.
 
 This protects the trust root every AICR consumer depends on: the release
 binaries, the signed recipe catalog, and the container images all chain to that
@@ -75,48 +75,49 @@ grows, so it can never keep up inside a bounded CI job: the earlier v1
 identity config timed out on every run and never completed a single scan
 ([#1623](https://github.com/NVIDIA/aicr/issues/1623)). Rekor **v2** is
 tile-based: bulk 256-entry reads let a single worker outpace the log, so the
-identity scan is a cheap job that always finishes. This is why the whole design
-is a single unbounded scan again rather than sharded paging.
-
-### Shard selection (automatic, no manual re-tune)
-
-The monitor selects Rekor v2 by pointing its `url` at a v2 shard listed in the
-Sigstore `SigningConfig`; a match on a v2 service switches it to v2, after which
-it auto-discovers the **full** shard set from TUF and refreshes it every run.
-
-The shard URL is **not hardcoded**. The `resolve-v2-shard` job computes it at run
-time from the same TUF-distributed signing config that release signing resolves,
-then feeds it to the monitor:
+identity scan is a cheap job that always finishes.
+
+### Why our own tool, not the upstream reusable workflow
+
+The upstream `sigstore/rekor-monitor` reusable workflow selects its Rekor API
+version and discovers shards from Sigstore's **default** signing config,
+`signing_config.v0.2.json`. That config lists only Rekor v1 and, per Sigstore's
+[rekor-evolution](https://blog.sigstore.dev/rekor-evolution/) plan, keeps v1 as
+the ecosystem default "for the foreseeable future". AICR opted into v2 **early**
+via a separate TUF target, `signing_config_rekor_v2.v0.2.json` (see `pkg/trust`),
+which the upstream tool never reads and exposes no flag to select. So pointing
+it at a v2 shard URL just falls through to v1 and fails.
+
+`tools/rekor-monitor` closes exactly that gap: it reads the v2 signing config
+AICR actually signs against (`trust.ResolveSigningConfig`) and then reuses the
+upstream rekor-monitor **library** packages for the security-critical work (tile
+consistency proofs and identity search), so we do not reimplement
+transparency-log verification. To inspect the current v2 shard the way the tool
+resolves it:
 
 ```bash
-# Same command the resolve-v2-shard job runs (works from a fresh checkout).
 go run ./cmd/aicr trust update --emit-signing-config signing-config.json
 jq -er '[.rekorTlogUrls[] | select(.majorApiVersion == 2)] | sort_by(.validFor.start) | last | .url' signing-config.json
 ```
 
-Because it reads the signing config directly, the monitor provably watches where
-releases actually write, and yearly shard rotation (`log2025-1` -> `log2026-1`
--> ...) needs no change to this workflow. If the resolve job ever fails (for
-example the TUF CDN is unreachable), the monitor job is skipped for that run and
-retries on the next hourly tick.
-
-### First run after switching from v1: reset the checkpoint
-
-The `checkpoint` artifact persists a **v1** checkpoint from the prior config; a
-v2 run cannot parse it and will fail. After merging a change that moves this
-workflow to v2, delete the stale artifact once so the first v2 run establishes a
-fresh v2 baseline (it saves the current v2 tree head and scans forward from
-there):
-
-```bash
-gh api "repos/NVIDIA/aicr/actions/artifacts?name=checkpoint" \
-  --jq '.artifacts[].id' \
-  | xargs -I{} gh api -X DELETE "repos/NVIDIA/aicr/actions/artifacts/{}"
-```
-
-The first v2 run then watches forward from the current head; historical entries
-predating the baseline are covered by release-time verification (the `aicr
-verify` path), not by this monitor.
+When Sigstore makes v2 the ecosystem default, `signing_config.v0.2.json` will
+list the v2 shards, the upstream reusable workflow can monitor v2 directly, and
+this tool can be retired. Until then upstream exposes no flag to point the
+monitor at a non-default signing config (it always reads
+`signing_config.v0.2.json`); a feature request for that would let early v2
+adopters drop this tool.
+
+### Checkpoint and first run
+
+The monitor persists its cursor as the `rekor-v2-checkpoint` artifact between
+runs (a deliberately fresh name, so the stale v1 `checkpoint` artifact from the
+earlier design is simply ignored, no migration). The **first** run has no prior
+checkpoint, so it establishes a baseline at the current v2 tree head and skips
+the identity scan; every run after that scans only the newly-added window.
+Entries predating the baseline are covered by release-time verification (the
+`aicr verify` path), not by this monitor. Shard rotation (`log2025-1` ->
+`log2026-1` -> ...) needs no change here: the tool reads the live shard set from
+the signing config every run.
 
 ## Reviewing Recipe Contributions
 

THIRD_PARTY_NOTICES.md

@@ -17,6 +17,7 @@ require (
 	github.com/opencontainers/image-spec v1.1.1
 	github.com/prometheus/client_golang v1.23.2
 	github.com/sigstore/protobuf-specs v0.5.1
+	github.com/sigstore/rekor-monitor v0.0.0-20260622135518-170374c6119e
 	github.com/sigstore/sigstore v1.10.8
 	github.com/sigstore/sigstore-go v1.2.2
 	github.com/sigstore/sigstore/pkg/signature/kms/aws v1.10.8
@@ -25,6 +26,7 @@ require (
 	github.com/sigstore/sigstore/pkg/signature/kms/hashivault v1.10.8
 	github.com/stretchr/testify v1.11.1
 	github.com/theupdateframework/go-tuf/v2 v2.4.2
+	github.com/transparency-dev/formats v0.1.1
 	github.com/urfave/cli/v3 v3.10.1
 	golang.org/x/mod v0.38.0
 	golang.org/x/sync v0.22.0
@@ -189,8 +191,8 @@ require (
 	github.com/spf13/pflag v1.0.10 // indirect
 	github.com/stretchr/objx v0.5.3 // indirect
 	github.com/theupdateframework/go-tuf v0.7.0 // indirect
-	github.com/transparency-dev/formats v0.1.1 // indirect
 	github.com/transparency-dev/merkle v0.0.2 // indirect
+	github.com/transparency-dev/tessera v1.0.2 // indirect
 	github.com/x448/float16 v0.8.4 // indirect
 	github.com/xlab/treeprint v1.2.0 // indirect
 	github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 // indirect

docs/contributor/maintaining.md

@@ -286,6 +286,8 @@ github.com/hashicorp/go-secure-stdlib/strutil v0.1.2 h1:kes8mmyCpxJsI7FTwtzRqEy9
 github.com/hashicorp/go-secure-stdlib/strutil v0.1.2/go.mod h1:Gou2R9+il93BqX25LAKCLuM+y9U2T4hlwvT1yprcna4=
 github.com/hashicorp/go-sockaddr v1.0.7 h1:G+pTkSO01HpR5qCxg7lxfsFEZaG+C0VssTy/9dbT+Fw=
 github.com/hashicorp/go-sockaddr v1.0.7/go.mod h1:FZQbEYa1pxkQ7WLpyXJ6cbjpT8q0YgQaK/JakXqGyWw=
+github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
+github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
 github.com/hashicorp/hcl v1.0.1-vault-7 h1:ag5OxFVy3QYTFTJODRzTKVZ6xvdfLLCA1cy/Y6xGI0I=
 github.com/hashicorp/hcl v1.0.1-vault-7/go.mod h1:XYhtn6ijBSAj6n4YqAaf7RBPS4I06AItNorpy+MoQNM=
 github.com/hashicorp/vault/api v1.22.0 h1:+HYFquE35/B74fHoIeXlZIP2YADVboaPjaSicHEZiH0=
@@ -417,6 +419,8 @@ github.com/sigstore/protobuf-specs v0.5.1 h1:/5OPaNuolRJmQfeZLayJGFXMpsRJEdgC6ah
 github.com/sigstore/protobuf-specs v0.5.1/go.mod h1:DRBzpFuE+LnvQMN10/dU6nBeKwVLGEQ6o2FovN2Rats=
 github.com/sigstore/rekor v1.5.3 h1:0Tyolw3zreRgm7PUW8dccFLXGBThi08278jI8EXNSr4=
 github.com/sigstore/rekor v1.5.3/go.mod h1:h3GK5dDqCcWJJZUJwdpKGSSmEV2GEjPUjJy3WTjBwzA=
+github.com/sigstore/rekor-monitor v0.0.0-20260622135518-170374c6119e h1:Ny+E65WYMRlniv/Z7el+d3FZ+9HAU/WCBk0d10qh6kU=
+github.com/sigstore/rekor-monitor v0.0.0-20260622135518-170374c6119e/go.mod h1:gR0/nivRiMAwhovKEomBpLn4t0R5W5IGoyxMyDSLpoI=
 github.com/sigstore/rekor-tiles/v2 v2.3.0 h1:HhMgH61UP0t899V8Fjt7pz1YdgOBptbaQdnCF+79cdc=
 github.com/sigstore/rekor-tiles/v2 v2.3.0/go.mod h1:DEFiKSyQ4nF75QRVNdOPaIH3cmvMkO2B6xDZjNYngPc=
 github.com/sigstore/sigstore v1.10.8 h1:1Mgkxvkw4AXMfIP1DOjc6kw0GkUgA8pGVpveN/EfOq4=
@@ -473,6 +477,8 @@ github.com/transparency-dev/formats v0.1.1 h1:4bVHJc+KdBgpA1OJD1yjI+g0i5Z1graCpp
 github.com/transparency-dev/formats v0.1.1/go.mod h1:qtZ8goRuJ8FTBG9c9+Bj0rn2rUG7eG/AUTkr+Aw3jFw=
 github.com/transparency-dev/merkle v0.0.2 h1:Q9nBoQcZcgPamMkGn7ghV8XiTZ/kRxn1yCG81+twTK4=
 github.com/transparency-dev/merkle v0.0.2/go.mod h1:pqSy+OXefQ1EDUVmAJ8MUhHB9TXGuzVAT58PqBoHz1A=
+github.com/transparency-dev/tessera v1.0.2 h1:PNfGPFfJHpCFVswlrsQvRghxqYz2xy/OtP8qTjJuzFQ=
+github.com/transparency-dev/tessera v1.0.2/go.mod h1:WD/EMM6RXWRyImk9yyJ2hrs8xdknN/lpwUrFR2GemfU=
 github.com/urfave/cli/v3 v3.10.1 h1:7Kx9H50hrHbRbyxgO1KP6/BcbiGRz0uYh5YyQ30JEEY=
 github.com/urfave/cli/v3 v3.10.1/go.mod h1:ysVLtOEmg2tOy6PknnYVhDoouyC/6N42TMeoMzskhso=
 github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
@@ -517,8 +523,8 @@ go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRk
 go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA=
 go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
 go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
-go.step.sm/crypto v0.77.7 h1:6azC+pD678Vjju8yXnMDHCZJ+HzFaEmL3sCryiezTIA=
-go.step.sm/crypto v0.77.7/go.mod h1:OW/2sEHwTtDKq70PvSQ5B0JGy/CrLyDKOiVy3YvZMTQ=
+go.step.sm/crypto v0.83.0 h1:llCPEiL2f+kUPY+CUJrOCsuSBoJZ2qooFG9EqGast6w=
+go.step.sm/crypto v0.83.0/go.mod h1:qyLTv666WJ6ImFPUjljux+684Y/GGYUjAZcKCnc6yBs=
 go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
 go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
 go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=

go.mod

@@ -0,0 +1,78 @@
+# rekor-monitor: AICR release-signer transparency-log monitor (Rekor v2)
+
+Watches the Rekor **v2** transparency log for AICR's release supply chain. On
+each run it does two checks and exits non-zero if either is unhappy (so the
+calling workflow can alert):
+
+- **Consistency**: proves the log stayed append-only from the last checkpoint
+  to the current tree head (an O(log n) tile proof). Its main job here is to
+  *anchor the incremental identity scan*: it guarantees nothing was removed or
+  rewritten between runs, which is what makes "scan only the new window" sound
+  (otherwise a rewritten view could hide a malicious entry). It is also the
+  standard append-only tamper check, though ecosystem witnesses are the primary
+  guarantee of that.
+- **Identity**: scans the entries added since the last checkpoint for AICR's
+  release signing identity. An entry under that identity that no release
+  produced signals OIDC/key compromise. This is the AICR-specific check.
+
+It is run hourly by `.github/workflows/rekor-monitor.yaml`.
+
+## Why not the upstream `sigstore/rekor-monitor` reusable workflow?
+
+**Because upstream cannot monitor the public-good Rekor v2 that AICR signs to.**
+
+The upstream reusable workflow decides which Rekor API version to talk to (and
+which shards to read) from Sigstore's **default** signing config,
+`signing_config.v0.2.json`. That config lists **only Rekor v1**, and per
+Sigstore's [rekor-evolution](https://blog.sigstore.dev/rekor-evolution/) plan it
+keeps v1 as the ecosystem default "for the foreseeable future".
+
+AICR opted into Rekor v2 **early** (see [#1650](https://github.com/NVIDIA/aicr/issues/1650))
+via a *separate* TUF target, `signing_config_rekor_v2.v0.2.json`, which the
+upstream tool never reads and exposes no flag to select. Both its version
+detection (`getRekorVersion`) and its v2 shard discovery (`RefreshSigningConfig`)
+go through `root.GetSigningConfig`, which is hardcoded to the v1-only default
+target. So passing it a v2 shard URL just falls through to v1, and its v1 client
+then times out against the v2 tile endpoint. (We tried exactly that first; see
+the history on [#1623](https://github.com/NVIDIA/aicr/issues/1623).)
+
+This tool changes **only that one AICR-specific bit**: it reads the v2 signing
+config AICR actually signs against (`pkg/trust`), and otherwise **reuses the
+upstream rekor-monitor library packages** (`pkg/rekor/v2`, `pkg/tiles`,
+`pkg/identity`) for the security-critical verification, so we are not
+reimplementing transparency-log crypto, just pointing it at the right config.
+
+When Sigstore makes v2 the ecosystem default, `signing_config.v0.2.json` will
+list the v2 shards, the upstream reusable workflow can monitor v2 directly, and
+**this tool can be retired**. Until then, upstream has no flag to target a
+non-default signing config.
+
+## Layout
+
+| File | Responsibility |
+|------|----------------|
+| `doc.go` | Package doc: rationale (above) and structure, surfaced by `go doc`. |
+| `main.go` | Flags, `run()` orchestration, process exit codes. |
+| `monitor.go` | `monitor` (resolved v2 shards + watched identity) and its two checks; the `outcome` of a pass. |
+| `checkpoint.go` | `checkpointStore`: restore the cursor from the fetched artifact zip, read the last checkpoint, write the new one. |
+
+## Usage
+
+```bash
+go run ./tools/rekor-monitor \
+  --file checkpoint_v2.txt \
+  --restore-zip checkpoint.zip \
+  --cert-subject '^https://github\.com/NVIDIA/aicr/\.github/workflows/on-tag\.yaml@refs/tags/.*$' \
+  --cert-issuer '^https://token\.actions\.githubusercontent\.com$'
+```
+
+| Flag | Purpose |
+|------|---------|
+| `--file` | Path to the persisted v2 checkpoint (the cursor). Missing/empty = first run: baseline at the current head and skip the identity scan. |
+| `--restore-zip` | Optional GitHub-artifact zip to seed `--file` from before monitoring. Missing file = first run. |
+| `--cert-subject` | Regex for the monitored certificate SAN. Empty = consistency-only (no identity scan). |
+| `--cert-issuer` | Regex for the monitored certificate issuer (requires `--cert-subject`). |
+| `--user-agent` | User-Agent for requests to the log. |
+
+Exit status: `0` clean; non-zero on a consistency break, a scan error, or an
+identity match.