Skip to content

Commit 39d5867

Browse files
authored
feat(sdk): expose verification and signing on pkg/client/v1 (NVIDIA#2218)
Signed-off-by: Mark Chmarny <mark@chmarny.com>
1 parent 9591f11 commit 39d5867

32 files changed

Lines changed: 2471 additions & 169 deletions

docs/contributor/api-server.md

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -183,7 +183,7 @@ Environment variables read at startup:
183183
| `AICR_SIGNING_CONFIG_PATH` | unset | `parseSigningConfig` (both modes; Rekor v2) |
184184
| `AICR_TLOG_UPLOAD` | `true` | `parseSigningConfig` (Mode A only) |
185185
| `AICR_BINARY_ATTESTATION_FILE` | unset → `<executable>-attestation.sigstore.json` next to the binary | `resolveBinaryAttestationPath` (override for ko `KO_DATA_PATH` layouts) |
186-
| `AICR_BINARY_ATTESTATION_IDENTITY_REGEXP` | unset → `verifier.TrustedRepositoryPattern` (release `on-tag.yaml`) | `resolveBinaryAttestationIdentityPattern` (must contain `NVIDIA/aicr`, validated by `verifier.ValidateIdentityPattern`; retargets the attesting NVIDIA workflow, e.g. an e2e build) |
186+
| `AICR_BINARY_ATTESTATION_IDENTITY_REGEXP` | unset → `verifier.TrustedRepositoryPattern` (release `on-tag.yaml`) | `resolveBinaryAttestationIdentityPattern` (must be confined to `NVIDIA/aicr` — begins with the repository prefix, no top-level alternation, and no match against foreign-identity canaries — validated by `verifier.ValidateIdentityPattern`; retargets the attesting NVIDIA workflow, e.g. an e2e build) |
187187

188188
See [Server-Side Bundle Signing](#server-side-bundle-signing) for the identity
189189
model and validation rules behind these variables.
@@ -262,9 +262,12 @@ the running `os.Executable()` binary's digest.
262262
pattern the attestation is verified against: `verifier.TrustedRepositoryPattern`
263263
(the release `on-tag.yaml` workflow) by default, or the
264264
`AICR_BINARY_ATTESTATION_IDENTITY_REGEXP` override when set. The override is
265-
validated by `verifier.ValidateIdentityPattern`, which requires it to contain
266-
`NVIDIA/aicr`, so it can only retarget which NVIDIA workflow attested the binary
267-
(e.g. the server-kms e2e build), never widen the org. A bad override fails
265+
validated by `verifier.ValidateIdentityPattern`, which requires it to *begin
266+
with* `https://github.com/NVIDIA/aicr/` (a leading `^` is allowed) and to avoid
267+
top-level alternation, so it can only retarget which NVIDIA workflow attested
268+
the binary (e.g. the server-kms e2e build), never widen the org. Merely
269+
containing `NVIDIA/aicr` is not enough: a pattern that reaches the repository
270+
down one branch and something else down another is rejected. A bad override fails
268271
startup fast. This mirrors the CLI's `--certificate-identity-regexp`, and a
269272
custom pattern is logged because bundles the server then signs will not pass a
270273
verifier using the default identity.

docs/integrator/go-library.md

Lines changed: 243 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -443,6 +443,232 @@ generated from an externally-decoded recipe reloadable by the file loader. The
443443
caller's own `RecipeResult` is never mutated, and `APIVersion` is validated but
444444
never rewritten.
445445

446+
## Verifying artifacts
447+
448+
Every artifact AICR produces can be checked back through the same facade,
449+
so an integrator never has to reach into `pkg/bundler/verifier`,
450+
`pkg/evidence/verifier`, or `pkg/recipe/catalog` to establish trust.
451+
452+
### Verifying a bundle
453+
454+
`VerifyBundle` checks a bundle's checksums and attestation chain, then
455+
evaluates the policy assertions you supply:
456+
457+
```go
458+
verification, err := client.VerifyBundle(ctx, "./my-bundle", aicr.BundleVerifyOptions{
459+
MinTrustLevel: "verified",
460+
RequireCreator: "release@nvidia.com",
461+
})
462+
if err != nil {
463+
log.Fatal(err) // could not verify: missing bundle, bad trust root, bad options
464+
}
465+
if verification.PolicyFailure != "" {
466+
log.Fatalf("policy not met: %s", verification.PolicyFailure)
467+
}
468+
if len(verification.Report.Errors) > 0 {
469+
log.Fatalf("verification failed: %v", verification.Report.Errors)
470+
}
471+
log.Printf("trust level %s, created by %s",
472+
verification.Report.TrustLevel, verification.Report.BundleCreator)
473+
```
474+
475+
A failed policy is **data, not an error**: the call still returns the
476+
full `Report` so you can log or render why the bundle fell short. A
477+
non-nil error means verification could not run at all.
478+
479+
`MinTrustLevel` is the one field whose empty value is not "no
480+
constraint". Leaving it empty means `"max"` — auto-detect the highest
481+
level this bundle could achieve and require it. Naming a level
482+
explicitly (`aicr.TrustLevels()` returns the valid values) can *lower*
483+
the floor as readily as raise it.
484+
485+
Verification is offline: the chain resolves against the locally cached
486+
or embedded Sigstore trusted root. The one network path is a KMS URI in
487+
`Key`, which makes a live `GetPublicKey` call.
488+
489+
`BundleVerifyOptions` mirrors the [`spec.verify`](../user/cli-config.md#specverify)
490+
section of an `AICRConfig` field-for-field — the first three fields come from
491+
`spec.verify.trust`, the next three from `spec.verify.policy` — so a
492+
team that has standardized on a committed policy can populate this
493+
struct without a translation table. `IgnoreTLog` deliberately has no
494+
config counterpart: it drops the transparency-log requirement, and
495+
keeping it out of the schema means a checked-in file can never silently
496+
disable that check.
497+
498+
### Verifying evidence
499+
500+
`VerifyEvidence` checks a recipe-evidence bundle's signature and hash
501+
chain. The input is auto-detected as a pointer file, an OCI reference,
502+
or an unpacked directory:
503+
504+
```go
505+
result, err := client.VerifyEvidence(ctx, aicr.EvidenceVerifyOptions{
506+
Input: "recipes/evidence/h100-eks-training/eks/sha256-abc.yaml",
507+
})
508+
if err != nil {
509+
log.Fatal(err) // verification could not be attempted
510+
}
511+
512+
switch result.Exit {
513+
case aicr.EvidenceExitValidPassed:
514+
log.Printf("valid: %s", result.RecipeName)
515+
case aicr.EvidenceExitValidPhaseFailures:
516+
log.Printf("evidence sound, but recorded phases failed")
517+
case aicr.EvidenceExitInvalid:
518+
log.Fatalf("bundle invalid")
519+
case aicr.EvidenceExitIncomplete:
520+
if result.FailureCause != nil && result.FailureCause.Class == aicr.EvidenceCauseCanceled {
521+
log.Fatalf("run canceled before a verdict")
522+
}
523+
log.Fatalf("could not read the bundle (storage or registry fault)")
524+
}
525+
526+
fmt.Print(aicr.RenderEvidenceMarkdown(result))
527+
```
528+
529+
An invalid bundle is a verdict, not an error — branch on `Exit`. The
530+
`EvidenceExitIncomplete` case is the one worth handling separately: it
531+
means "we could not check this", which is different from "we checked it
532+
and it failed".
533+
534+
Pair it with `RecipeDigest` to detect evidence that has gone stale
535+
against the recipe on your branch:
536+
537+
```go
538+
current, err := client.RecipeDigest(ctx, aicr.RecipeDigestOptions{
539+
Path: "recipes/overlays/h100-eks-training.yaml",
540+
})
541+
if err != nil {
542+
log.Fatal(err)
543+
}
544+
if result.Predicate.Recipe.Digest != current {
545+
log.Fatal("evidence is stale: the recipe changed since it was signed")
546+
}
547+
```
548+
549+
### Verifying the recipe catalog
550+
551+
`VerifyCatalog` recomputes the deterministic digest over the Client's
552+
recipe catalog and verifies it against the Sigstore bundle shipped as
553+
the `recipe-catalog.sigstore.json` release asset:
554+
555+
```go
556+
catalog, err := client.VerifyCatalog(ctx, "./recipe-catalog.sigstore.json",
557+
aicr.CatalogVerifyOptions{})
558+
if err != nil {
559+
log.Fatalf("catalog verification failed: %v", err)
560+
}
561+
log.Printf("catalog sha256:%s signed by %s", catalog.Digest, catalog.Identity)
562+
```
563+
564+
The digest is computed over **this Client's** `DataProvider`, not the
565+
process-wide embedded catalog. A Client built on `EmbeddedSource()`
566+
verifies the catalog NVIDIA signed. A Client whose source layers
567+
external data over the embedded tree is verifying different content, so
568+
verification will not match the released signature — that is the
569+
correct answer to "is the catalog I am resolving against the signed
570+
one", not a bug.
571+
572+
### Verifying the binary
573+
574+
`VerifyBinaryAttestation` proves an `aicr` binary was built by NVIDIA
575+
CI. It is package-level rather than a `Client` method because it
576+
involves no recipe catalog and no configurable policy:
577+
578+
```go
579+
identity, err := aicr.VerifyBinaryAttestation(ctx, aicr.BinaryAttestationVerifyOptions{
580+
Attestation: attestationBytes, // raw Sigstore bundle
581+
BinaryDigest: rawSHA256, // raw bytes, not hex
582+
})
583+
```
584+
585+
Passing bytes rather than a path is deliberate: it lets you verify the
586+
exact content you are about to use, with no verify-then-reread window.
587+
Override the pinned identity with `IdentityRegexp` (defaults to
588+
`aicr.TrustedIdentityPattern`); `aicr.ValidateIdentityPattern`
589+
pre-validates operator-supplied input against the same rule the verify
590+
entry points apply internally.
591+
592+
An override must be *confined* to the NVIDIA repository, not merely
593+
mention it. Two rules enforce that, and they are load-bearing together:
594+
the pattern must **begin with** `https://github.com/NVIDIA/aicr/` (a
595+
leading `^` is allowed, and `github\.com` is accepted too), and it must
596+
not use **top-level alternation**. Both exist because the identity
597+
matcher pins only the OIDC issuer beyond this pattern, so a widened
598+
pattern silently degrades the gate to "any GitHub Actions workflow in
599+
any repository" rather than failing visibly.
600+
601+
```go
602+
// Rejected: begins with the prefix, but the second branch matches
603+
// anything — only one branch of an alternation has to match.
604+
aicr.ValidateIdentityPattern(`^https://github\.com/NVIDIA/aicr/.*|.*$`)
605+
606+
// Rejected: the alternation is nested in a group, so the pattern no
607+
// longer begins with the prefix and one branch escapes the repository.
608+
aicr.ValidateIdentityPattern(
609+
`(https://github.com/NVIDIA/aicr/.*|https://github.com/attacker/x/.*)`)
610+
611+
// Accepted: alternatives sit AFTER the prefix, so every branch is
612+
// already behind the pin.
613+
aicr.ValidateIdentityPattern(
614+
`^https://github\.com/NVIDIA/aicr/\.github/workflows/(on-tag|release)\.yaml@.*`)
615+
```
616+
617+
## Signing artifacts
618+
619+
The producing half of the supply chain is on the facade too.
620+
`EmitRecipeEvidence` builds a bundle from a completed validation run;
621+
`PublishEvidence` signs and pushes one that already exists on disk:
622+
623+
```go
624+
err := client.PublishEvidence(ctx, aicr.EvidencePublishOptions{
625+
BundleDir: "./out",
626+
Push: "ghcr.io/myorg/aicr-evidence",
627+
})
628+
```
629+
630+
Splitting emit from publish lets the cluster-bound step run where the
631+
cluster is reachable and the Sigstore-bound step run where Fulcio and
632+
Rekor are. The result is content-identical to the one-shot path.
633+
634+
`SignCatalog` is the counterpart to `VerifyCatalog`, signing this
635+
Client's catalog and returning the serialized Sigstore bundle.
636+
637+
**`SignCatalog` rejects the signing modes it can tell `VerifyCatalog`
638+
will not verify** — with one documented exception, below. Verification
639+
checks against the public-good Sigstore root, requires a
640+
transparency-log entry, and accepts keyless GitHub OIDC certificates
641+
only, so these four `OIDCResolve` settings are rejected with
642+
`ErrCodeInvalidRequest` before any signing work runs:
643+
644+
| Setting | Why it is rejected |
645+
|---|---|
646+
| `SigningKey` | A key-signed catalog has no verification path at all. |
647+
| `FulcioURL` | A private CA's certificate does not chain to the public-good root. |
648+
| `RekorURL` | A private log's entries do not verify against the public-good root either. A public-good v1 URL would verify, but the two are indistinguishable from the URL alone, so this fails closed. |
649+
| `DisableTLogUpload` | Verification requires a transparency-log entry. |
650+
651+
The point of the guard is that you should not be able to sign a catalog
652+
successfully and then discover the documented counterpart refuses it;
653+
if private catalog signing is ever needed, both halves move together.
654+
655+
**The exception: `SigningConfigPath` is not validated.** It passes
656+
through because the release path requires it, and a Sigstore signing
657+
config can itself name a private Fulcio or Rekor — so a signing config
658+
*can* still produce a catalog `VerifyCatalog` rejects. Treat the guard
659+
as covering the four settings above, not as a guarantee about every
660+
input. Each rejected setting exists *only* to depart from the
661+
public-good defaults, which is what makes rejecting it unambiguous; a
662+
signing config does not, and rejecting it would break the release.
663+
Validating the loaded config against the public-good endpoints is the
664+
principled fix if this exception ever bites.
665+
666+
Neither signing method imposes a facade timeout, unlike their
667+
verification counterparts: keyless OIDC can block on a human completing
668+
a browser or device-code flow. Pass a context with a deadline for
669+
unattended use. Neither prompts — interactive signing disclosure is a UI
670+
concern the caller owns, so both can run unattended from a server.
671+
446672
## Errors
447673

448674
All errors returned by the facade are `*pkg/errors.StructuredError`
@@ -465,12 +691,17 @@ if stderrors.As(err, &se) && se.Code == aicrerrors.ErrCodeInvalidRequest {
465691
## Context handling
466692

467693
`ResolveRecipe` (and every other context-aware facade method) honours
468-
context cancellation. Each facade entry point unconditionally wraps the
469-
caller's context with `context.WithTimeout` against its per-operation
470-
cap. The effective deadline is the smaller of the caller's deadline
471-
and the facade cap, per `context.WithTimeout` semantics — a caller
472-
passing a tighter deadline keeps it; a caller passing
473-
`context.Background()` gets the facade cap.
694+
context cancellation. Capped entry points wrap the caller's context
695+
with `context.WithTimeout` against their per-operation cap; the
696+
effective deadline is then the smaller of the caller's deadline and the
697+
facade cap, per `context.WithTimeout` semantics — a caller passing a
698+
tighter deadline keeps it; a caller passing `context.Background()` gets
699+
the facade cap.
700+
701+
Not every entry point is capped. `PublishEvidence` and `SignCatalog`
702+
never are, and `MakeBundle` is not when `BundleOptions.Timeout` is `0`
703+
(its default). Those run under the caller's context unchanged, so a
704+
caller passing `context.Background()` gets no deadline at all.
474705

475706
Per-operation caps:
476707

@@ -482,6 +713,12 @@ Per-operation caps:
482713
result retrieval sit outside it, so a bare cap would silently shrink the
483714
completion budget you asked for.
484715
- `ValidateState`: `defaults.ValidationOperationTimeout`
716+
- `VerifyBundle` / `VerifyEvidence` / `VerifyCatalog` / `RecipeDigest`:
717+
`defaults.VerifyOperationTimeout`
718+
- `PublishEvidence` / `SignCatalog`: **no facade cap** — the caller's
719+
context governs unchanged. Keyless OIDC can block on a human
720+
completing a browser or device-code flow, so a fixed cap would cut
721+
short a run that legitimately waits.
485722
- `MakeBundle`: opt-in via `BundleOptions.Timeout`. When unset (`0`) the
486723
caller's context governs unchanged — large bundles, `--vendor-charts`,
487724
and attestation/signing can exceed any fixed cap. The REST `/v1/bundle`

docs/integrator/supply-chain-verification.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,13 @@ verification.
99
For a quick trust overview and how to report a vulnerability, see the
1010
top-level [`SECURITY.md`](../../SECURITY.md).
1111

12+
Everything below drives verification from the shell. To do the same
13+
from Go — verifying bundles, evidence, the recipe catalog, and the
14+
`aicr` binary itself — see
15+
[Verifying artifacts](./go-library.md#verifying-artifacts) in the Go
16+
library guide. The two paths run the same verification code, so a CI
17+
gate written either way reaches the same verdict.
18+
1219
## Prerequisites and Setup
1320

1421
Verification uses [Cosign](https://docs.sigstore.dev/cosign/system_config/installation/),

docs/user/api-reference.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -805,7 +805,7 @@ certificate from Fulcio using its own OIDC identity. Operator setup for Mode B:
805805
| `AICR_SIGNING_CONFIG_PATH` | A, B | Sigstore SigningConfig JSON for Rekor v2 targeting. |
806806
| `AICR_TLOG_UPLOAD` | A | Set `false` to skip the Rekor upload for air-gapped KMS signing. KMS-only; keyless always uploads. |
807807
| `AICR_BINARY_ATTESTATION_FILE` | A, B | Absolute path to the aicrd binary attestation. Unset defaults to the conventional `<executable>-attestation.sigstore.json` next to the running binary. Set it when the attestation ships elsewhere in the image, e.g. a ko build stages assets under `KO_DATA_PATH` (`/var/run/ko/aicrd-attestation.sigstore.json`) rather than next to the binary. |
808-
| `AICR_BINARY_ATTESTATION_IDENTITY_REGEXP` | A, B | Certificate-identity pattern the server pins its own binary attestation to. Unset uses the release-workflow default (`on-tag.yaml`). A custom value MUST still contain `NVIDIA/aicr` so it stays pinned to the NVIDIA org; it retargets which NVIDIA workflow attested the binary (e.g. an e2e workflow), not the org, and a value that is not so pinned fails startup. Mirrors the CLI's `--certificate-identity-regexp`. |
808+
| `AICR_BINARY_ATTESTATION_IDENTITY_REGEXP` | A, B | Certificate-identity pattern the server pins its own binary attestation to. Unset uses the release-workflow default (`on-tag.yaml`). A custom value MUST begin with `https://github.com/NVIDIA/aicr/` (leading `^` allowed) and must not use top-level alternation, so it stays confined to the NVIDIA repository; it retargets which NVIDIA workflow attested the binary (e.g. an e2e workflow), not the org, and a value that is not so pinned fails startup. Mirrors the CLI's `--certificate-identity-regexp`. |
809809

810810
Setting both `AICR_SIGNING_KEY` and the keyless variables is ambiguous and the
811811
server refuses to start.

docs/user/cli-config.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -201,7 +201,7 @@ spec:
201201
requireCreator: ci@myorg.example.com
202202
cliVersionConstraint: ">= 0.16.0"
203203
trust: # material verification runs against
204-
certificateIdentityRegexp: "" # must contain NVIDIA/aicr when set
204+
certificateIdentityRegexp: "" # when set, must BEGIN with https://github.com/NVIDIA/aicr/
205205
key: "" # KMS URI or local PEM public-key path
206206
trustRoot: "" # private Sigstore trusted_root.json
207207
```
@@ -303,7 +303,7 @@ checked after verification runs, `trust` holds the material it verifies against.
303303
| `policy.minTrustLevel` | string | `unknown` \| `unverified` \| `attested` \| `verified`, or `max` (the CLI default) to auto-detect the highest level the bundle can reach |
304304
| `policy.requireCreator` | string | Pins the OIDC identity in the bundle attestation's signing certificate |
305305
| `policy.cliVersionConstraint` | string | Constrains the `aicr` version in the attestation predicate; supports `>=`, `>`, `<=`, `<`, `==`, `!=`, and a bare version means `>=` |
306-
| `trust.certificateIdentityRegexp` | string | Certificate identity pattern for binary attestation verification; must contain `NVIDIA/aicr` |
306+
| `trust.certificateIdentityRegexp` | string | Certificate identity pattern for binary attestation verification; must *begin with* `https://github.com/NVIDIA/aicr/` (leading `^` allowed) and must not use top-level alternation, so it stays confined to the repository |
307307
| `trust.key` | string | KMS key URI (`awskms://` \| `gcpkms://` \| `azurekms://` \| `hashivault://`) or local PEM public-key path; the verify counterpart to `spec.bundle.attestation.signingKey` |
308308
| `trust.trustRoot` | string | Path to a private Sigstore `trusted_root.json`, additive to the built-in public-good root |
309309

docs/user/cli-reference.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -782,7 +782,7 @@ aicr recipe verify-catalog <bundle-path> [flags]
782782

783783
| Flag | Type | Default | Description |
784784
|------|------|---------|-------------|
785-
| `--identity-pattern` | string | | Override the NVIDIA CI certificate identity regexp. Must contain `NVIDIA/aicr` — overrides that drop the repo prefix are rejected. Also reads `AICR_CATALOG_IDENTITY_PATTERN`. |
785+
| `--identity-pattern` | string | | Override the NVIDIA CI certificate identity regexp. Must *begin with* `https://github.com/NVIDIA/aicr/` (a leading `^` is allowed; `github\.com` also accepted) and must not use top-level alternation, so the pattern stays confined to the repository. Put any alternatives after the prefix, e.g. `.../aicr/\.github/workflows/(on-tag\|release)\.yaml@.*`. Also reads `AICR_CATALOG_IDENTITY_PATTERN`. |
786786

787787
**Examples:**
788788

0 commit comments

Comments
 (0)