feat(recipes): adopt k8s-aibom in the h100-gke-cos-inference recipe (#2328)

Signed-off-by: Mark Chmarny <mark@chmarny.com>

Author: mchmarny

docs/design/019-k8s-aibom-runtime-inventory.md

@@ -620,11 +620,42 @@ the workflow without asserting stock stability.
 
 ### C. Recipe in scope
 
-`h100-gke-cos-inference`, a single leaf overlay. Not a recipe family, not
-`gke-cos-inference`, not `gke-cos`, not a mixin, and not a shared base.
-Inference is chosen because an AI BOM inventories models and AI workloads, so
-a served model exercises the component's actual purpose. Substituting a
-different recipe requires amending this section.
+`h100-gke-cos-inference`. Not a recipe family, not `gke-cos-inference`, not
+`gke-cos`, not a mixin, and not a shared base. Inference is chosen because an
+AI BOM inventories models and AI workloads, so a served model exercises the
+component's actual purpose. Substituting a different recipe requires amending
+this section.
+
+**Correction, 2026-08-21.** This section originally called
+`h100-gke-cos-inference` "a single leaf overlay." It is not a leaf:
+`h100-gke-cos-inference-dynamo` declares it as its `base`, so a componentRef
+added here is inherited by that recipe and the component would ship in two
+stock recipes rather than one. The scope decision is unchanged — one recipe —
+so `h100-gke-cos-inference-dynamo` declines the inherited component with
+`overrides.install: false`, the same gate `--runtime-inventory disabled` sets.
+The declined ref is retained rather than omitted so the resolved recipe records
+the decision.
+
+Two consequences worth stating, because neither is obvious from the overlay
+files:
+
+- Declining it there is a scope decision, not a judgment about Dynamo. AICR has
+  qualified `k8s-aibom` on its own, not alongside `grove` and
+  `dynamo-platform`. Widening adoption to that recipe is a later decision that
+  needs its own evidence.
+- The stock render golden covers *leaves only*, so it pins
+  `h100-gke-cos-inference-dynamo` and not `h100-gke-cos-inference`. The target
+  recipe of this amendment therefore has no rendered-bytes parity coverage. The
+  golden that does move is the resolution golden, recording the declined ref on
+  the descendant, while the render golden stays byte-identical.
+
+  That unchanged render golden is **supporting evidence, not the proof**: it
+  establishes only that the descendant's rendered bytes did not change. It
+  cannot establish the one-recipe blast radius, precisely because the target
+  recipe has no render golden of its own. The scope assertion lives in
+  `TestK8sAIBOMStockAdoption` (`pkg/recipe`), which pins that the target
+  enables the component, the descendant declines it, the generation-time
+  opt-out declines it, and a sibling recipe does not declare it at all.
 
 ### D. User-demand case
 

docs/user/component-catalog.md

@@ -28,7 +28,7 @@ The source of truth is [`recipes/registry.yaml`](https://github.com/NVIDIA/aicr/
 | **prometheus-adapter** | Exposes custom metrics from Prometheus to the Kubernetes metrics API. Enables HPA scaling based on GPU utilization and other custom metrics. | [prometheus-adapter](https://github.com/kubernetes-sigs/prometheus-adapter) |
 | **aws-ebs-csi-driver** | CSI driver for Amazon EBS volumes. Provides persistent storage for workloads on EKS. EKS-specific. **Cluster-wide default StorageClass:** AICR enables `defaultStorageClass.enabled`, so this component provisions a **cluster-default** gp3 StorageClass (`ebs-csi-default-sc`) on **every** EKS cluster that includes it — not just inference recipes; training overlays inherit it too. EKS ships no default SC of its own, so this makes dynamic provisioning (e.g. the inference-perf model cache) work zero-config. Two consequences to note: (1) if the cluster already has a default SC, Kubernetes treats multiple defaults as ambiguous — unset the other; (2) a PVC that previously failed-fast on "no default SC" will now silently bind gp3, which can mask a misconfiguration. | [AWS EBS CSI Driver](https://github.com/kubernetes-sigs/aws-ebs-csi-driver) |
 | **k8s-ephemeral-storage-metrics** | Exports ephemeral storage usage metrics per pod. Useful for monitoring scratch space consumption on GPU nodes. | [k8s-ephemeral-storage-metrics](https://github.com/jmcgrath207/k8s-ephemeral-storage-metrics) |
-| **k8s-aibom** | Optional runtime AI workload inventory. Produces namespace-scoped CycloneDX 1.6 ML-BOM resources for explicitly opted-in namespaces. Registry-only: no stock recipe installs it. CLI aliases: `k8saibom`, `aibom`. See [k8s-aibom Runtime Inventory](#k8s-aibom-runtime-inventory). | [k8s-aibom](https://github.com/GoogleCloudPlatform/k8s-aibom) |
+| **k8s-aibom** | Optional runtime AI workload inventory. Produces namespace-scoped CycloneDX 1.6 ML-BOM resources for explicitly opted-in namespaces. Installed by one stock recipe, `h100-gke-cos-inference`; every other stock recipe leaves it out. Decline it with `aicr recipe --runtime-inventory disabled`. CLI aliases: `k8saibom`, `aibom`. See [k8s-aibom Runtime Inventory](#k8s-aibom-runtime-inventory). | [k8s-aibom](https://github.com/GoogleCloudPlatform/k8s-aibom) |
 | **kai-scheduler** | Gang scheduler with hierarchical queues and topology-aware placement; works with device-plugin (`nvidia.com/gpu`) and DRA GPU allocation alike. Ensures distributed training jobs land on nodes with optimal interconnect topology. | [KAI Scheduler](https://github.com/kai-scheduler/KAI-Scheduler) |
 | **grove** | Pod lifecycle management for Dynamo inference platform. Installed as a standalone component. | [Grove](https://github.com/ai-dynamo/grove) |
 | **dynamo-platform** | NVIDIA Dynamo inference serving platform with bundled CRDs. Distributed inference with KV-cache-aware routing, Dynamo request-plane traffic, a NATS-backed Kubernetes event plane for KV-cache events, and disaggregated prefill/decode. | [Dynamo](https://github.com/ai-dynamo/dynamo) |
@@ -291,9 +291,17 @@ AICR enforces and surfaces inference-gateway exposure in two places:
 ## k8s-aibom Runtime Inventory
 
 AICR qualifies k8s-aibom v1.3.0 as an optional Helm component. It is not in
-the base, a mixin, or any stock overlay. To enable it, add this reference to a
-custom or external overlay and keep that overlay's criteria as narrow as the
-intended rollout:
+the base or a mixin. Exactly one stock recipe installs it,
+`h100-gke-cos-inference`, under [ADR-019](https://github.com/NVIDIA/aicr/blob/main/docs/design/019-k8s-aibom-runtime-inventory.md)'s
+stock-adoption amendment. Decline it at generation time with
+`aicr recipe --runtime-inventory disabled`, described below.
+
+`h100-gke-cos-inference-dynamo` inherits from that recipe and deliberately
+declines the component, so the Dynamo platform recipe deploys exactly what it
+did before. Adoption beyond the one recipe is a later decision.
+
+To enable it anywhere else, add this reference to a custom or external overlay
+and keep that overlay's criteria as narrow as the intended rollout:
 
 ```yaml
 spec:
@@ -366,21 +374,28 @@ storage. So the check catches a stranded upgrade that crosses a
 storage-version boundary, such as the 1.2.0 to 1.3.0 move this pin made, and
 does not catch one within a boundary, such as 1.0.0 to 1.2.0.
 
-**Declining the component.** No stock recipe declares `k8s-aibom` today, so the
-flag applies to a recipe that adds it through a custom overlay — the shape shown
-above. Point `--data` at the directory holding that overlay and generate with
-`--runtime-inventory disabled`:
+**Declining the component.** `h100-gke-cos-inference` installs `k8s-aibom` by
+default. Decline it at generation time:
+
+```bash
+aicr recipe --service gke --accelerator h100 --os cos --intent inference \
+  --runtime-inventory disabled -o recipe.yaml
+```
+
+The same flag works for a recipe that adds the component through a custom
+overlay — the shape shown above. Point `--data` at the directory holding that
+overlay:
 
 ```bash
 aicr recipe --service gke --accelerator h100 --os cos --intent inference \
   --data ./my-recipes --runtime-inventory disabled -o recipe.yaml
 ```
 
 Passing the flag against a recipe that does not declare the component is an
-error, not a silent no-op:
+error, not a silent no-op. Training recipes do not, so:
 
 ```console
-$ aicr recipe --service gke --accelerator h100 --os cos --intent inference \
+$ aicr recipe --service gke --accelerator h100 --os cos --intent training \
     --runtime-inventory disabled
 [INVALID_REQUEST] runtime inventory mode "disabled" requires the recipe to
 declare component "k8s-aibom"; this recipe does not resolve it
@@ -519,12 +534,25 @@ underneath it, so it reconciles against resources that are disappearing.
 
 1. Remove the component reference from the custom overlay and regenerate the
    recipe and bundle.
-2. Uninstall the release with the deployer-appropriate procedure in
-   [Bundle Uninstall](cli-reference.md#bundle-uninstall) — `helm uninstall
-   k8s-aibom -n k8s-aibom-system` for the `helm` deployer, `helmfile destroy`
-   for Helmfile bundles, deleting the owning `Application` for Argo CD, and the
-   `HelmRelease` for Flux. Confirm the controller Deployment is gone before
-   continuing.
+2. Uninstall the release, scoped to this component only:
+
+   ```bash
+   # helm and helmfile bundles alike: helmfile installs through Helm, so the
+   # release is an ordinary Helm release and this removes exactly one.
+   helm uninstall k8s-aibom -n k8s-aibom-system
+   ```
+
+   For Argo CD, delete the owning `Application`; for Flux, the `HelmRelease`.
+   Confirm the controller Deployment is gone before continuing.
+
+   **Do not use `helmfile destroy` for this.** It tears down *every* release in
+   the bundle in reverse dependency order, not just this component. It is also
+   ineffective here: step 1 regenerated the bundle without `k8s-aibom`, so the
+   release is no longer declared in it and `destroy` would not remove the one
+   release you actually want gone while removing all the ones you do not. If
+   you prefer a Helmfile-native command, run it against a bundle that still
+   declares the component and scope it explicitly with
+   `helmfile destroy --selector name=k8s-aibom`.
 3. Only then delete retained AIBOMs and, last, the CRDs.
 
 Deleting the CRDs cascades to every AIBOM stored cluster-wide, including any

pkg/bundler/k8s_aibom_render_parity_test.go

@@ -35,12 +35,23 @@ import (
 	"github.com/NVIDIA/aicr/pkg/recipe"
 )
 
-// ADR-019 admits k8s-aibom registry-only: no stock recipe declares it, so no
-// stock-recipe render path exercises it and the KWOK deployer lanes never see
-// it. These tests are the substitute — they build the same single-component
-// recipe an adopter would, from the live registry entry rather than from
-// hardcoded coordinates, so a registry edit flows into the assertions instead
-// of silently diverging from them.
+// One stock recipe declares k8s-aibom, h100-gke-cos-inference, under ADR-019's
+// stock-adoption amendment. That recipe is not a leaf — h100-gke-cos-inference-dynamo
+// bases on it and declines the component — and the stock render golden covers
+// leaves only, so no golden pins this component's rendered bytes and no KWOK
+// deployer lane exercises it per deployer.
+//
+// These tests cover the per-deployer render: they build the same
+// single-component recipe an adopter would, from the live registry entry rather
+// than from hardcoded coordinates, so a registry edit flows into the assertions
+// instead of silently diverging from them.
+//
+// They deliberately do not assert the stock-adoption contract — that the target
+// recipe enables the component, the Dynamo descendant declines it, and the
+// generation-time opt-out works. A synthetic single-component fixture cannot
+// see any of that, so flipping the target ref to `install: false` would leave
+// every assertion here green. That contract is pinned separately by
+// TestK8sAIBOMStockAdoption in pkg/recipe.
 const (
 	k8sAIBOMComponentName = "k8s-aibom"
 	k8sAIBOMValuesFile    = "components/k8s-aibom/values.yaml"

pkg/cli/query_test.go

@@ -342,14 +342,21 @@ func TestRecipeAndQueryCommandsRejectInvalidRuntimeInventoryMode(t *testing.T) {
 // TestRecipeCommandRejectsRuntimeInventoryWithoutComponent covers the flag-set
 // branch reaching a successful parse and then failing closed at build time,
 // which is the path an operator hits after a typo in --service.
+//
+// The criteria must name a recipe that does not declare k8s-aibom. Inference on
+// gke/h100/cos is the stock-adoption target under ADR-019's amendment and now
+// declares it, so this asserts against a training recipe instead. Mirrors the
+// same precondition in pkg/client/v1's TestResolveRecipeRuntimeInventoryMode.
 func TestRecipeCommandRejectsRuntimeInventoryWithoutComponent(t *testing.T) {
 	err := recipeCmd().Run(t.Context(), []string{
 		"recipe", "--service", "gke", "--accelerator", "h100",
-		"--os", "cos", "--intent", "inference",
+		"--os", "cos", "--intent", "training",
 		"--runtime-inventory", "disabled",
 	})
 	if err == nil {
-		t.Fatal("command error = nil, want rejection for a recipe that does not declare the component")
+		t.Fatal("command error = nil, want rejection for a recipe that does not declare " +
+			"the component; if these criteria now declare k8s-aibom, pick criteria that " +
+			"do not rather than relaxing this assertion")
 	}
 	if !stderrors.Is(err, errors.New(errors.ErrCodeInvalidRequest, "")) {
 		t.Errorf("command error = %v, want ErrCodeInvalidRequest", err)

pkg/client/v1/aicr_test.go

@@ -2573,8 +2573,10 @@ func TestResolveRecipeRuntimeInventoryMode(t *testing.T) {
 		{name: "invalid mode is rejected", mode: "off", wantErr: true},
 		{name: "empty mode is rejected", mode: "", wantErr: true},
 		{
-			// Valid value, but no stock recipe declares the component, so the
-			// build must refuse rather than record a mode it cannot apply.
+			// Valid value, but these criteria do not declare the component,
+			// so the build must refuse rather than record a mode it cannot
+			// apply. The baseline assertion above fails loudly if a future
+			// overlay change makes these criteria declare it.
 			name: "valid mode on a recipe without the component is rejected",
 			mode: "disabled", wantErr: true,
 		},