feat(recipes)!: GKE bundle-installer replaces driver-installer

Add the gcp-driver-installer component — Google's cos-gpu-installer
DaemonSet as a values-gated, AICR-managed component — and replace the
GKE gpuStack value driver-installer with bundle-installer, which owns it.

The gpuStack family on GKE COS is now:

- gke-default (default, unchanged): GKE's managed driver install and
  GKE's managed device plugin as the nvidia.com/gpu advertiser.
- bundle-installer: opt-out pools
  (gke-no-default-nvidia-gpu-device-plugin=true, created with
  gpu-driver-version=disabled) — the GPU Operator's device plugin is the
  sole advertiser and the bundle's gcp-driver-installer component
  supplies the driver. The version is pinned in the recipe
  (gcp-driver-installer.driverVersion, default matching the GPU Operator
  chart's driver pin; must be COS-qualified) and upgrades roll with the
  bundle — nothing is applied out-of-band.

The component sits unconditionally in the gke-cos chain, gated on the
nested installer.enabled path (top-level install/enabled are
component-presence gates and would deadlock profile resolution — ADR-015
sketch amended); under gke-default it renders a template-less no-op
chart. The two values are distinguished at generation time by the
opt-out pool label (negated vs positive), which resolves ADR-015
Deferred Decision 5 without any post-deployment signal.

BREAKING: driver-installer (shipped v0.19.0) is removed; selecting it
fails closed with the valid-values list. Migration for pools that used a
hand-applied standalone nvidia-driver-installer DaemonSet: delete that
DaemonSet (the bundle's DaemonSet shares its name in kube-system and
Helm will not adopt the pre-existing object), regenerate with
--profile gpuStack=bundle-installer, and deploy the bundle. Nodes with a
loaded driver are untouched (installer fast path).

Also: CheckNVSentinelDriverLabelDetectable accepts bundle-installer (the
bundle-carried installer IS a driver pod the labeler detects); the
deployment validator's static-assert suppression is generalized
(gatedHealthCheckSuppressed) so a values-gated component whose render is
empty skips its DaemonSet health check; cos-nvidia-installer:fixed gets
a digest-pin exemption (COS-node-local image, never pulled or mirrored);
parity goldens, BOM, and recipe-health docs regenerated.

Fixes #1716
Related #2344

Signed-off-by: Atif Mahmood <atif1996@users.noreply.github.com>

Author: atif1996

docs/design/015-recipe-configuration-profiles.md

@@ -213,10 +213,12 @@ target state, gated by this rule.
 
 ```yaml
 # recipes/overlays/gke-cos.yaml — device-plugin ownership
-# Shown with the post-DD5 value set. DD5's durable marker, not yet
-# identified, will additionally land as symmetric constraints on
-# operator and operator-selfdriver at that event — the constraints
-# drawn here are therefore not yet the declarable post-DD5 state.
+# Shown with the value set as originally drawn. Amended 2026-08-24: the
+# family ships as gke-default plus bundle-installer — the drawn
+# operator-selfdriver, which REPLACED the operator value (shipped
+# v0.19.0 as driver-installer). The two shipped values differ in the
+# opt-out pool label at generation time, so Deferred Decision 5 needed
+# no durable marker.
 spec:
   profile:
     name: gpuStack
@@ -277,7 +279,14 @@ spec:
         componentRefs:
           - name: gcp-driver-installer
             overrides:
-              install: true       # the chart-level gate
+              # Amended 2026-08-24: shipped as the nested installer.enabled,
+              # not the top-level `install` drawn here — top-level
+              # `install`/`enabled` are component-PRESENCE gates (IsEnabled),
+              # so a false default would make the component "not enabled in
+              # the surviving composition" and deadlock resolution for every
+              # value; root `overrides.enabled` is separately rejected in
+              # fragments. A nested key is an ordinary owned value path.
+              installer: {enabled: true}
           - name: gpu-operator
             overrides:
               devicePlugin: {enabled: true}
@@ -297,9 +306,10 @@ step 2 describes. Selection:
 # gke-default (declared default; drawn above as csp-managed) — no flag needed
 aicr recipe --service gke --os cos --accelerator h100 --intent inference
 
-# explicit alternative configuration (shipped name; drawn above as operator)
+# explicit alternative configuration (shipped name; drawn above as
+# operator-selfdriver)
 aicr recipe --service gke --os cos --accelerator h100 --intent inference \
-  --profile gpuStack=driver-installer
+  --profile gpuStack=bundle-installer
 ```
 
 A profile fragment may reference only components **enabled in the
@@ -324,7 +334,8 @@ explicit toggles from changing the pre-existing presence state, with
 subset-filter semantics defined under Override locking.
 
 **Conditional installation is expressible in v1 through a values-gated
-component** — the `operator-selfdriver` value above is the pattern.
+component** — the `operator-selfdriver` value above (shipped as
+`bundle-installer`) is the pattern.
 The component sits unconditionally in the composition and renders
 nothing unless its gate value is selected; the profile value flips a
 plain values path — owned, locked, and validated like any other —
@@ -1457,13 +1468,27 @@ recurrence — the shape the Problem section expects.
    installer's synthetic `enabled` joins `ownedPaths`), which is a
    family-wide re-qualification and evidence re-signing event.
 
+   *Amended 2026-08-24 (issue #1716).* The landing event shipped the value
+   as `bundle-installer`, **replacing** `driver-installer` rather than
+   joining it: the two named the same pool shape
+   (`gpu-driver-version=disabled` + the opt-out label), differing only in
+   who applied the installer DaemonSet — precisely Deferred Decision 5's
+   ambiguity, which the replacement resolves without any signal. The
+   shipped family is `gke-default` / `bundle-installer`, distinguished at
+   generation time by the pool label alone. Selecting
+   `gpuStack=driver-installer` fails closed with the valid-values list;
+   migration: delete any hand-applied `nvidia-driver-installer` DaemonSet
+   (the bundle's DaemonSet shares its name) and regenerate with
+   `--profile gpuStack=bundle-installer`.
+
    Any dcgm-exporter GPU-ID-mapping adjustment for `csp-managed` is an
    external GKE behavior not verifiable from this repository. It is
    verified and added during this step if required, with upstream
    citations recorded in that PR. Before conversion, this step also
    checks the family's `/v1` usage or announces a deprecation window so
    clients do not discover the `/v1` rejection at cut-over.
-4. Other consumers: once `operator-selfdriver` is declared, internal
+4. Other consumers: once `operator-selfdriver` (shipped as
+   `bundle-installer`) is declared, internal
    recipes (DGXC/NKX) migrate the cos-gpu-installer arrangement
    (internal MR #27) to the public value. The values-gated
    `gcp-driver-installer` component makes the case expressible without
@@ -1495,8 +1520,8 @@ work that resolves it.
    GKE's managed driver install, so the standalone gate's prerequisite
    needed the profile's per-value pairing), and the GKE `gpuStack`
    profile now consumes the form per selected value (#1761 rollout
-   PR 3): positive for `driver-installer`, negated for the
-   `gke-default` default.
+   PR 3): positive for `driver-installer` (later `bundle-installer`),
+   negated for the `gke-default` default.
 3. **AKS node-pool-mode signal — resolved by the 2026-07-27 amendment.**
    The provider-facing AgentPool `gpuProfile.driver` property is the
    durable ownership marker. AKS adoption projects it into a snapshot
@@ -1522,3 +1547,9 @@ work that resolves it.
    absence, so the two values stay mutually distinguishable.
    **Proposed: identify a durable signal during the value's adoption;
    the `operator` and `csp-managed` values do not wait on it.**
+
+   *Resolved 2026-08-24 (issue #1716): no signal was needed.* The value
+   shipped as `bundle-installer`, **replacing** `operator`
+   (shipped `driver-installer`) — the two named the same pool shape, and
+   with the sibling gone the family's values are mutually distinguishable
+   at generation time by the opt-out pool label alone.

docs/integrator/automation.md

@@ -366,7 +366,7 @@ are not time-bounded and persist until `Client.Close()` is called.
 > family (profiled and unprofiled) with the same criteria parameters, plus
 > optional `profile=gpuStack=azure-managed` or
 > `profile=gpuStack=operator-managed` on AKS, and
-> `profile=gpuStack=gke-default` or `profile=gpuStack=driver-installer`
+> `profile=gpuStack=gke-default` or `profile=gpuStack=bundle-installer`
 > on GKE
 > (see [GKE GPU Setup](gke-gpu-setup.md#gpu-device-plugin-ownership)).
 > `/v1/recipe` still works for unprofiled compositions but rejects any

docs/integrator/data-extension.md

@@ -231,7 +231,7 @@ When in doubt, `aicr --debug recipe ... --data <dir>` logs the resolved source
 `recipes/overlays/aks.yaml` declares the `gpuStack` configuration profile
 (`azure-managed` default, `operator-managed` alternative) over the GPU driver/toolkit
 ownership paths, and `recipes/overlays/gke-cos.yaml` declares its own `gpuStack`
-(`gke-default` default, `driver-installer` alternative) over device-plugin ownership —
+(`gke-default` default, `bundle-installer` alternative) over device-plugin ownership —
 the GKE default value (`gke-default`) additionally declares
 `advertiser: external`, and both GKE values trigger the #1327
 allocation-policy closure, so their effective lock set is larger than the

docs/integrator/gke-gpu-setup.md

@@ -38,7 +38,7 @@ synthesizes the GPU-node universe from the snapshot's `NodeTopology.label`
 readings (nodes carrying `cloud.google.com/gke-accelerator`) and quantifies a
 label predicate over it, in both directions — the positive form
 `gke-no-default-nvidia-gpu-device-plugin=true` (every GPU node carries the
-label) qualifies `driver-installer`, and the negated form
+label) qualifies `bundle-installer`, and the negated form
 `!gke-no-default-nvidia-gpu-device-plugin` (no GPU node carries the key)
 qualifies `gke-default`.
 
@@ -54,8 +54,8 @@ aicr snapshot -o snapshot.yaml
 aicr recipe --service gke --accelerator h100 --os cos --intent training \
   --platform kubeflow \
   --snapshot snapshot.yaml -o recipe.yaml                 # gke-default default
-#   ... or, for labeled pools with the standalone driver installer:
-#   --profile gpuStack=driver-installer
+#   ... or, for labeled pools (bundle-carried driver installer):
+#   --profile gpuStack=bundle-installer
 
 # 3. Bundle. NVSentinel needs no overrides: the gpuStack profile sets its
 #    labeler flag per value. See the NVSentinel note below.
@@ -65,7 +65,7 @@ aicr bundle -r recipe.yaml -o ./bundles
 The reading qualifies the selection — it does not choose for you. Every
 combination is deterministic:
 
-| GPU-node labels read | Default (`gke-default`) | `--profile gpuStack=driver-installer` |
+| GPU-node labels read | Default (`gke-default`) | `--profile gpuStack=bundle-installer` |
 |---|---|---|
 | all GPU nodes label-absent | ✅ resolves | ❌ fails closed: constraint expects the label on every GPU node |
 | all GPU nodes `gke-no-default-nvidia-gpu-device-plugin=true` | ❌ fails closed: constraint expects no labeled GPU node | ✅ resolves |
@@ -85,9 +85,9 @@ the snapshot, and the check is NEVER skipped when a snapshot is present:
 | Invocation | Selected value | Node-label check |
 |---|---|---|
 | no `--profile`, no `--snapshot` | declaration default (`gke-default`) | none possible (no cluster data) — the constraint is still recorded in the recipe and enforced at `aicr validate` readiness |
-| `--profile gpuStack=driver-installer`, no `--snapshot` | `driver-installer` | same — deferred to validate |
+| `--profile gpuStack=bundle-installer`, no `--snapshot` | `bundle-installer` | same — deferred to validate |
 | no `--profile`, `--snapshot` | default (`gke-default`) | checked at generation: no GPU node may carry the opt-out label, else generation fails closed naming the observed state |
-| `--profile gpuStack=driver-installer`, `--snapshot` | `driver-installer` | checked at generation: every GPU node must carry `gke-no-default-nvidia-gpu-device-plugin=true`, else fails closed |
+| `--profile gpuStack=bundle-installer`, `--snapshot` | `bundle-installer` | checked at generation: every GPU node must carry `gke-no-default-nvidia-gpu-device-plugin=true`, else fails closed |
 
 If you need an unverified recipe deliberately, generate criteria-only (drop
 `--snapshot`): the artifact is honest about being unqualified, and the
@@ -165,7 +165,7 @@ override is needed:
 
 Because the path is profile-owned, a bundle-time `--set` diverging from the
 selected value is **rejected** rather than silently applied. The explicit
-`false` under `driver-installer` is deliberate: skipping detection there would
+`false` under `bundle-installer` is deliberate: skipping detection there would
 keep the label applied across an unloaded driver.
 
 The value renders the labeler's `--assume-driver-installed` argument — the
@@ -176,7 +176,7 @@ recommended, permanent mechanism for host-installed drivers (no automatic
 detection fallback will be added). Under `gke-default` a recipe that reaches
 bundle generation without it is a **blocking error**
 (`CheckNVSentinelDriverLabelDetectable`), so the silent half-rollout cannot
-ship. Under `driver-installer` the gate does not fire: the standalone installer
+ship. Under `bundle-installer` the gate does not fire: the bundle's installer
 supplies an observable driver pod.
 
 **Labeling the nodes by hand does not persist.** Applying the label manually:
@@ -191,28 +191,34 @@ its next reconcile. Design 018 documents manual labeling as the procedure for
 this case, so an operator following it will see it work and later find the
 DaemonSets back at 0 desired.
 
-### Alternative: Let GPU Operator Manage the Device Plugin
+### Alternative: Let the Bundle Own the GPU Stack
 
 If you prefer the GPU Operator's device plugin to own `nvidia.com/gpu`
 advertisement, select the mode at recipe generation:
 
 ```shell
 aicr recipe --service gke --accelerator h100 --os cos --intent training \
   --platform kubeflow \
-  --profile gpuStack=driver-installer -o recipe.yaml
+  --profile gpuStack=bundle-installer -o recipe.yaml
 aicr bundle -r recipe.yaml -o ./bundles
 ```
 
-This value has real cluster prerequisites. The opt-out label forfeits GKE's
+This value has real pool prerequisites. The opt-out label forfeits GKE's
 managed driver install: the managed install (`gpu-driver-version=default` or
 `latest`) is finalized by an init container of the **same** kube-system
 DaemonSet the label disables, so a labeled pool paired with the managed
 install comes up **driverless** — never combine the label with
-`gpu-driver-version=default`/`latest`. Pools for the `driver-installer` value
-must instead be created with `gpu-driver-version=disabled`, with driver
-provisioning supplied by Google's standalone
-[`nvidia-driver-installer` DaemonSet](https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#installing_drivers)
-applied to the cluster.
+`gpu-driver-version=default`/`latest`. Pools for the `bundle-installer` value
+must instead be created with `gpu-driver-version=disabled`. Driver
+provisioning is carried **inside the bundle**: the `gcp-driver-installer`
+component ships Google's cos-gpu-installer DaemonSet
+([#1716](https://github.com/NVIDIA/aicr/issues/1716)), ordered ahead of the
+GPU Operator, with the driver version pinned in the recipe
+(`gcp-driver-installer.driverVersion`, default matching the GPU Operator
+chart's driver pin). The pin must be COS-qualified: the installer validates
+the request against the COS build's curated per-GPU-type list and rejects
+unqualified versions. Version bumps take effect on replaced or rebooted
+nodes only (the installer skips nodes with a loaded nvidia module).
 
 Set the label when you create the GPU node pool, alongside the disabled
 managed install:
@@ -283,7 +289,7 @@ and GPU pods will not schedule. That brief advertiser-free window is the
 accepted cost of the handoff direction — do **not** invert it by deploying
 the Operator's plugin onto a still-unlabeled pool, which would put two
 advertisers on the same nodes (the dual-advertisement state the
-[allocation-policy gates](#the-three-driver-installer-settings) exist to
+[allocation-policy gates](#the-three-bundle-installer-settings) exist to
 prevent). Have the bundle from step 5 generated in advance to keep the
 window short, and avoid scheduling GPU work during it.
 Note that `--node-labels` on update **replaces** the pool's full user-label
@@ -310,7 +316,7 @@ which the update rejects. Omitting an existing label removes it from the
 pool's nodes, which can break scheduling that depends on it.
 
 **Step 5 — deploy the GPU Operator and wait for its plugin.** Deploy the
-AICR bundle generated with `--profile gpuStack=driver-installer`, then wait
+AICR bundle generated with `--profile gpuStack=bundle-installer`, then wait
 until the Operator's device-plugin pods are Running on the labeled nodes and
 every GPU node again reports non-zero allocatable `nvidia.com/gpu` — that
 closes the advertiser-free window opened in step 4. Confirm the full result
@@ -351,7 +357,7 @@ advertiser — if the GPU Operator is not yet deployed (or its plugin is not
 Ready), labeling leaves the node with **no** `nvidia.com/gpu` advertiser at
 all, and GPU pods will not schedule until the Operator's plugin comes up.
 
-#### The three driver-installer settings
+#### The three bundle-installer settings
 
 The three settings cover different parts of the GPU stack:
 
@@ -360,14 +366,26 @@ The three settings cover different parts of the GPU stack:
   effect, forfeits GKE's managed driver install (the installer rides the
   DaemonSet the label disables).
 - `gpu-driver-version=disabled` records that GKE does not own driver
-  provisioning on the pool — and is what makes the standalone installer act
-  on its nodes (the installer ignores automatic-install pools). Never pair
-  the label with `gpu-driver-version=default` — labeled pools come up
-  driverless.
-- The standalone `nvidia-driver-installer` DaemonSet supplies the driver.
+  provisioning on the pool — and is what makes the bundle's installer act
+  on its nodes (it ignores automatic-install pools). Never pair the label
+  with `gpu-driver-version=default` — labeled pools come up driverless.
+- The bundle's `gcp-driver-installer` DaemonSet supplies the driver.
   AICR's GKE-COS overlays keep `driver.enabled: false` because the GPU
   Operator cannot install a driver on COS node images.
 
+#### Migrating from a hand-applied installer DaemonSet
+
+Earlier arrangements (including AICR's replaced `driver-installer` profile
+value, shipped in v0.19.0) supplied the driver by applying Google's standalone
+[`nvidia-driver-installer` DaemonSet](https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#installing_drivers)
+by hand on the same pool shape. Do not run that alongside a
+`bundle-installer` bundle: the bundle's DaemonSet shares the name
+`nvidia-driver-installer` in `kube-system`, and Helm will not adopt the
+pre-existing object. To migrate: delete the hand-applied DaemonSet,
+regenerate with `--profile gpuStack=bundle-installer`, and deploy the
+bundle. Nodes with a loaded driver are untouched (the installer's fast path
+skips them); the bundle takes over provisioning for new and rebooted nodes.
+
 ## Troubleshooting
 
 ### Labeled pool comes up driverless
@@ -394,11 +412,10 @@ the pairing "label + managed driver install" is never functional.
   [Retrofitting an existing pool](#retrofitting-an-existing-pool)) so GKE's
   DaemonSet returns and finalizes the managed install, and generate (or keep)
   recipes with the default selection.
-- **Commit to `driver-installer`:** apply Google's standalone
-  `nvidia-driver-installer` DaemonSet and recreate the pools with
+- **Commit to `bundle-installer`:** recreate the pools with
   `gpu-driver-version=disabled` (or update their driver mode in place — see
   [Retrofitting an existing pool](#retrofitting-an-existing-pool)), then
-  generate recipes with `--profile gpuStack=driver-installer`.
+  generate recipes with `--profile gpuStack=bundle-installer`.
 
 ### No advertiser at all
 

docs/integrator/recipe-development.md

@@ -548,7 +548,7 @@ directions fail closed on a truncated node list (a snapshot captured with
 reading), on an empty GPU-node universe, and on malformed or ambiguous
 label readings (an encoding collision between a disambiguated entry and a
 distinct dotted label name — see #2003). It is consumed by the GKE
-`gpuStack` profile values (the positive form qualifies `driver-installer`, the
+`gpuStack` profile values (the positive form qualifies `bundle-installer`, the
 negated form `gke-default`), where each selected value's constraint is
 verified at generation when generating from a snapshot (criteria-only
 generation has no snapshot evaluator and defers entirely to the

docs/user/api-reference.md

@@ -514,7 +514,7 @@ converted workflow to v2 as one cut-over.
 reject. Move GET clients to `GET /v2/recipe` / `GET /v2/query` (identical
 query parameters, plus optional `profile=gpuStack=azure-managed` or
 `profile=gpuStack=operator-managed` on AKS, and `profile=gpuStack=gke-default`
-or `profile=gpuStack=driver-installer` on GKE); move POST
+or `profile=gpuStack=bundle-installer` on GKE); move POST
 clients to `POST /v2/recipe` / `POST /v2/query`, converting the body to the
 strict envelope described above (a plain `criteria` object with an explicit
 `Content-Type`, not the v1 `RecipeCriteria` resource). Then POST the
@@ -523,12 +523,12 @@ unaffected on `/v1` until they adopt a profile.
 
 ```shell
 # GKE migration: /v2/recipe (omit profile= for the gke-default default,
-# or select gpuStack=driver-installer explicitly), then POST to /v2/bundle.
+# or select gpuStack=bundle-installer explicitly), then POST to /v2/bundle.
 # -f stops on an HTTP error so a 4xx/5xx recipe body is never staged and
 # an error response is never written to bundles.zip.
 set -euo pipefail
 curl -fsS -o recipe.json \
-  "http://localhost:8080/v2/recipe?service=gke&accelerator=h100&os=cos&intent=training&profile=gpuStack=driver-installer"
+  "http://localhost:8080/v2/recipe?service=gke&accelerator=h100&os=cos&intent=training&profile=gpuStack=bundle-installer"
 curl -fsS -X POST "http://localhost:8080/v2/bundle" \
   -H "Content-Type: application/json" -d @recipe.json -o bundles.zip
 ```