Skip to content

Commit e64f193

Browse files
committed
feat(recipes)!: GKE bundle-installer replaces driver-installer
Remove the driver-installer gpuStack value (shipped in v0.19.0) and ship the previously deferred third value as bundle-installer. The two removed modes were the same cluster shape — pools created gpu-driver-version=disabled with the opt-out label — differing only in who applied the installer DaemonSet (by hand vs the bundle's gcp-driver-installer component). That sameness was exactly ADR-015 Deferred Decision 5's ambiguity; withdrawing the hand-applied mode resolves DD5 by construction: the two remaining values (gke-default, bundle-installer) are distinguished at generation time by the opt-out pool label alone (negated vs positive). Consequences: - No post-deployment ownership marker: the NFD feature-file stamping is removed from the installer DaemonSet, and both values' readinessConstraints are deleted — the GKE family no longer uses the mechanism (the OKE family remains its consumer; ADR DD5 amendment rewritten accordingly). - BREAKING: selecting gpuStack=driver-installer now fails closed with the valid-values list. Migration for clusters on the hand-applied arrangement: delete the standalone nvidia-driver-installer DaemonSet (the bundle's DaemonSet shares its name in kube-system and Helm will not adopt the pre-existing object), regenerate with --profile gpuStack=bundle-installer, and deploy the bundle. Nodes with a loaded driver are untouched (installer fast path). - CheckNVSentinelDriverLabelDetectable accepts the single value; docs (gke-gpu-setup, component-catalog, cli/api reference, ADR-015 sketch and adoption amendments) and tests swept; parity goldens regenerated. Related #1716 #2344 Signed-off-by: Atif Mahmood <atif1996@users.noreply.github.com>
1 parent 27fe4fb commit e64f193

24 files changed

Lines changed: 231 additions & 376 deletions

docs/contributor/recipe.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -252,7 +252,8 @@ Resolution enforces these invariants:
252252
5. Evaluate selected profile constraints fail closed. A missing reading has a
253253
distinct invalid-request diagnostic; other evaluator failures propagate.
254254
A value's `readinessConstraints` are exempt from this step by design:
255-
they name post-deployment properties (ADR-015 DD5) and route into
255+
they name deployment-created properties (e.g. deployed ClusterPolicy
256+
state — the OKE gpuStack family's distinguishers) and route into
256257
`spec.validation.readiness.constraints`, where the `aicr validate`
257258
readiness pre-flight evaluates them fail closed. Names deduplicate
258259
per phase — the same measurement path may carry a generation-time

docs/design/015-recipe-configuration-profiles.md

Lines changed: 74 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -213,10 +213,13 @@ target state, gated by this rule.
213213

214214
```yaml
215215
# recipes/overlays/gke-cos.yaml — device-plugin ownership
216-
# Shown with the post-DD5 value set. DD5's durable marker, not yet
217-
# identified, will additionally land as symmetric constraints on
218-
# operator and operator-selfdriver at that event — the constraints
219-
# drawn here are therefore not yet the declarable post-DD5 state.
216+
# Shown with the post-DD5 value set as originally drawn. Amended
217+
# 2026-08-24: DD5 was resolved by WITHDRAWING the ambiguous sibling
218+
# rather than marking it — the shipped family is gke-default plus
219+
# bundle-installer (the drawn operator-selfdriver), and no durable
220+
# marker or symmetric constraint pair exists. The operator value drawn
221+
# below shipped as driver-installer in v0.19.0 and was withdrawn when
222+
# bundle-installer landed.
220223
spec:
221224
profile:
222225
name: gpuStack
@@ -254,11 +257,10 @@ spec:
254257
constraints:
255258
- name: NodeTopology.gpu-nodes.label # requires #1755
256259
value: gke-no-default-nvidia-gpu-device-plugin=true
257-
# Amended 2026-08-22: DD5 symmetry — this value asserts the
258-
# ABSENCE of the installer's ownership marker at readiness.
259-
readinessConstraints:
260-
- name: NodeTopology.gpu-nodes.label
261-
value: "!feature.node.kubernetes.io/gcp-driver-installer"
260+
# Amended 2026-08-24: this value shipped as driver-installer in
261+
# v0.19.0 and was WITHDRAWN when bundle-installer landed — the
262+
# hand-applied installer arrangement is a migration source, not a
263+
# named mode (see the DD5 amendment).
262264
# GKE-installed driver AND GKE's managed device plugin — a
263265
# default-provisioned GKE cluster (no node label required). The
264266
# declared default: the only value satisfied with zero setup.
@@ -296,13 +298,10 @@ spec:
296298
constraints:
297299
- name: NodeTopology.gpu-nodes.label
298300
value: gke-no-default-nvidia-gpu-device-plugin=true
299-
# Amended 2026-08-22: the DD5 distinguishing signal — a property
300-
# the value's own deployment creates, absent from any
301-
# pre-deployment snapshot — is declared under readinessConstraints
302-
# and evaluated by the validate pre-flight only.
303-
readinessConstraints:
304-
- name: NodeTopology.gpu-nodes.label
305-
value: feature.node.kubernetes.io/gcp-driver-installer=true
301+
# Amended 2026-08-24: no readiness marker. DD5 was resolved by
302+
# withdrawing the operator (driver-installer) sibling; this value
303+
# shipped as bundle-installer, distinguishable from csp-managed
304+
# (gke-default) at generation time by the pool label alone.
306305
```
307306

308307
The GKE declaration lives once in `gke-cos`; accelerator/intent leaves
@@ -316,9 +315,10 @@ step 2 describes. Selection:
316315
# gke-default (declared default; drawn above as csp-managed) — no flag needed
317316
aicr recipe --service gke --os cos --accelerator h100 --intent inference
318317

319-
# explicit alternative configuration (shipped name; drawn above as operator)
318+
# explicit alternative configuration (shipped name; drawn above as
319+
# operator-selfdriver)
320320
aicr recipe --service gke --os cos --accelerator h100 --intent inference \
321-
--profile gpuStack=driver-installer
321+
--profile gpuStack=bundle-installer
322322
```
323323

324324
A profile fragment may reference only components **enabled in the
@@ -343,7 +343,8 @@ explicit toggles from changing the pre-existing presence state, with
343343
subset-filter semantics defined under Override locking.
344344

345345
**Conditional installation is expressible in v1 through a values-gated
346-
component** — the `operator-selfdriver` value above is the pattern.
346+
component** — the `operator-selfdriver` value above (shipped as
347+
`bundle-installer`) is the pattern.
347348
The component sits unconditionally in the composition and renders
348349
nothing unless its gate value is selected; the profile value flips a
349350
plain values path — owned, locked, and validated like any other —
@@ -1476,13 +1477,27 @@ recurrence — the shape the Problem section expects.
14761477
installer's synthetic `enabled` joins `ownedPaths`), which is a
14771478
family-wide re-qualification and evidence re-signing event.
14781479

1480+
*Amended 2026-08-24 (issue #1716).* The landing event resolved DD5 by
1481+
**withdrawing the ambiguous sibling instead of marking it**: the
1482+
deferred value shipped as `bundle-installer` (the `gcp-driver-installer`
1483+
component carries the installer; nested gate `installer.enabled` — see
1484+
the gate-key correction in the sketch), and `driver-installer` (shipped
1485+
in v0.19.0) was removed in the same event. The two remaining values are
1486+
distinguishable at generation time by the opt-out pool label alone
1487+
(positive vs negated), so no post-deployment marker exists on GKE.
1488+
Selecting `gpuStack=driver-installer` after the event fails closed with
1489+
the valid-values list; the migration is: delete any hand-applied
1490+
`nvidia-driver-installer` DaemonSet (the bundle's DaemonSet shares its
1491+
name) and regenerate with `--profile gpuStack=bundle-installer`.
1492+
14791493
Any dcgm-exporter GPU-ID-mapping adjustment for `csp-managed` is an
14801494
external GKE behavior not verifiable from this repository. It is
14811495
verified and added during this step if required, with upstream
14821496
citations recorded in that PR. Before conversion, this step also
14831497
checks the family's `/v1` usage or announces a deprecation window so
14841498
clients do not discover the `/v1` rejection at cut-over.
1485-
4. Other consumers: once `operator-selfdriver` is declared, internal
1499+
4. Other consumers: once `operator-selfdriver` (shipped as
1500+
`bundle-installer`) is declared, internal
14861501
recipes (DGXC/NKX) migrate the cos-gpu-installer arrangement
14871502
(internal MR #27) to the public value. The values-gated
14881503
`gcp-driver-installer` component makes the case expressible without
@@ -1514,8 +1529,8 @@ work that resolves it.
15141529
GKE's managed driver install, so the standalone gate's prerequisite
15151530
needed the profile's per-value pairing), and the GKE `gpuStack`
15161531
profile now consumes the form per selected value (#1761 rollout
1517-
PR 3): positive for `driver-installer`, negated for the
1518-
`gke-default` default.
1532+
PR 3): positive for `driver-installer` (later `bundle-installer`),
1533+
negated for the `gke-default` default.
15191534
3. **AKS node-pool-mode signal — resolved by the 2026-07-27 amendment.**
15201535
The provider-facing AgentPool `gpuProfile.driver` property is the
15211536
durable ownership marker. AKS adoption projects it into a snapshot
@@ -1542,40 +1557,40 @@ work that resolves it.
15421557
**Proposed: identify a durable signal during the value's adoption;
15431558
the `operator` and `csp-managed` values do not wait on it.**
15441559

1545-
*Amended 2026-08-22 (issue #1716).* Two parts land with the value's
1546-
adoption:
1547-
1548-
- **Mechanism.** `ProfileValue` gains `readinessConstraints` — same
1549-
catalog-load validation as `constraints` with per-phase name
1550-
deduplication (the same measurement path may carry a generation
1551-
pre-condition and a readiness post-deployment state — this is
1552-
exactly the DD5 shape, since both signals here are
1553-
`NodeTopology.gpu-nodes.label` readings), routed into
1554-
`spec.validation.readiness.constraints` at resolution and **never
1555-
evaluated at generation time**. This is required for any
1556-
post-deployment signal: generation-time evaluation runs against a
1557-
pre-deployment snapshot in which the signal cannot yet exist, and
1558-
the overlay-level readiness block cannot vary per value. The
1559-
`aicr validate` readiness pre-flight evaluates them with the same
1560-
fail-closed exit as every other readiness gate.
1561-
- **Signal.** GCP-native labels cannot distinguish the two unmanaged
1562-
values: both require identical pool shapes
1563-
(`gpu-driver-version=disabled` + the opt-out label; Google's own
1564-
installer DaemonSet schedules only where
1565-
`cloud.google.com/gke-gpu-driver-version` is absent, so a
1566-
version-labeled pool is unreachable for either). The signal is
1567-
therefore AICR-owned: the `gcp-driver-installer` DaemonSet stamps
1568-
a durable node label after a successful install;
1569-
`operator-selfdriver` asserts it under `readinessConstraints` and
1570-
`operator` (shipped name `driver-installer`) asserts its absence —
1571-
both declarations land together with the value's adoption (the
1572-
sketch above shows the declared shape).
1573-
A further hardening — a generation-time
1574-
`!cloud.google.com/gke-gpu-driver-version` constraint on both
1575-
unmanaged values, converting the documented "opt-out label +
1576-
managed install = driverless pool" misconfiguration into a
1577-
fail-closed recipe error — is DEFERRED: a value may carry one
1578-
constraint per measurement path per phase, and
1579-
`NodeTopology.gpu-nodes.label` is already occupied at generation
1580-
by the pool-label constraint. It requires a conjunction grammar
1581-
for the label form, tracked as follow-up work.
1560+
*Amended 2026-08-24 (issue #1716).* **Resolved by withdrawing the
1561+
ambiguous sibling.** The `operator` value (shipped `driver-installer`)
1562+
and the deferred `operator-selfdriver` were the same cluster shape with
1563+
different installer owners — the only distinction a durable marker could
1564+
have expressed. Instead of marking it, the landing event removed
1565+
`driver-installer` and shipped the deferred value as `bundle-installer`:
1566+
the two remaining values (`gke-default`, `bundle-installer`) differ in
1567+
the pre-existing opt-out pool label (negated vs positive), a
1568+
generation-time signal, so **no post-deployment marker exists on GKE**
1569+
and this decision needs no signal identification.
1570+
1571+
The mechanism this decision originally motivated survives with a
1572+
different first consumer: `ProfileValue` gains `readinessConstraints` —
1573+
same catalog-load validation as `constraints` with per-phase name
1574+
deduplication, routed into `spec.validation.readiness.constraints` at
1575+
resolution and **never evaluated at generation time**. It exists for
1576+
families whose values are distinguishable only by deployment-created
1577+
state, where no sibling can be withdrawn because the values are
1578+
distinct cluster shapes. The OKE `gpuStack` family is that consumer:
1579+
its three values (image driver + OKE plugin / image driver + operator
1580+
plugin / operator-managed everything) are three real pool shapes, and
1581+
`operator-plugin` vs `operator-managed` has exactly the
1582+
destroyed-by-success structure this decision described — the
1583+
pre-condition (no driver loaded) is erased by the value working, so
1584+
the durable distinguisher is the deployed ClusterPolicy state
1585+
(`K8s.policy.driver.enabled`, `K8s.policy.devicePlugin.enabled`),
1586+
evaluated by the `aicr validate` readiness pre-flight with the same
1587+
fail-closed exit as every other readiness gate.
1588+
1589+
Deferred hardening (unchanged): a generation-time
1590+
`!cloud.google.com/gke-gpu-driver-version` constraint on
1591+
`bundle-installer`, converting the documented "opt-out label + managed
1592+
install = driverless pool" misconfiguration into a fail-closed recipe
1593+
error, requires a conjunction grammar for the label form — a value may
1594+
carry one constraint per measurement path per phase, and
1595+
`NodeTopology.gpu-nodes.label` is already occupied at generation by the
1596+
pool-label constraint.

docs/integrator/automation.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -366,7 +366,7 @@ are not time-bounded and persist until `Client.Close()` is called.
366366
> family (profiled and unprofiled) with the same criteria parameters, plus
367367
> optional `profile=gpuStack=azure-managed` or
368368
> `profile=gpuStack=operator-managed` on AKS, and
369-
> `profile=gpuStack=gke-default` or `profile=gpuStack=driver-installer`
369+
> `profile=gpuStack=gke-default` or `profile=gpuStack=bundle-installer`
370370
> on GKE
371371
> (see [GKE GPU Setup](gke-gpu-setup.md#gpu-device-plugin-ownership)).
372372
> `/v1/recipe` still works for unprofiled compositions but rejects any

docs/integrator/data-extension.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -231,7 +231,7 @@ When in doubt, `aicr --debug recipe ... --data <dir>` logs the resolved source
231231
`recipes/overlays/aks.yaml` declares the `gpuStack` configuration profile
232232
(`azure-managed` default, `operator-managed` alternative) over the GPU driver/toolkit
233233
ownership paths, and `recipes/overlays/gke-cos.yaml` declares its own `gpuStack`
234-
(`gke-default` default, `driver-installer` alternative) over device-plugin ownership —
234+
(`gke-default` default, `bundle-installer` alternative) over device-plugin ownership —
235235
the GKE default value (`gke-default`) additionally declares
236236
`advertiser: external`, and both GKE values trigger the #1327
237237
allocation-policy closure, so their effective lock set is larger than the

0 commit comments

Comments
 (0)