feat(recipes): replace AKS ib-node-config DaemonSet with nodewright tuning

docs/contributor/validator.md

@@ -651,6 +651,7 @@ audit existing condition blocks per CLAUDE.md's enum-expansion rule.
 | `CheckWorkloadSelectorMissing` | nodewright `--workload-selector` set when conditions match |
 | `CheckAcceleratedSelectorMissing` | nodewright `--accelerated-node-selector` set |
 | `CheckHostMofedWithoutNetworkOperator` | Host-mode MOFED component paired with `network-operator` |
+| `CheckWildcardAcceleratedToleration` | Accelerated-node tolerations carry no wildcard (keyless) entry — on AKS a wildcard deadlocks nodewright interrupt packages ([nodewright#296](https://github.com/NVIDIA/nodewright/issues/296)); wired at `severity: error`, skipped when the component is disabled via `--set` |
 
 Registered in `pkg/bundler/validations/checks.go::init()`.
 

docs/integrator/aks-gpu-setup.md

@@ -232,6 +232,46 @@ in `recipes/components/gpu-operator/values-aks.yaml`).
 
 This profile has not yet been validated end-to-end with network-operator/RDMA.
 
+## InfiniBand RDMA Host Setup (nodewright)
+
+AKS recipes deliver the InfiniBand RDMA host configuration (persistent
+`ib_umad`/`rdma_ucm` module loading, `LimitMEMLOCK=infinity` for containerd and
+kubelet) through nodewright: the `nodewright-customizations` component applies
+`nvidia-setup` and `nvidia-tuned` packages to GPU nodes, with reboots handled
+as nodewright interrupts. This replaces the earlier privileged
+`ib-node-config` DaemonSet.
+
+**Pass a keyed toleration on AKS.** AKS admission collapses a pod's toleration
+list to just the wildcard (`operator: Exists`, no key) when one is present,
+which defeats the nodewright operator's drain exemption for its own package
+pods and deadlocks packages that declare interrupts on first install
+([nodewright#296](https://github.com/NVIDIA/nodewright/issues/296)). Recovering
+from that deadlock requires manually cordoning and rebooting the node. Because
+`aicr bundle` injects a wildcard toleration by default when
+`--accelerated-node-toleration` is not set, always bundle AKS recipes with a
+keyed toleration matching your GPU node taint:
+
+```shell
+aicr bundle -r recipe.yaml \
+  --accelerated-node-toleration nvidia.com/gpu:NoSchedule \
+  -o ./bundles
+```
+
+Bundling an AKS recipe without a keyed toleration is a **blocking error**
+(`CheckWildcardAcceleratedToleration`): the bundle is not produced until you
+supply one, so the deadlock cannot ship silently.
+
+To opt out of the RDMA stack entirely (e.g., on non-InfiniBand SKUs) — this
+disables `nodewright-customizations`, so no keyed toleration is required:
+
+```shell
+aicr bundle -r recipe.yaml \
+  --set networkoperator:enabled=false \
+  --set gpuoperator:driver.rdma.useHostMofed=false \
+  --set nodewrightcustomizations:enabled=false \
+  -o ./bundles
+```
+
 ## References
 
 - [GPU Operator on AKS](https://learn.microsoft.com/en-us/azure/aks/nvidia-gpu-operator)

docs/integrator/components/nodewright.md

@@ -75,12 +75,14 @@ The table below is generated from the recipes by `make tuning-docs` — **do not
 
 | Service | Accelerator  | Profile | Setup              | Tuning                  |
 |---------|--------------|---------|--------------------|-------------------------|
+| aks     | a100         | h100    | nvidia-setup 0.5.0 | nvidia-tuned 0.3.1      |
+| aks     | h100         | -       | nvidia-setup 0.5.0 | nvidia-tuned 0.3.1      |
 | bcm     | *            | h100    | nvidia-setup 0.3.0 | -                       |
 | bcm     | h100         | -       | nvidia-setup 0.3.0 | -                       |
-| eks     | a100         | h100    | nvidia-setup 0.4.0 | nvidia-tuned 0.3.1      |
-| eks     | gb200        | -       | nvidia-setup 0.4.0 | nvidia-tuned 0.3.1      |
-| eks     | h100         | -       | nvidia-setup 0.4.0 | nvidia-tuned 0.3.1      |
-| eks     | h200         | h100    | nvidia-setup 0.4.0 | nvidia-tuned 0.3.1      |
+| eks     | a100         | h100    | nvidia-setup 0.5.0 | nvidia-tuned 0.3.1      |
+| eks     | gb200        | -       | nvidia-setup 0.5.0 | nvidia-tuned 0.3.1      |
+| eks     | h100         | -       | nvidia-setup 0.5.0 | nvidia-tuned 0.3.1      |
+| eks     | h200         | h100    | nvidia-setup 0.5.0 | nvidia-tuned 0.3.1      |
 | eks     | rtx-pro-6000 | generic | -                  | nvidia-tuned 0.3.1      |
 | gke     | a100         | h100    | -                  | nvidia-tuning-gke 0.1.2 |
 | gke     | b200         | -       | -                  | nvidia-tuning-gke 0.1.2 |

docs/user/container-images.md

@@ -20,7 +20,7 @@ A machine-readable **CycloneDX 1.6 JSON** companion to this page is produced by
 ## Summary
 
 - Components: **33**
-- Unique images: **86**
+- Unique images: **85**
 - Distinct registries: **11**
 
 Registries: `602401143452.dkr.ecr.us-west-2.amazonaws.com`, `cr.agentgateway.dev`, `docker.io`, `gcr.io`, `ghcr.io`, `gke.gcr.io`, `nvcr.io`, `public.ecr.aws`, `quay.io`, `registry.k8s.io`, `us-docker.pkg.dev`
@@ -49,7 +49,7 @@ _Rendering fidelity:_ `catalog-parity: charts are rendered with the shared recip
 | kube-prometheus-stack | helm | prometheus-community/kube-prometheus-stack | 84.4.0 | 8 |
 | kubeflow-trainer | helm | kubeflow-trainer | 2.2.0 | 3 |
 | kueue | helm | kueue | 0.17.1 | 1 |
-| network-operator | helm | nvidia/network-operator | 26.1.1 | 5 |
+| network-operator | helm | nvidia/network-operator | 26.1.1 | 4 |
 | network-operator-ocp | manifest | — | — | 0 |
 | network-operator-ocp-olm | manifest | — | — | 0 |
 | nfd | helm | node-feature-discovery | 0.19.0 | 1 |
@@ -188,7 +188,6 @@ _No images extracted._
 
 ### network-operator
 
-- `busybox:1.38@sha256:fd8d9aa63ba2f0982b5304e1ee8d3b90a210bc1ffb5314d980eb6962f1a9715d`
 - `nvcr.io/nvidia/cloud-native/network-operator:v26.1.1`
 - `nvcr.io/nvidia/doca/doca_telemetry:1.22.5-doca3.1.0-host`
 - `nvcr.io/nvidia/mellanox/doca-driver:doca3.2.0-25.10-1.2.8.0-2`
@@ -217,7 +216,7 @@ _No images extracted._
 ### nodewright-customizations
 
 - `ghcr.io/nvidia/nodewright-packages/nvidia-setup:0.3.0@sha256:f17c951d60b519d097c20a3d9f49668f043a996adb31b9bb4db24a112a8f60a2`
-- `ghcr.io/nvidia/nodewright-packages/nvidia-setup:0.4.0@sha256:187c56c6d2104d48c7632c293def79c714b04dd85c587b7793818ceba9d4fb04`
+- `ghcr.io/nvidia/nodewright-packages/nvidia-setup:0.5.0@sha256:f3994267c9b5e62fb7720012dcd4d473fc2f8474f4276e203bba842c970307ad`
 - `ghcr.io/nvidia/nodewright-packages/nvidia-tuned:0.3.1@sha256:6dedbbb8627dec88acdfd68867f60a3ddca88e5f4c62beddfb704a8e0cfe51d1`
 - `ghcr.io/nvidia/nodewright-packages/nvidia-tuning-gke:0.1.2@sha256:6671d49f006afdbeefd8858f1fa1216f7748205bc42edab3340210a2cc459a81`
 - `ghcr.io/nvidia/skyhook-packages/shellscript:1.1.1`

pkg/bundler/validations/checks.go

@@ -23,6 +23,11 @@ import (
 	"github.com/NVIDIA/aicr/pkg/recipe"
 )
 
+// overrideValueFalse is the string form of a disabled `--set <key>:enabled=false`
+// override — value overrides are collected as strings, so booleans arrive as
+// their string literal.
+const overrideValueFalse = "false"
+
 // init auto-registers validation functions in this package.
 // This allows the registry to discover validation functions automatically.
 func init() {
@@ -31,6 +36,7 @@ func init() {
 	registerCheck("CheckWorkloadSelectorMissing", CheckWorkloadSelectorMissing)
 	registerCheck("CheckAcceleratedSelectorMissing", CheckAcceleratedSelectorMissing)
 	registerCheck("CheckHostMofedWithoutNetworkOperator", CheckHostMofedWithoutNetworkOperator)
+	registerCheck("CheckWildcardAcceleratedToleration", CheckWildcardAcceleratedToleration)
 }
 
 // registerCheck is a helper to register validation functions from checks.go.
@@ -167,6 +173,89 @@ func checkConditions(recipeResult *recipe.RecipeResult, conditions map[string][]
 	return true
 }
 
+// nodewrightCustomizationsOverrideAliases are the registry valueOverrideKeys for
+// the nodewright-customizations component — the aliases (beyond the exact
+// component name) a user passes to --set to disable it, e.g.
+// --set nodewrightcustomizations:enabled=false. The bundler resolves --set
+// overrides under the exact component name AND these aliases
+// (DefaultBundler.componentOverrideKeys), so the disable check below mirrors
+// that set to avoid a false positive when a user disables via one form and the
+// check reads another.
+var nodewrightCustomizationsOverrideAliases = []string{"nodewrightcustomizations", "skyhookcustomizations"}
+
+// CheckWildcardAcceleratedToleration reports when the effective accelerated-node
+// tolerations for a component include a wildcard (keyless operator: Exists)
+// toleration. Scope it via registry conditions to services where the wildcard
+// is harmful — on AKS, admission collapses a pod's toleration list to just the
+// wildcard when one is present, which defeats the nodewright operator's drain
+// exemption for its own package pods and deadlocks packages that declare
+// interrupts (NVIDIA/nodewright#296). That deadlock requires manual node
+// cordon/reboot to recover, so the registry wires this at severity: error to
+// block the bundle until a keyed toleration is supplied.
+//
+// The default bundle path always hits this: with no
+// --accelerated-node-toleration flag the CLI falls back to
+// snapshotter.DefaultTolerations() (a single bare operator: Exists). An empty
+// toleration list is flagged too, because the tuning manifest template renders
+// its own wildcard fallback when none are injected.
+//
+// A component disabled via --set (e.g. the documented RDMA opt-out
+// --set nodewrightcustomizations:enabled=false) renders no package pods and
+// cannot deadlock, so it is skipped regardless of the toleration shape.
+func CheckWildcardAcceleratedToleration(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error) {
+	if bundlerConfig == nil {
+		return nil, nil
+	}
+
+	// Check if component exists in recipe
+	hasComponent := false
+	for _, ref := range recipeResult.ComponentRefs {
+		if ref.Name == componentName {
+			hasComponent = true
+			break
+		}
+	}
+
+	if !hasComponent {
+		return nil, nil
+	}
+
+	// Check conditions (e.g., service: aks)
+	if !checkConditions(recipeResult, conditions) {
+		return nil, nil
+	}
+
+	// A disabled component renders nothing, so it cannot deadlock — skip it.
+	// Check the exact component name and its registry aliases, mirroring how
+	// the bundler resolves --set overrides so any disable form is honored.
+	overrides := bundlerConfig.ValueOverrides()
+	for _, key := range append([]string{componentName}, nodewrightCustomizationsOverrideAliases...) {
+		if overrides[key]["enabled"] == overrideValueFalse {
+			return nil, nil
+		}
+	}
+
+	tolerations := bundlerConfig.AcceleratedNodeTolerations()
+	wildcard := len(tolerations) == 0 // template falls back to its own wildcard
+	for _, tol := range tolerations {
+		if tol.Key == "" {
+			wildcard = true
+			break
+		}
+	}
+
+	if !wildcard {
+		return nil, nil
+	}
+
+	baseMsg := fmt.Sprintf("%s renders a wildcard (keyless) accelerated-node toleration", componentName)
+	slog.Warn(baseMsg,
+		"component", componentName,
+		"conditions", conditions,
+	)
+	return []string{baseMsg}, nil
+}
+
 // CheckHostMofedWithoutNetworkOperator warns when network-operator is disabled
 // via --set but gpu-operator still has driver.rdma.useHostMofed=true (the
 // AKS default). Without network-operator, no host MOFED is present and
@@ -189,14 +278,14 @@ func CheckHostMofedWithoutNetworkOperator(ctx context.Context, componentName str
 	}
 
 	enabledVal, hasEnabled := netOpOverrides["enabled"]
-	if !hasEnabled || enabledVal != "false" {
+	if !hasEnabled || enabledVal != overrideValueFalse {
 		return nil, nil
 	}
 
 	// network-operator is disabled — check if useHostMofed is overridden to false
 	gpuOpOverrides := overrides["gpuoperator"]
 	if gpuOpOverrides != nil {
-		if mofedVal, ok := gpuOpOverrides["driver.rdma.useHostMofed"]; ok && mofedVal == "false" {
+		if mofedVal, ok := gpuOpOverrides["driver.rdma.useHostMofed"]; ok && mofedVal == overrideValueFalse {
 			return nil, nil
 		}
 	}