feat(recipes): replace AKS ib-node-config DaemonSet with nodewright tuning

Removes the privileged ib-node-config-aks DaemonSet from the AKS
network-operator component and wires the AKS accelerator overlays
(h100-aks-inference, h100-aks-training, a100-aks-training) to the same
nodewright-customizations tuning manifest EKS uses, with service: aks.

nvidia-setup is bumped to 0.5.0 in tuning.yaml, which adds the aks-h100
combination (NVIDIA/nodewright-packages#42): host IB RDMA setup
(ib_umad/rdma_ucm module loading persisted via modules-load.d, memlock
limits, LimitMEMLOCK=infinity drop-ins for containerd/kubelet), replacing
what the DaemonSet did. 0.5.0 is otherwise identical to 0.4.0 for EKS, so
the shared manifest keeps working for the existing services. On AKS the
nvidia-setup-kernel package is a designed no-op (aks defaults omit KERNEL)
and nvidia-tuned applies the aks-h100-{intent} profiles.

Documents the AKS wildcard-toleration caveat in the shared manifest: AKS
admission collapses toleration lists to the wildcard, which breaks the
nodewright drain exemption for its own package pods (NVIDIA/nodewright#296),
so keyed acceleratedTolerations must be injected on AKS.

Verified end to end on AKS v1.35.5 with 2x Standard_ND96isr_H100_v5 running
this package chain (kernel no-op, nvidia-tuned aks-h100-inference profile,
IB modules + effective unlimited memlock, reboot persistence).

Regenerates the tuning-status table (aks rows) and the container-images BOM,
and updates the tuning structure test rows.

Signed-off-by: Alex Yuskauskas <ayuskauskas@nvidia.com>

Author: ayuskauskas

docs/integrator/components/nodewright.md

@@ -75,12 +75,14 @@ The table below is generated from the recipes by `make tuning-docs` — **do not
 
 | Service | Accelerator  | Profile | Setup              | Tuning                  |
 |---------|--------------|---------|--------------------|-------------------------|
+| aks     | a100         | h100    | nvidia-setup 0.5.0 | nvidia-tuned 0.3.1      |
+| aks     | h100         | -       | nvidia-setup 0.5.0 | nvidia-tuned 0.3.1      |
 | bcm     | *            | h100    | nvidia-setup 0.3.0 | -                       |
 | bcm     | h100         | -       | nvidia-setup 0.3.0 | -                       |
-| eks     | a100         | h100    | nvidia-setup 0.4.0 | nvidia-tuned 0.3.1      |
-| eks     | gb200        | -       | nvidia-setup 0.4.0 | nvidia-tuned 0.3.1      |
-| eks     | h100         | -       | nvidia-setup 0.4.0 | nvidia-tuned 0.3.1      |
-| eks     | h200         | h100    | nvidia-setup 0.4.0 | nvidia-tuned 0.3.1      |
+| eks     | a100         | h100    | nvidia-setup 0.5.0 | nvidia-tuned 0.3.1      |
+| eks     | gb200        | -       | nvidia-setup 0.5.0 | nvidia-tuned 0.3.1      |
+| eks     | h100         | -       | nvidia-setup 0.5.0 | nvidia-tuned 0.3.1      |
+| eks     | h200         | h100    | nvidia-setup 0.5.0 | nvidia-tuned 0.3.1      |
 | eks     | rtx-pro-6000 | generic | -                  | nvidia-tuned 0.3.1      |
 | gke     | a100         | h100    | -                  | nvidia-tuning-gke 0.1.2 |
 | gke     | b200         | -       | -                  | nvidia-tuning-gke 0.1.2 |

docs/user/container-images.md

@@ -20,7 +20,7 @@ A machine-readable **CycloneDX 1.6 JSON** companion to this page is produced by
 ## Summary
 
 - Components: **33**
-- Unique images: **82**
+- Unique images: **81**
 - Distinct registries: **11**
 
 Registries: `602401143452.dkr.ecr.us-west-2.amazonaws.com`, `cr.agentgateway.dev`, `docker.io`, `gcr.io`, `ghcr.io`, `gke.gcr.io`, `nvcr.io`, `public.ecr.aws`, `quay.io`, `registry.k8s.io`, `us-docker.pkg.dev`
@@ -47,7 +47,7 @@ Registries: `602401143452.dkr.ecr.us-west-2.amazonaws.com`, `cr.agentgateway.dev
 | kube-prometheus-stack | helm | prometheus-community/kube-prometheus-stack | 84.4.0 | 8 |
 | kubeflow-trainer | helm | kubeflow-trainer | 2.2.0 | 3 |
 | kueue | helm | kueue | 0.17.1 | 1 |
-| network-operator | helm | nvidia/network-operator | 26.1.1 | 5 |
+| network-operator | helm | nvidia/network-operator | 26.1.1 | 4 |
 | network-operator-ocp | manifest | — | — | 0 |
 | network-operator-ocp-olm | manifest | — | — | 0 |
 | nfd | helm | node-feature-discovery | 0.18.3 | 1 |
@@ -177,7 +177,6 @@ _No images extracted._
 
 ### network-operator
 
-- `busybox:1.38@sha256:fd8d9aa63ba2f0982b5304e1ee8d3b90a210bc1ffb5314d980eb6962f1a9715d`
 - `nvcr.io/nvidia/cloud-native/network-operator:v26.1.1`
 - `nvcr.io/nvidia/doca/doca_telemetry:1.22.5-doca3.1.0-host`
 - `nvcr.io/nvidia/mellanox/doca-driver:doca3.2.0-25.10-1.2.8.0-2`
@@ -206,7 +205,7 @@ _No images extracted._
 ### nodewright-customizations
 
 - `ghcr.io/nvidia/nodewright-packages/nvidia-setup:0.3.0@sha256:f17c951d60b519d097c20a3d9f49668f043a996adb31b9bb4db24a112a8f60a2`
-- `ghcr.io/nvidia/nodewright-packages/nvidia-setup:0.4.0@sha256:187c56c6d2104d48c7632c293def79c714b04dd85c587b7793818ceba9d4fb04`
+- `ghcr.io/nvidia/nodewright-packages/nvidia-setup:0.5.0@sha256:f3994267c9b5e62fb7720012dcd4d473fc2f8474f4276e203bba842c970307ad`
 - `ghcr.io/nvidia/nodewright-packages/nvidia-tuned:0.3.1@sha256:6dedbbb8627dec88acdfd68867f60a3ddca88e5f4c62beddfb704a8e0cfe51d1`
 - `ghcr.io/nvidia/nodewright-packages/nvidia-tuning-gke:0.1.2@sha256:6671d49f006afdbeefd8858f1fa1216f7748205bc42edab3340210a2cc459a81`
 - `ghcr.io/nvidia/skyhook-packages/shellscript:1.1.1`

pkg/tuning/compute_test.go

@@ -37,6 +37,8 @@ func TestCompute_Structure(t *testing.T) {
 		t.Fatalf("Compute: %v", err)
 	}
 	want := []wantRow{
+		{"aks", "a100", "h100", "nvidia-setup", "nvidia-tuned"},
+		{"aks", "h100", "-", "nvidia-setup", "nvidia-tuned"},
 		{"bcm", "*", "h100", "nvidia-setup", ""},
 		{"bcm", "h100", "-", "nvidia-setup", ""},
 		{"eks", "a100", "h100", "nvidia-setup", "nvidia-tuned"},

recipes/components/network-operator/manifests/ib-node-config-aks.yaml

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# Nodewright EKS Ubuntu GPU Customization
+# Nodewright Ubuntu GPU Customization (EKS and AKS)
 # Minimal templating - only tolerations/nodeSelectors are dynamic for CLI flag support
 #
 # This customization configures GPU nodes on Ubuntu with:
@@ -49,6 +49,11 @@ spec:
     # Tolerate-all default, matching ParseTolerations()/DefaultTolerations()
     # (operator Exists, no key). Only reached if no acceleratedTolerations are
     # injected from scheduling defaults/flags.
+    # AKS caveat: AKS admission collapses a pod toleration list to just the
+    # wildcard when one is present, which defeats the nodewright operator
+    # drain exemption for its own package pods and deadlocks packages that
+    # declare interrupts (NVIDIA/nodewright#296). On AKS always inject keyed
+    # acceleratedTolerations (for example nvidia.com/gpu).
     - operator: Exists
     {{- end }}
   # Dynamic: Supports --accelerated-node-selector CLI flag
@@ -65,11 +70,13 @@ spec:
   {{- end }}
   # Tuning
   packages:
-    # Check for kernel being >= to 6.17.0-1019 and if not install
+    # Check for kernel being >= to 6.17.0-1019 and if not install.
+    # On AKS this package is a designed no-op: the aks defaults omit KERNEL
+    # (AKS manages its own Ubuntu kernel) and ensure_kernel exits early.
     nvidia-setup-kernel:
       image: ghcr.io/nvidia/nodewright-packages/nvidia-setup
-      version: "0.4.0"
-      containerSHA: sha256:187c56c6d2104d48c7632c293def79c714b04dd85c587b7793818ceba9d4fb04
+      version: "0.5.0"
+      containerSHA: sha256:f3994267c9b5e62fb7720012dcd4d473fc2f8474f4276e203bba842c970307ad
       configMap:
         {{- if $cust.service }}
         service: {{ $cust.service }}
@@ -104,13 +111,13 @@ spec:
         service: {{ $cust.service }}
         {{- end }}
       dependsOn:
-        nvidia-setup-kernel: "0.4.0"
+        nvidia-setup-kernel: "0.5.0"
 
     # Full setup after kernel is in place
     nvidia-setup-full:
       image: ghcr.io/nvidia/nodewright-packages/nvidia-setup
-      version: "0.4.0"
-      containerSHA: sha256:187c56c6d2104d48c7632c293def79c714b04dd85c587b7793818ceba9d4fb04
+      version: "0.5.0"
+      containerSHA: sha256:f3994267c9b5e62fb7720012dcd4d473fc2f8474f4276e203bba842c970307ad
       interrupt:
         type: reboot
       resources:

recipes/components/nodewright-customizations/manifests/tuning.yaml

@@ -34,9 +34,6 @@ spec:
     - name: K8s.server.version
       value: ">= 1.30"
 
-  # Nodewright customizations omitted — Nodewright packages do not support
-  # service: aks. The nodewright-operator itself is inherited from base
-  # and still deploys. This follows the H100 AKS training pattern.
   componentRefs:
     # A100-specific GPU Operator overrides (inherits valuesFile from aks-training)
     - name: gpu-operator
@@ -45,10 +42,27 @@ spec:
         - nfd
         - cert-manager
         - kube-prometheus-stack
+        - nodewright-customizations
       overrides:
         gdrcopy:
           enabled: true
 
+    # A100 reuses the h100 nodewright tuning profile (tuning.yaml), mirroring
+    # a100-eks-training: nvidia-setup ships aks configs only for aks-h100, and
+    # per the nodewright maintainer the h100 tuning is the correct profile for
+    # A100 (the A100-vs-H100 deltas pertain only to baremetal). The recipe
+    # criteria above stays a100; only this tuning profile selector is h100.
+    - name: nodewright-customizations
+      type: Helm
+      manifestFiles:
+        - components/nodewright-customizations/manifests/tuning.yaml
+      overrides:
+        service: aks
+        accelerator: h100
+        intent: multiNodeTraining
+      dependencyRefs:
+        - nodewright-operator
+
     - name: nfd
       type: Helm
       overrides:

recipes/overlays/a100-aks-training.yaml

@@ -53,12 +53,16 @@ spec:
     # Manifests deliver the full RDMA stack that the Helm chart does not template:
     #   - NicClusterPolicy: MOFED driver, RDMA shared device plugin, DOCA telemetry
     #   - NFD NodeFeatureRule: labels IB-capable nodes (pci-15b3.present=true)
-    #   - ib-node-config: host MEMLOCK limits so workloads inherit unlimited rlimit
+    # Host MEMLOCK limits and IB module loading moved to nodewright
+    # (nvidia-setup aks-h100 via the nodewright-customizations tuning manifest,
+    # wired in the accelerator overlays), replacing the privileged
+    # ib-node-config DaemonSet.
     # Reference: https://azure.github.io/aks-rdma-infiniband/configurations/network-operator
     #
     # RDMA is enabled by default. To opt out at bundle time:
     #   --set networkoperator:enabled=false \
-    #   --set gpuoperator:driver.rdma.useHostMofed=false
+    #   --set gpuoperator:driver.rdma.useHostMofed=false \
+    #   --set nodewrightcustomizations:enabled=false
     - name: network-operator
       type: Helm
       source: https://helm.ngc.nvidia.com/nvidia
@@ -67,11 +71,11 @@ spec:
       manifestFiles:
         - components/network-operator/manifests/nfd-network-rule.yaml
         - components/network-operator/manifests/nic-cluster-policy-aks.yaml
-        - components/network-operator/manifests/ib-node-config-aks.yaml
       dependencyRefs:
         - nfd
         - cert-manager
 
+
     # Prometheus persistent storage (provide --storage-class at bundle time, e.g. managed-csi)
     - name: kube-prometheus-stack
       type: Helm

recipes/overlays/aks.yaml

@@ -32,17 +32,29 @@ spec:
     - name: K8s.server.version
       value: ">= 1.32.4"
 
-  # Nodewright customizations omitted — Nodewright packages do not support
-  # service: aks. The nodewright-operator itself is inherited from base
-  # and still deploys. This follows the same pattern as Kind overlays
-  # (h100-kind-inference.yaml) which also omit Nodewright tuning.
   componentRefs:
     - name: gpu-operator
       type: Helm
       dependencyRefs:
         - nfd
         - cert-manager
         - kube-prometheus-stack
+        - nodewright-customizations
+
+    # Same tuning manifest as EKS: nvidia-setup 0.5.0 supports service: aks
+    # (aks-h100), covering host IB RDMA + memlock setup that the
+    # ib-node-config-aks DaemonSet used to perform, and nvidia-tuned applies
+    # the aks-h100-inference profile.
+    - name: nodewright-customizations
+      type: Helm
+      manifestFiles:
+        - components/nodewright-customizations/manifests/tuning.yaml
+      overrides:
+        service: aks
+        accelerator: h100
+        intent: inference
+      dependencyRefs:
+        - nodewright-operator
 
     - name: nfd
       type: Helm