feat(validation): detect missing GKE GPU NIC networks at deployment phase (NVIDIA#2248)

Signed-off-by: Yuan Chen <yuanchen97@gmail.com>

Author: yuanchen8911

docs/integrator/gke-tcpxo-networking.md

@@ -13,10 +13,83 @@ GKE clusters must have multi-NIC networking configured before deploying AICR bun
 - `nccl-tcpxo-installer` DaemonSet on GPU nodes (included in AICR bundle)
 - `nri-device-injector` DaemonSet on GPU nodes (included in AICR bundle)
 
+The last two ship in the AICR bundle. The first two are **cluster provisioning**
+— AICR detects them but does not create them.
+
+### Provisioning multi-NIC networking
+
+These four steps are ordered. Step 2 is the one that cannot be undone later;
+steps 3 and 4 must both complete before any TCPXO workload — or `aicr validate` —
+will work.
+
+1. **Create the VPCs and subnets** — one dedicated VPC + subnet per GPU NIC,
+   eight in total, in the cluster's region.
+2. **Create the cluster** with `--enable-multi-networking`, plus its two
+   prerequisites `--enable-dataplane-v2` and `--enable-ip-alias`.
+3. **Create the GPU node pool** on an `a3-megagpu-8g` machine type with
+   `--enable-gvnic`, attaching the eight VPC/subnet pairs as repeated
+   `--additional-node-network` entries, one per pair, each in the form
+   `network=NETWORK,subnetwork=SUBNET`.
+4. **Apply the `Network` and `GKENetworkParamSet` CRs** — one pair per GPU NIC,
+   binding each additional node network into the cluster so pods can reference
+   it by name. **Each `Network` name must contain `gpu-nic`** — for example
+   `gpu-nic-0` through `gpu-nic-7`, optionally with a cluster prefix such as
+   `aicr-demo2-gpu-nic-0`. The paired `GKENetworkParamSet` is referenced by the
+   `Network` through `spec.parametersRef`, so its own name is unconstrained.
+
+> **The `gpu-nic` naming is a requirement, not a convention.** AICR discovers
+> these networks by matching `gpu-nic` in the `Network` object's
+> `metadata.name` — both the `gke-gpu-nic-networks` deployment check and the
+> NCCL benchmark's own interface mapping. Google's sample manifests name the
+> Device networks `vpc1`–`vpc8`; applied verbatim those are invisible to AICR,
+> and the deployment check reports 0 of 8 on a cluster that is otherwise
+> correctly provisioned. Rename them when following that procedure.
+>
+> Beyond containing `gpu-nic`, the exact names are yours to choose — but the
+> workload annotation below must reference the names your cluster actually has.
+> The example there uses `gpu-nic0`–`gpu-nic7`; if you provisioned
+> `gpu-nic-0`–`gpu-nic-7`, use those instead.
+
+> **Multi-networking cannot be enabled after cluster creation.** `--enable-multi-networking`
+> is a create-time flag; there is no `gcloud container clusters update` equivalent,
+> so a cluster created without it must be **recreated**. Steps 3 and 4, by
+> contrast, can be done on an existing multi-networking cluster — a node pool can
+> be added later, and the CRs can be applied at any point.
+
+AICR installs the TCPXO DaemonSets and detects the CRs; it does not provision any
+of this networking. These steps are a summary of the prerequisite AICR depends
+on, not a complete provisioning runbook — for the full procedure, including the
+per-VPC firewall rules and the supported GKE version floors, follow Google's
+[GPUDirect and multi-networking guide](https://cloud.google.com/kubernetes-engine/docs/how-to/gpu-bandwidth-gpudirect-tcpx).
+
+Completing steps 1–3 without step 4 is the failure mode worth knowing: the VMs
+come up with all nine NICs attached (the node's primary interface plus the eight
+GPU NICs) and the AICR TCPXO DaemonSets roll out cleanly, but with no `Network`
+objects bound into the cluster no pod can reference a GPU NIC and TCPXO cannot
+function.
+
+### Verifying
+
+```shell
+kubectl get network.networking.gke.io
+```
+
+Expect eight GPU NIC entries (plus the `default` network). Match on the
+`gpu-nic` substring rather than an exact name: the rest of each name is chosen
+at provisioning time and may carry a local prefix, such as
+`aicr-demo2-gpu-nic-0`.
+
+Fewer than eight means the prerequisite is incomplete. AICR's
+`gke-gpu-nic-networks` deployment check asserts this same count, so
+`aicr validate --phase deployment` reports the shortfall by name rather than
+letting it surface later as a performance-phase abort with no bandwidth number.
+
 **Important:** The GPU node pool must be provisioned with only the 8 GPU NIC
 networks (`gpu-nic-0` through `gpu-nic-7`). Do **not** include a gVNIC additional
 network — it takes a GPU NIC PCI slot (`0000:06:00.0`), leaving only 7/8 GPUs
-available for TCPXO.
+available for TCPXO. This is distinct from the `--enable-gvnic` node-pool flag,
+which selects the gVNIC driver and **is** required: pass the flag, but do not add
+a ninth `--additional-node-network` entry for it.
 
 ## Workload Pod Configuration (NRI Profile)
 

recipes/overlays/h100-gke-cos-training.yaml

@@ -81,6 +81,11 @@ spec:
         - expected-resources
         - gpu-operator-version
         - check-nvidia-smi
+        # Fails closed when the cluster lacks the Network / GKENetworkParamSet
+        # objects GPUDirect TCPXO needs. The gke-nccl-tcpxo DaemonSets roll out
+        # cleanly without them, so without this the gap only surfaces as a
+        # performance-phase abort with no bandwidth number produced (#2216).
+        - gke-gpu-nic-networks
       constraints:
         - name: Deployment.gpu-operator.version
           value: ">= v24.6.0"

recipes/validators/README.md

@@ -41,6 +41,7 @@ Applied by `catalog.Load` (`pkg/validator/catalog/catalog.go`) in order:
 | `expected-resources` | Verify expected Kubernetes resources exist and are healthy (runs ExpectedResources + Chainsaw assert paths side-by-side) | 8m |
 | `gpu-operator-version` | Validate GPU Operator version against recipe constraints | 2m |
 | `check-nvidia-smi` | Verify nvidia-smi works on all schedulable GPU nodes (cordoned nodes are disclosed, not silently skipped) | 10m |
+| `gke-gpu-nic-networks` | Verify the GKE cluster has the GPU NIC networks GPUDirect TCPXO requires (skipped unless the recipe declares `gke-nccl-tcpxo`) | 2m |
 
 ### Performance Phase
 

recipes/validators/catalog.yaml

@@ -65,6 +65,13 @@ validators:
     timeout: 10m
     args: ["check-nvidia-smi"]
     env: []
+  - name: gke-gpu-nic-networks
+    phase: deployment
+    description: "Verify the GKE cluster has the GPU NIC networks GPUDirect TCPXO requires (skipped unless the recipe declares gke-nccl-tcpxo)"
+    image: ghcr.io/nvidia/aicr-validators/deployment:latest
+    timeout: 2m
+    args: ["gke-gpu-nic-networks"]
+    env: []
   # Performance phase
   #
   # The NCCL checks (nccl-all-reduce-bw and the -net/-nvls variants at the

validators/deployment/gke_gpu_nic_networks.go

@@ -0,0 +1,121 @@
+// Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+	"fmt"
+	"log/slog"
+
+	"github.com/NVIDIA/aicr/pkg/errors"
+	"github.com/NVIDIA/aicr/validators"
+	"github.com/NVIDIA/aicr/validators/internal/gkenet"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+)
+
+// tcpxoComponent is the recipe componentRef that supplies GPUDirect TCPXO.
+const tcpxoComponent = "gke-nccl-tcpxo"
+
+// checkGKEGPUNICNetworks verifies the cluster has the GKE multi-NIC networking
+// objects GPUDirect TCPXO depends on.
+//
+// The gke-nccl-tcpxo component ships two DaemonSets, and both roll out cleanly
+// on a cluster that has zero Network / GKENetworkParamSet objects — so the
+// component's health check reports Synced+Healthy while TCPXO cannot function.
+// Without this check the gap surfaces hours later as a performance-phase abort
+// in the NCCL benchmark's own discovery, with no bandwidth number produced.
+//
+// The Network CRs are infrastructure: creating and binding them belongs to
+// cluster provisioning, not AICR. This check only detects their absence and
+// names the prerequisite, at the deployment phase where it is actionable.
+func checkGKEGPUNICNetworks(ctx *validators.Context) error {
+	if ctx.DynamicClient == nil {
+		return errors.New(errors.ErrCodeInvalidRequest, "dynamic client is not available")
+	}
+
+	slog.Info("listing GKE networks", "gvr", gkenet.NetworkGVR.String())
+
+	gpuNICs, listErr := gkenet.DiscoverGPUNICNetworks(ctx.Ctx, ctx.DynamicClient)
+
+	capability := validators.Capability{
+		Component: tcpxoComponent,
+		Subject:   "GKE Networks (networks.networking.gke.io)",
+		AbsentMsg: absentPrerequisiteMsg("the cluster does not serve the networks.networking.gke.io " +
+			"API at all, so it has 0"),
+		InapplicableMsg: tcpxoComponent + " not declared in recipe and the cluster has no GKE Network " +
+			"API — cluster does not use GPUDirect TCPXO",
+	}
+
+	// An ABSENT Network API is clean absence, not an infrastructure failure: the
+	// CRD arrives with --enable-multi-networking, so a cluster created without it
+	// legitimately does not serve this GVR. Route that shape through Require,
+	// which is declaration-gated — an undeclared recipe skips, a declared one gets
+	// the actionable message. RequireList would classify it as a blocking INTERNAL
+	// error, which both false-fails an undeclared recipe and hides the missing
+	// prerequisite behind "failed to read" on a declared one.
+	if apierrors.IsNotFound(listErr) {
+		// present is unused here: Require consults it only when probeErr is nil.
+		return capability.Require(ctx, listErr, false)
+	}
+
+	// Every other list error blocks regardless of declaration — an RBAC denial or
+	// an apiserver hiccup is not evidence that TCPXO is inapplicable.
+	if err := capability.RequireList(listErr); err != nil {
+		return err
+	}
+
+	// The prerequisite belongs to gke-nccl-tcpxo: a recipe that does not declare
+	// the component is not asking for TCPXO, so its cluster's networking is not
+	// this check's business. This also covers the #1327 standalone-run boundary,
+	// where there is no recipe context at all.
+	if !validators.RecipeDeclares(ctx, tcpxoComponent) {
+		return validators.Skip(
+			tcpxoComponent + " not declared in recipe — GPUDirect TCPXO networking is inapplicable")
+	}
+
+	// Evidence to stdout.
+	fmt.Printf("Found %d GPU NIC network(s) (need %d):\n", len(gpuNICs), gkenet.RequiredGPUNICNetworks)
+	for _, name := range gpuNICs {
+		fmt.Printf("  %s\n", name)
+	}
+
+	if len(gpuNICs) < gkenet.RequiredGPUNICNetworks {
+		return errors.New(errors.ErrCodeNotFound, absentPrerequisiteMsg(fmt.Sprintf(
+			"the cluster has %d of %d", len(gpuNICs), gkenet.RequiredGPUNICNetworks)))
+	}
+
+	return nil
+}
+
+// absentPrerequisiteMsg builds the operator-facing message for a missing GPU NIC
+// networking prerequisite. detail names what was actually observed; the rest is
+// the constant remediation, kept in one place so the absent-API path and the
+// short-count path cannot drift.
+//
+// The message names the required naming convention because it is a real way to
+// hit a zero count on an otherwise correctly provisioned cluster: discovery
+// matches the substring against the NETWORK name only, and Google's own sample
+// manifests name the Device networks vpc1..vpc8.
+func absentPrerequisiteMsg(detail string) string {
+	return fmt.Sprintf(
+		"recipe declares %s but %s GPU NIC networks — GPUDirect TCPXO requires one Network "+
+			"per GPU NIC, each bound to a GKENetworkParamSet and each with %q in its own "+
+			"metadata.name (this check counts Network names; it does not verify the "+
+			"GKENetworkParamSet binding or readiness). "+
+			"These are provisioned with the cluster, not by AICR, and multi-networking "+
+			"(--enable-multi-networking) cannot be enabled after cluster creation. "+
+			"Verify with: kubectl get network.networking.gke.io "+
+			"(see docs/integrator/gke-tcpxo-networking.md)",
+		tcpxoComponent, detail, gkenet.GPUNICNameSubstring)
+}