Skip to content

component-catalog.md

Latest commit

 

History

History
506 lines (365 loc) · 61.5 KB

component-catalog.md

File metadata and controls

506 lines (365 loc) · 61.5 KB
Raw

Component Catalog

AICR recipes are composed of components — the individual software packages that make up a GPU-accelerated Kubernetes runtime. This page lists every component that can appear in a recipe.

Note: Components are included as appropriate in recipes. Not every component listed here will appear in a recipe.

The source of truth is recipes/registry.yaml. Each entry in the registry defines the component's Helm chart (or Kustomize source), default version, namespace, and node scheduling configuration. If a component is not listed there, it cannot appear in a recipe.

See also: Recipe Health reports the structural health of every recipe these components compose into — resolvability and chart-pin hygiene across the whole criteria matrix.

Components

Component Description Source
gpu-operator Manages the GPU driver and runtime lifecycle on Kubernetes nodes. Handles driver installation, container runtime configuration, device plugin, and GPU feature discovery. NVIDIA GPU Operator
network-operator Manages high-performance networking for GPU workloads. Configures RDMA, SR-IOV, and host networking for multi-node communication. NVIDIA Network Operator
nfd Node Feature Discovery — labels nodes with hardware features (PCI device IDs, kernel modules, CPU capabilities). Both gpu-operator and network-operator consume these labels. On production GPU recipes, the Topology Updater publishes per-node NodeResourceTopology CRDs describing NUMA zones and GPU/NIC affinity for downstream NUMA-aware schedulers. Node Feature Discovery
gke-nccl-tcpxo NCCL TCPxO network plugin for GKE. Provides optimized collective communication for multi-node GPU workloads on Google Kubernetes Engine. GKE-specific. —
aws-efa Device plugin for AWS Elastic Fabric Adapter. Enables low-latency networking on EKS clusters with EFA-capable instances. EKS-specific. AWS EFA K8s Device Plugin
cert-manager Automates TLS certificate management. Required by several operators for webhook and API server certificates. cert-manager
gatekeeper Admission controller for Kubernetes. Enforces policies and governance across the cluster using OPA (Open Policy Agent) ConstraintTemplates and Constraints. Open Policy Agent Gatekeeper
nodewright-operator OS-level node tuning and configuration management. Applies kernel parameters, sysctl settings, and system-level optimizations to nodes. Nodewright
nodewright-customizations Environment-specific node tuning profiles applied via Nodewright. Extends the operator with kernel params, hugepages, and other host-level configurations. —
nvsentinel GPU health monitoring and automated remediation. Detects GPU errors and can cordon or drain affected nodes. On platforms where the provider installs the driver but no driver pod is observable by NVSentinel, the recipes set labeler.assumeDriverInstalled for you — see NVSentinel on provider-installed-driver platforms. NVSentinel
nvidia-dra-driver-gpu Dynamic Resource Allocation (DRA) driver. Advertises devices via the Kubernetes resource.k8s.io API (v1 on 1.34+, v1beta1/v1beta2 on 1.32/1.33) — ComputeDomain/IMEX channels for MNNVL platforms, and optionally whole GPUs. Stock recipes disable whole-GPU DRA advertisement (resources.gpus.enabled: false) — the device plugin is the production default whole-GPU advertiser, and DRA whole-GPU allocation is an experimental recipe-level opt-in (#1327). Whole-GPU DRA and the GPU Operator device plugin (nvidia.com/gpu) are mutually exclusive per node: recipe-backed validation rejects a configuration that enables both (at policy-resolution time — skipping validation bypasses the check), because the two allocators keep independent ledgers and concurrent advertisement can double-allocate the same physical GPUs (see the guidance in recipes/components/nvidia-dra-driver-gpu/values.yaml). See AKS GPU Setup for details. CLI alias: dradriver. NVIDIA DRA Driver
prometheus-operator-crds Custom Resource Definitions for the prometheus-operator (Alertmanager, AlertmanagerConfig, PodMonitor, Probe, Prometheus, PrometheusRule, ServiceMonitor, ThanosRuler). Shipped as a separate release so the CRDs land before any chart that creates monitoring CRs; this breaks the helm-diff self-reference that otherwise blocks helmfile apply on a fresh cluster. prometheus-operator-crds
kube-prometheus-stack Cluster monitoring: Prometheus, Grafana, Alertmanager, and node exporters. Provides GPU and cluster metrics collection and dashboards. CRDs are installed by the sibling prometheus-operator-crds release (this chart runs with crds.enabled: false). kube-prometheus-stack
prometheus-adapter Exposes custom metrics from Prometheus to the Kubernetes metrics API. Enables HPA scaling based on GPU utilization and other custom metrics. prometheus-adapter
aws-ebs-csi-driver CSI driver for Amazon EBS volumes. Provides persistent storage for workloads on EKS. EKS-specific. Cluster-wide default StorageClass: AICR enables defaultStorageClass.enabled, so this component provisions a cluster-default gp3 StorageClass (ebs-csi-default-sc) on every EKS cluster that includes it — not just inference recipes; training overlays inherit it too. EKS ships no default SC of its own, so this makes dynamic provisioning (e.g. the inference-perf model cache) work zero-config. Two consequences to note: (1) if the cluster already has a default SC, Kubernetes treats multiple defaults as ambiguous — unset the other; (2) a PVC that previously failed-fast on "no default SC" will now silently bind gp3, which can mask a misconfiguration. AWS EBS CSI Driver
k8s-ephemeral-storage-metrics Exports ephemeral storage usage metrics per pod. Useful for monitoring scratch space consumption on GPU nodes. k8s-ephemeral-storage-metrics
k8s-aibom Optional runtime AI workload inventory. Produces namespace-scoped CycloneDX 1.6 ML-BOM resources for explicitly opted-in namespaces. Registry-only: no stock recipe installs it. CLI aliases: k8saibom, aibom. See k8s-aibom Runtime Inventory. k8s-aibom
kai-scheduler Gang scheduler with hierarchical queues and topology-aware placement; works with device-plugin (nvidia.com/gpu) and DRA GPU allocation alike. Ensures distributed training jobs land on nodes with optimal interconnect topology. KAI Scheduler
grove Pod lifecycle management for Dynamo inference platform. Installed as a standalone component. Grove
dynamo-platform NVIDIA Dynamo inference serving platform with bundled CRDs. Distributed inference with KV-cache-aware routing, Dynamo request-plane traffic, a NATS-backed Kubernetes event plane for KV-cache events, and disaggregated prefill/decode. Dynamo
agentgateway-crds Custom Resource Definitions for agentgateway (Kubernetes Gateway API implementation for AI/ML inference). agentgateway
agentgateway Kubernetes Gateway API implementation for AI/ML inference. Implements the Gateway API Inference Extension for model-aware ingress routing to InferencePool backends. agentgateway
k8s-nim-operator NVIDIA NIM Operator for managing NIM (NVIDIA Inference Microservices) deployments on Kubernetes. AICR installs the operator only — it creates no NIMService and no credentials; see NIM workload credentials. K8s NIM Operator
kueue Kubernetes-native job queuing system. Manages quotas and admits jobs for batch and AI workloads. Ships default quota CRs (ResourceFlavor default-flavor, ClusterQueue cluster-queue, LocalQueue default in the default namespace) so admission works out of the box — tune the ClusterQueue's nominal quotas to cluster capacity to enact real limits. Managed frameworks are pinned to batch/job, JobSet, and TrainJob. Upgrade note: the quota CRs are helm post-install/post-upgrade hooks with a delete-and-recreate policy — quiesce queues before upgrading the bundle (Kueue's resource-in-use finalizer on an active ClusterQueue/ResourceFlavor blocks the delete and can wedge the upgrade), and re-apply tuned quotas afterwards since upgrades reset them to the shipped defaults. Uninstalling leaves the hook-created CRs behind; delete them manually when removing Kueue. Overlays that override the component's manifestFiles (replacing the default quota CRs) must also override its health check — the shipped check asserts the default CR names above. Kueue
kubeflow-trainer Kubeflow Training Operator for distributed training jobs (PyTorch, etc.). Manages multi-node training job lifecycle with JobSet integration. Kubeflow Trainer
mariadb-operator-crds Official MariaDB Operator CRDs. Declared in every Slurm recipe but installed only for accounting.mode: aicr-provided. MariaDB Operator
mariadb-operator Official MariaDB Operator controller, webhook, and certificate controller. AICR installs it only for accounting.mode: aicr-provided. MariaDB Operator
slurm-accounting-mariadb Installation-managed MariaDB instance whose initial database, all-privileges accounting user, and generated Secret reference are configured atomically on the MariaDB resource. Declared in every Slurm recipe and rendered only for accounting.mode: aicr-provided. MariaDB Cluster chart
slinky-slurm-operator-crds Custom Resource Definitions for the SchedMD Slinky Slurm operator. Installs the slinky.slurm.net CRDs (Controller, NodeSet, LoginSet, Accounting, RestApi, Token). Installed separately to support CRD lifecycle management. Slinky Slurm Operator
slinky-slurm-operator SchedMD Slinky Slurm operator and admission webhook. Manages the lifecycle of Slurm clusters declared via Slinky CRs (Controller, NodeSet, LoginSet, Accounting, RestApi, Token). AICR's system node-selector and toleration bundle flags apply to both deployments; affinity remains available through component values or typed overrides. Slinky Slurm Operator
slinky-slurm Slinky-managed Slurm cluster instance: Controller (slurmctld) + LoginSet (sackd/sshd) + NodeSet (slurmd) + RestApi (slurmrestd), with SlurmDBD derived from the recipe's typed accounting mode. Reconciled by slinky-slurm-operator. See Slurm Accounting, Slurm Enroot Configuration, and Slurm Shared Storage. Slinky Slurm Cluster Chart
slinky-topograph Slinky/Slurm-scoped instance of Topograph — queries cloud provider topology APIs (GCP, AWS, OCI …) to generate Slurm topology.conf, enabling topology-aware placement decisions in the Slinky-managed scheduler. Not installed by default; leaf overlays opt in by adding an explicit componentRef entry for slinky-topograph — the componentRef is what schedules the release; dependencyRefs alone does not install anything. That componentRef declares slinky-slurm as a dependencyRef to deploy after it: slinky-slurm renders and owns the slinky-slurm-config-extra ConfigMap (from its configFiles, mounted into slurmctld via the Controller CR's configFileRefs), and Topograph patches only that ConfigMap's topology.conf key on each sync, preserving the chart-owned cgroup.conf/gres.conf keys — Helm has to own the ConfigMap first. TopologyPlugin: topology/tree is set per-leaf via slinky-slurm's controller.extraConfMap. Includes the node-observer component, which watches the topograph API pod and regenerates topology on restarts or selected node/pod changes. Requires cloud provider IAM access (e.g. GCP roles/compute.viewer for Workload Identity). Topograph
nfd-ocp-olm OLM installer for Node Feature Discovery on OpenShift. Creates the OperatorGroup and Subscription resources that install NFD via the Operator Lifecycle Manager. Paired with nfd-ocp. OCP-specific. Node Feature Discovery (Certified)
nfd-ocp Node Feature Discovery CR for OpenShift. Configures NFD's operand (worker, topology updater) via a NodeFeatureDiscovery custom resource. Deployed after nfd-ocp-olm. OCP-specific. Node Feature Discovery
gpu-operator-ocp-olm OLM installer for the GPU Operator on OpenShift. Creates the OperatorGroup and Subscription resources that install the certified GPU Operator via the Operator Lifecycle Manager. Paired with gpu-operator-ocp. OCP-specific. NVIDIA GPU Operator (Certified)
gpu-operator-ocp GPU Operator ClusterPolicy CR for OpenShift. Configures the GPU Operator's runtime behavior (driver, toolkit, DCGM, device plugin, MIG manager) via a ClusterPolicy custom resource. Deployed after gpu-operator-ocp-olm. OCP-specific. NVIDIA GPU Operator
network-operator-ocp-olm OLM installer for the Network Operator on OpenShift. Creates the OperatorGroup and Subscription resources that install the certified Network Operator via the Operator Lifecycle Manager. Paired with network-operator-ocp. OCP-specific. NVIDIA Network Operator (Certified)
network-operator-ocp Network Operator NicClusterPolicy CR for OpenShift. Configures RDMA, MOFED driver, shared device plugin, and NV-IPAM via a NicClusterPolicy custom resource. Deployed after network-operator-ocp-olm. OCP-specific. NVIDIA Network Operator
cert-manager-ocp-olm OLM installer for cert-manager on OpenShift. Creates the OperatorGroup and Subscription resources that install the certified cert-manager Operator via the Operator Lifecycle Manager. Paired with cert-manager-ocp. OCP-specific. cert-manager (Certified)
cert-manager-ocp cert-manager CertManager CR for OpenShift. The operand Deployments (controller, cainjector, webhook) land in a hardcoded cert-manager namespace regardless of the operator's own namespace. Deployed after cert-manager-ocp-olm. OCP-specific. cert-manager
prometheus-adapter-ocp Prometheus Adapter for OpenShift. Reuses the same upstream chart as prometheus-adapter, pointed at OCP's built-in Thanos Querier instead of kube-prometheus-stack (which stays disabled on OCP). No certified OCP operator exists for this component. OCP-specific. prometheus-adapter
nvidia-dra-driver-gpu-ocp NVIDIA DRA GPU driver for OpenShift. Reuses the same upstream chart as nvidia-dra-driver-gpu, with an added SCC RoleBinding granting the kubelet-plugin DaemonSet the host device access OCP's default restricted-v2 SCC forbids. No certified OCP operator exists for this component. OCP-specific. Known limitation: some GPU-driver rollout protections and remedy hints do not yet cover the OCP aliases (gpu-operator-ocp, nvidia-dra-driver-gpu-ocp) — the deployer's stale-NVML migration wait/restart, driver-version annotation injection, and the driver-absent remedy's gpuoperator:/dradriver: override keys; tracked in #2136. NVIDIA DRA Driver
k8s-nim-operator-ocp NVIDIA NIM Operator for OpenShift. Reuses the same upstream chart as k8s-nim-operator, with OCP-specific RBAC. Requires cert-manager-ocp for admission-webhook TLS. OCP-specific. K8s NIM Operator

How Components Are Selected

Not every component appears in every recipe. The recipe engine selects components based on the overlay chain for your environment:

  • Base components (cert-manager, kube-prometheus-stack) appear in most recipes.
  • Cloud-specific components (aws-efa, aws-ebs-csi-driver) are added when the service matches. OCP recipes replace base components (gpu-operator, nfd, network-operator, cert-manager) with OLM+CR pairs where a certified operator exists (e.g., gpu-operator-ocp-olm + gpu-operator-ocp, cert-manager-ocp-olm + cert-manager-ocp). Components with no certified OCP operator (for example, prometheus-adapter, nvidia-dra-driver-gpu, k8s-nim-operator) instead reuse the same upstream Helm chart as their base component, with OCP-specific manifests (SCC RoleBindings, RBAC, CA bundle injection) layered on top. k8s-nim-operator-ocp is available on OCP via the ocp-inference-nim overlay and depends on cert-manager-ocp for webhook TLS.
  • Intent-specific components (agentgateway, agentgateway-crds) are added based on workload intent (e.g., inference recipes include the inference gateway).
  • Platform-specific components (slinky-slurm-operator, slinky-slurm, kubeflow-trainer, dynamo-platform) are added when the recipe selects a matching --platform. For --platform slurm, all three core Slinky pieces (slinky-slurm-operator-crds, slinky-slurm-operator, slinky-slurm) are declared inline per slurm leaf overlay — the same shape dynamo-platform uses across *-inference-dynamo leaves. IMEX-capable Slurm leaves attach a fixed ComputeDomain through slinky-slurm.preManifestFiles so slurmd pods can consume DRA-provisioned IMEX channels. Leaves that want the operator only inline the CRDs + operator and omit the slinky-slurm componentRef. For an end-to-end walkthrough (recipe → bundle → install → validate → srun smoke job on AKS, EKS, GKE, or Kind), see demos/cuj1-slinky-slurm.md.
  • Topology-aware optional components (slinky-topograph) are not installed by default. Opting in requires an explicit componentRef entry for slinky-topograph in the leaf overlay — the componentRef is what installs it; dependencyRefs alone does not. That componentRef declares slinky-slurm as a dependencyRef, so Topograph deploys after the Slurm cluster chart, which owns the ConfigMap Topograph patches. See the wiring example in the Recipe Development Guide.
  • Accelerator/OS-specific tuning (nodewright-customizations, nvidia-dra-driver-gpu) varies by hardware and OS combination.

NFD Topology Updater

Production GPU leaf recipes (H100, GB200, RTX Pro 6000 on EKS / AKS / GKE / OKE / LKE) enable the NFD Topology Updater. It publishes per-node NodeResourceTopology CRDs that describe NUMA zones, GPU-to-NUMA affinity, and NIC-to-NUMA affinity. Runtime consumers (NUMA-aware schedulers, debugging via kubectl get noderesourcetopologies) can read these CRDs without further configuration.

The Topology Updater requires the kubelet podResources gRPC socket. The KubeletPodResources feature gate has been on by default since Kubernetes 1.15 (Beta) and reached GA in Kubernetes 1.28; AICR's recipe constraints on the affected leaves require K8s ≥ 1.30 or higher, so this is satisfied in practice. Recipes targeting Kubernetes < 1.15 must enable the feature gate explicitly. Kind / KWOK simulated clusters do not run a real kubelet and therefore leave the Topology Updater disabled — kind-based recipes will not see NodeResourceTopology CRDs.

See the upstream Topology Updater docs for runtime consumer examples.

GPU Operator Driver Auto-Detect

When a recipe is resolved from a snapshot (aicr recipe --snapshot snap.yaml, or the ResolveRecipeFromSnapshot SDK entry point), AICR reads the sampled GPU node's driver-loaded measurement and, when the NVIDIA kernel module is already loaded, injects components.gpu-operator.overrides.driver.enabled=false into the resolved recipe. On recipes whose ADR-015 profile owns driver.enabled (the AKS family), the injector is subordinated: the profile fragment owns the path, the injector skips it without mutating (logging the skip), and the fragment's value is authoritative. Subordination follows path ownership, not mere profile presence — the GKE family is also profiled (gpuStack owns devicePlugin.enabled, not driver.enabled), so the injection/teardown discussion in this section applies to GKE-COS exactly as to unprofiled compositions (OKE, legacy AKS artifacts). The override lands at the top of the merge chain (base values.yaml → ValuesFile → Overrides), so the rendered Helm values a deployer installs carry driver.enabled: false regardless of what the resolved overlay's values file would default to. This prevents the GPU Operator from installing a second driver on top of one the platform has already provisioned. Explicit --set flags at bundle generation (aicr bundle --set gpuoperator:driver.enabled=true) retain higher precedence and can supersede the injection unless the path is profile-owned — on recipes carrying metadata.selectedProfile (the AKS family's gpuStack), a --set diverging from the selected value on an owned path fails closed at bundle time. --set is a bundle-time flag, not an aicr recipe flag.

Injection is gated on the resolved overlay already declaring driver.enabled=false in its merged base+valuesFile. That marker check inspects driver.enabled alone; the shipped preinstalled-driver overlays additionally carry coordinated ownership settings (AKS and OKE set toolkit.enabled=false; GKE-COS keeps the toolkit enabled with the COS-specific toolkit.installDir under the host-managed driver root) plus hostPaths.driverInstallDir, and the bundle-time CheckDriverOwnershipCoherence validation enforces full ownership coherence for any recipe. That scopes auto-detect to overlays like AKS, GKE-COS, and OKE where every dependent setting is already aligned. Bare EKS overlays lack the marker; the auto-detect skips them and logs a warning (gpu-operator driver auto-detect: pre-installed driver observed …) telling the operator to use a preinstalled-profile overlay rather than land a half-configured Operator (driver off, toolkit and gdrcopy still enabled with no operator-managed driver root). The case is still tracked as separate work:

  • EKS — GPU-optimized AMIs that ship an NVIDIA driver preinstalled on the AMI itself. Today this warns; a full preinstalled EKS overlay is tracked separately.

On preinstalled-driver overlays whose profile does not own driver.enabled (GKE-COS) or that carry no profile (OKE, legacy AKS artifacts), the injection is semantically idempotent — the rendered driver.enabled value is unchanged; the resolved recipe records the override explicitly (visible in aicr recipe -o recipe.yaml) so the reason for the value is auditable end-to-end. On profiled AKS there is no injection at all: the gpuStack fragment writes the value and the injector skips the owned path (see above). The AKS default is the AKS azure-managed profile (driver.enabled=false, toolkit.enabled=false, operator.runtimeClass=nvidia-container-runtime — the Azure default, where the node image preinstalls driver and toolkit); a --gpu-driver none pool selects the operator-managed profile value at recipe time (--profile gpuStack=operator-managed, see the GPU Operator-managed profile).

The inverse mismatch (no NVIDIA driver loaded on the sampled GPU node while the resolved overlay declares the preinstalled-driver profile) is handled differently depending on whether the family carries an ADR-015 configuration profile:

  • On AKS (profiled), a pool reading that mismatches the SELECTED value fails closed at resolution. Selection comes from --profile (or the azure-managed default); the K8s.aks-gpu-pools.gpu-driver reading then verifies it before any driver-state post-processing. Pools reading None fail the azure-managed default but qualify --profile gpuStack=operator-managed — rerun with that selection against the same snapshot; no pool change or recapture is needed. Mixed and Managed reject either selection naming the observed state — fix the pools and recapture; a missing reading (snapshot captured without --aks-gpu-pools) rejects with "reading unavailable" — recapture with the pool dump (see the AKS GPU Operator-managed profile).
  • On AKS (profiled), Install-mode pools whose sampled node has no driver loaded still enter the record-and-gate flow. Pool mode is the ownership contract, not live state: gpu-driver: Install satisfies the azure-managed constraint even while a failed AKS driver install or a mid-reimage node samples no loaded driver. Resolution succeeds, records metadata.gpuDriverState: absent, and the bundle-time CheckDriverOwnershipCoherence gate blocks aicr bundle with the AKS remedy (repair the pools and recapture, or switch pools to --gpu-driver none, recapture, and regenerate with --profile gpuStack=operator-managed). The driver-ownership paths are profile-owned, so the pre-profile per-path --set override tuple is rejected at bundle time.
  • On families without a profile (and legacy pre-profile AKS artifacts), every inverse mismatch takes that same warn-record-gate path: resolution logs a warning, records metadata.gpuDriverState: absent in the recipe, and the bundle-time CheckDriverOwnershipCoherence validation blocks aicr bundle (#1757) unless the values are flipped to operator-managed mode or the GPU pools are reprovisioned with the platform's default driver install and re-snapshotted. Ownership overrides are bundle-time flags there, so resolution itself cannot fail hard without cutting off the supported override path — bundle generation is the first point where the final effective values are known.

One per-OS limitation is intentionally out of the check's scope: the check verifies value coherence (driver ownership and driver-root lockstep), not per-OS install capability. On GKE COS node images a deliberate --set gpuoperator:driver.enabled=true clears the gate — the values are internally coherent — but the GPU Operator cannot install a driver on COS, so the deployment fails at deploy time, where the gpu-operator-health deployment-phase check is the backstop. The check's GKE remedy therefore points COS clusters at the GKE-managed driver install (gpu-driver-version) rather than the override tuple.

The policy is only-false: the auto-detect never forces driver.enabled=true, so recipes resolved without a snapshot (or targeting a node without a loaded driver) fall back to today's static defaults. Two operational consequences:

  • Criteria-only resolves (aicr recipe --service ... --accelerator ...) and no-cluster mode see zero behavior change — no snapshot, no override.
  • A stale snapshot from an older CLI that omits the driver-loaded reading is treated as unknown, not absent, so it cannot flip a hardened overlay.

Capture the snapshot BEFORE deploying the GPU Operator (unprofiled compositions; on profiled AKS the pool reading, not driver-loaded, gates resolution, and a post-deploy snapshot cannot flip an owned path). The driver-loaded reading is installer-agnostic — it reports whether the nvidia kernel module is currently loaded, not who loaded it. A snapshot taken after a prior AICR deploy has run the operator's driver container will still report driver-loaded=true, and a re-resolve from that post-deploy snapshot would flip a working overlay toward driver.enabled=false, tearing the operator-managed driver DaemonSet down and leaving new or rebooted GPU nodes driverless. AICR emits a gpu-operator driver auto-detect: driver-loaded=true AND a ClusterPolicy is already present… warning when both signals appear together in the same snapshot, but the guard is observability, not prevention: a pre-deploy snapshot is the intended workflow.

The signal is a single-node sample: the snapshotter Job runs on one nvidia.com/gpu.present=true node, so its driver-loaded reading is representative only when every GPU pool is in the same driver state. Mixed-pool clusters (some nodes with a preinstalled driver, some without) are out of scope for the auto-detect and tracked in #464; AICR emits a topology reports non-uniform GPU labels… warning when the snapshot's node-topology labels indicate divergent GPU nodes so the fail-direction (some non-preinstalled pools may come up driverless) is at least observable.

To see exactly which components appear in a given recipe, generate one:

aicr recipe --service eks --accelerator h100 --os ubuntu --intent training -o recipe.yaml

The output lists every component with its pinned version and configuration values.

GKE Device-Plugin Ownership

Device-plugin ownership is a configuration profile. The GKE recipes declare an ADR-015 gpuStack profile with two qualified values, selected at recipe generation and recorded in metadata.selectedProfile:

  • gke-default (the default) — GKE's managed device plugin is the nvidia.com/gpu advertiser (recorded as advertiser: external), and the recipe disables the GPU Operator's plugin (devicePlugin.enabled: false, profile-owned). Its constraint requires that no GPU node — identified by its cloud.google.com/gke-accelerator label — carries the opt-out label gke-no-default-nvidia-gpu-device-plugin. This is the default GKE cluster shape: create GPU node pools normally (with gpu-driver-version=default or latest for GKE's managed driver install) and no further cluster setup is required.
  • driver-installer (aicr recipe ... --profile gpuStack=driver-installer) — the GPU Operator's device plugin is the sole advertiser (devicePlugin.enabled: true, profile-owned), and the constraint inverts: every GPU node must carry gke-no-default-nvidia-gpu-device-plugin=true. This value has real cluster prerequisites — the label on every GPU pool at creation plus Google's standalone nvidia-driver-installer DaemonSet, with pools created gpu-driver-version=disabled.

For the end-to-end setup flow (snapshot → recipe → validate), the qualification and selection-vs-verification matrices, and troubleshooting, see GKE GPU Setup.

Why exactly one advertiser: two plugins registering nvidia.com/gpu on one node is not a benign overlap. Kubelet's device manager keys its endpoint and device inventory by resource name, so competing registrations and ListAndWatch updates replace each other. Ownership becomes nondeterministic, and one plugin's device IDs (GKE uses nvidia0-style names, NVIDIA uses GPU UUIDs) can reach the other plugin's Allocate. Expect intermittent allocation and runtime failures.

Driver-installer cluster setup. The opt-out label forfeits GKE's managed driver install: the managed install (gpu-driver-version=default/latest) is finalized by an init container of the same kube-system DaemonSet the label disables, so a labeled pool paired with gpu-driver-version=default comes up driverless — never combine the label with the managed driver install. Pools for the driver-installer value must instead be created with gpu-driver-version=disabled, with driver provisioning supplied by Google's standalone nvidia-driver-installer DaemonSet applied to the cluster. (AICR's GKE-COS overlays keep driver.enabled: false in either mode — the GPU Operator cannot install a driver on COS node images.) The operational procedures live in GKE GPU Setup: pool creation with the label, retrofitting an existing pool (a --node-labels update replaces the pool's full label set), verifying the handoff after the asynchronous plugin eviction, and the three driver-installer settings and what each covers.

aicr validate enforces the selected value deterministically, before any phase runs. The selected value's NodeTopology.gpu-nodes.label constraint (#1755) is verified against the snapshot at recipe generation and re-evaluated by the validate readiness pre-flight. The check fails closed: labels contradicting the selected value, mixed labels, an empty GPU-node set, and readings that --max-nodes-per-entry actually truncated (a cap larger than the node count truncates nothing and validates normally) all fail with exit 2 and remediation text pointing back at this section, before any check Jobs deploy. See Validation for the readiness-gate mechanics.

The profile also locks the ownership tuple: devicePlugin.enabled is profile-owned, and because both values govern advertisement, the #1327 allocation-policy paths (devicePlugin.enabled, DRA resources.gpus.enabled / gpuResourcesEnabledOverride) are closure-locked — a bundle- or install-time override diverging at any of them is rejected rather than warned. Switching modes is a recipe-generation decision (--profile), never a --set.

The selected constraint is the only deterministic detection point. aicr bundle is offline by design and cannot read node labels. The operator-health deployment check passes under the conflict because it verifies only that GPU Operator controller pods are Running — it never inspects the device plugin. Allocation probes such as check-nvidia-smi schedule a pod requesting nvidia.com/gpu on each schedulable GPU node, but skip cordoned nodes and skip entirely when any schedulable GPU node is busy; when they do run, they may fail nondeterministically without identifying the missing label as the cause.

See GKE's GPU node-pool guide for the authoritative pool-creation and driver-installer procedures. The NVIDIA GPU Operator GKE guide documents the same gpu-driver-version=disabled + driver-installer DaemonSet combination the driver-installer value builds on. Shipping the installer as an AICR-managed component (the ADR-015 operator-selfdriver direction) is tracked separately in #1716.

NVSentinel on Provider-Installed-Driver Platforms

The recipes configure NVSentinel for you on every platform that needs it. This section explains what they set and why, so the values are recognizable in a generated bundle and the failure signatures are diagnosable if they ever reappear.

Symptom. NVSentinel's metadata-collector and both syslog-health-monitor DaemonSets report 0 desired pods and never schedule, while everything else looks fine (#2175).

This is easy to miss. A DaemonSet whose node selector matches no node is not unhealthy — it reports no error and emits no event — and gpu-health-monitor keeps running normally because it selects on the DCGM label instead. The stack presents as fully rolled out.

Cause. Those three DaemonSets select on the node label nvsentinel.dgxc.nvidia.com/driver.installed, which the NVSentinel labeler applies by watching for a GPU driver pod. Where the driver ships in the node image and no driver pod exists, the labeler never applies the label — so labeler.assumeDriverInstalled must be set to skip driver-pod detection and label GPU nodes unconditionally.

The recipes now carry that value wherever it is needed (#2181):

Platform Driver pod the labeler can observe labeler.assumeDriverInstalled Supplied by
AKS gpuStack=azure-managed (default) none — driver is in the node image true the gpuStack profile
AKS gpuStack=operator-managed the operator's driver pod false the gpuStack profile
GKE COS gpuStack=gke-default (default) none the labeler can observe — the driver is finalized by an init container of GKE's kube-system DaemonSet true the gpuStack profile
GKE COS gpuStack=driver-installer Google's standalone nvidia-driver-installer DaemonSet false the gpuStack profile
OKE none — driver is in the node image true the overlay (OKE has no profile)
EKS the operator's driver pod unset (chart default false) —
Kind (nvkind) none — driver is host-installed true the overlay (Kind has no profile)

The explicit false on the operator-managed variants is deliberate rather than redundant: it keeps the path profile-owned, so it cannot be flipped into an unsafe hybrid later. Do not assume a preinstalled driver where the GPU Operator installs one — skipping detection there would keep the label applied across an unloaded or unhealthy driver.

NVSentinel is mandatory on the profiled families. Because the AKS and GKE-COS gpuStack profiles name nvsentinel, its presence is profile-owned: --set nv-sentinel:enabled=false and a bundlers= list that omits it are both rejected on those platforms. That is intended — NVSentinel is a required component for these deployments. It remains optional on platforms with no gpuStack profile, such as OKE and EKS.

Only AKS and GKE-COS get the install-time profile lock; OKE and Kind set the value at overlay level, so a bundle-time or declared-dynamic change is still rejected by the gate below, but a manual post-generation edit to the rendered Helm values is not.

If you do need to set it yourself on an unlisted platform, it is an ordinary override:

aicr bundle -r recipe.yaml \
  --set nv-sentinel:labeler.assumeDriverInstalled=true \
  -o ./bundles

The value renders the labeler's --assume-driver-installed argument. That is the chart-level automation of the Manual Labeling Procedure documented in NVSentinel design 018. Upstream has settled the design question: it is the recommended, permanent mechanism for host-installed drivers — no automatic detection fallback will be added (NVIDIA/NVSentinel#1583).

Do not label the nodes by hand. kubectl label node <node> nvsentinel.dgxc.nvidia.com/driver.installed=true takes effect immediately — the DaemonSets roll out — and then silently reverts. With no driver pod to observe, the labeler computes an empty desired value and removes the label on its next reconcile. Because upstream design 018 documents manual labeling as the procedure for this case, an operator following it will see it work and later find the DaemonSets back at 0 desired.

aicr bundle rejects a configuration that would reintroduce the gap. A recipe that includes nvsentinel with no observable driver pod and without labeler.assumeDriverInstalled fails bundle generation with a blocking error (CheckNVSentinelDriverLabelDetectable). On the profiled families the value is also profile-owned, so a --set diverging from the selected gpuStack value is rejected before the gate even runs.

One exception, for completeness: the gate is silent if you disable both label consumers (--set nv-sentinel:global.metadataCollector.enabled=false and --set nv-sentinel:global.syslogHealthMonitor.enabled=false). Nothing then reads the label, so there is no gap to reintroduce. Disabling only one still requires the value, and no recipe disables either — both default to enabled, and the gate fails closed when it cannot prove otherwise. Everything above therefore applies to every shipped configuration.

A second, distinct failure on AKS azure-managed: RuntimeClass mismatch. The metadata-collector DaemonSet requests a RuntimeClass by name, and the GPU Operator's ClusterPolicy controller names that object after operator.runtimeClass. The AKS azure-managed profile retargets it to nvidia-container-runtime, so a metadata-collector left on its chart default nvidia finds no such RuntimeClass and the API server rejects every pod at admission (pod rejected: RuntimeClass "nvidia" not found — #2176).

The two signatures differ: the label gap above shows 0 DESIRED pods (never scheduled, no error, no event); the RuntimeClass mismatch shows N desired / 0 CREATED with a FailedCreate event on the DaemonSet and no pod object to describe.

The AKS gpuStack profile now owns both names — gpu-operator.operator.runtimeClass and nvsentinel.metadata-collector.runtimeClassName — in the same profile value, so they agree by construction under either value and no override is needed:

AKS profile value operator.runtimeClass metadata-collector.runtimeClassName
azure-managed (default) nvidia-container-runtime nvidia-container-runtime
operator-managed nvidia nvidia

Every other platform leaves operator.runtimeClass at the shared chart default nvidia, so neither side needs a value. CheckNVSentinelRuntimeClassCoherence still compares the two resolved names as defense in depth, treating either side unset as nvidia.

An AKS bundle therefore needs no NVSentinel overrides at all — only the keyed toleration AKS requires independently of NVSentinel (bundling an AKS recipe without one is itself a blocking error, CheckWildcardAcceleratedToleration):

aicr bundle -r recipe.yaml \
  --accelerated-node-toleration nvidia.com/gpu:NoSchedule \
  -o ./bundles

See AKS GPU Setup for the per-profile guidance.

NIM Workload Credentials

AICR installs the k8s-nim-operator only. It does not create a NIMService and does not create credentials — deploying a workload is an operator step, and there are two ways to supply the model.

Whichever path you take, spec.authSecret is required by the NIMService schema and must name an existing secret in the workload's namespace. spec.image.pullSecrets is optional; the image block requires only repository and tag.

NGC path

Model artifacts come from NGC, so the secret must carry a valid NGC_API_KEY:

kubectl create secret generic ngc-api-secret \
  --from-literal=NGC_API_KEY="$NGC_API_KEY" -n nim-workload

Add a docker-registry secret and reference it from image.pullSecrets when the image lives in a private or authenticated registry path. See demos/workloads/inference/nimservice-llama-3-2-1b.yaml for a complete example.

Credential-free path (Hugging Face)

Setting NIM_MODEL_NAME to an hf:// URI puts the operator on its Hugging Face path, where it marks NGC_API_KEY optional and injects HF_TOKEN from the same authSecret. With an ungated Hugging Face model and a NIM image that pulls anonymously, no NGC credential is needed anywhere:

kubectl create secret generic hf-secret --from-literal=HF_TOKEN="" -n nim-workload
spec:
  authSecret: hf-secret                                  # holds only HF_TOKEN
  image:
    repository: nvcr.io/nim/meta/llama-3.1-8b-instruct   # pulls anonymously; no pullSecrets
    tag: "2.0.10"                                        # pin a version; avoid the mutable latest
  env:
    - name: NIM_MODEL_NAME
      value: hf://Qwen/Qwen3-0.6B                        # ungated model
    - name: NIM_SERVED_MODEL_NAME
      value: Qwen/Qwen3-0.6B                             # the OpenAI-API `model` id

The HF_TOKEN key must exist in the secret — that reference is not optional — but an empty value is sufficient for an ungated model. A gated Hugging Face repository needs a real token here.

Model-specific NIM repositories (for example nim/meta/llama-3.1-8b-instruct) serve anonymous registry tokens; the generic Multi-LLM image nim/nvidia/llm-nim does not and requires a pull secret.

Note that pairing a model-specific image with an unrelated hf:// model is off-label: the container runs its own profile against the downloaded weights. It works, but nim/nvidia/llm-nim is the image intended for arbitrary Hugging Face models — and because that repository is gated, choosing it trades the credential-free property for a supported pairing. Pin an image tag rather than latest so the pairing you validated is the one you ship.

See demos/workloads/inference/nimservice-hf-nocred.yaml for a complete example.

Inference Gateway Network Exposure

Inference recipes include the agentgateway component, which deploys an inference-gateway Gateway. The agentgateway controller materializes that Gateway into a Service of type LoadBalancer, so on every cloud the platform provisions a load balancer for the (plaintext HTTP, unauthenticated) inference endpoint. Left unrestricted that load balancer is internet-facing, so aicr bundle scopes it to private networks by default — the opt-in path for public exposure and the validation behavior are described below.

aicr bundle is private by default: when a bundle includes agentgateway and agentgateway.allowedSourceRanges is empty or unset, the bundler injects the private RFC1918 ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) into the generated Service's spec.loadBalancerSourceRanges. The deployed gateway is therefore reachable from inside the cluster/VPC (and from privately-routed peers) but denied to the public internet — it is never emitted open to 0.0.0.0/0 without an explicit opt-in. (Kubernetes treats an empty loadBalancerSourceRanges as allow-all, so a safe default has to be a real list, not an empty one.) A bundle note records when the default was applied.

To restrict it to specific trusted networks instead — for example to allow a corporate VPN, which egresses from a public IP and is therefore not covered by the RFC1918 default — set agentgateway.allowedSourceRanges to a list of CIDR (Classless Inter-Domain Routing) blocks. The values replace the default and are rendered into the generated Service's spec.loadBalancerSourceRanges, which the AWS, GCP, Azure, and OCI cloud load balancers all honor — so one setting locks the gateway down on every platform.

Do not use plain --set for this key. --set agentgateway:allowedSourceRanges=<cidr> writes loadBalancerSourceRanges as a bare string instead of a list; the bundler rejects that with ErrCodeInvalidRequest (a bare scalar would render a type-invalid Service). Use the list-aware --set-json / --set-file flags from the CLI:

aicr bundle -r recipe.yaml \
  --set-json agentgateway:allowedSourceRanges='["216.228.127.128/30"]'

or scope the gateway through a recipe overlay or componentRef override:

componentRefs:
  - name: agentgateway
    type: Helm
    overrides:
      allowedSourceRanges:
        - 216.228.127.128/30   # e.g. corporate egress

The default is the generic RFC1918 private set rather than a fixed customer CIDR: a baked-in specific range would firewall every downstream deployment to one network and lock other operators out of their own gateway. RFC1918 is universal — it trusts only privately-routed traffic — so it is a safe default that still denies the public internet. Override it whenever you need to admit a specific public client.

If public exposure is genuinely intended, opt in explicitly with an any-source CIDR — bundle generation then succeeds but logs a loud warning that the gateway is open to the entire internet:

aicr bundle -r recipe.yaml \
  --set-json agentgateway:allowedSourceRanges='["0.0.0.0/0"]'

This setting filters by source IP only; it does not add TLS or authentication to the gateway listener.

Exposure guardrails

AICR enforces and surfaces inference-gateway exposure in two places:

  • Bundle-time private-by-default. When a bundle includes agentgateway and allowedSourceRanges is empty/unset, aicr bundle injects the RFC1918 private ranges so the deployed gateway denies the public internet, and records a bundle note. An invalid value (a bare-string --set, a non-list, an unparseable CIDR, or a non-canonical CIDR such as 1.2.3.4/24 that Kubernetes' strict validation would reject at apply time) is rejected with ErrCodeInvalidRequest. A scoped list passes silently; an explicit any-source CIDR (0.0.0.0/0 or ::/0) passes with a loud warning as a deliberate opt-in. See #1373.
  • Conformance check. The inference-gateway conformance check (run during aicr validate --phase conformance on a live cluster) inspects the gateway's LoadBalancer Service and records its exposure as evidence — the source ranges if scoped, or an explicit "open to 0.0.0.0/0" finding if not. Set AICR_REQUIRE_SCOPED_INFERENCE_GATEWAY=true on the validator environment to escalate an open gateway to a check failure.

k8s-aibom Runtime Inventory

AICR qualifies k8s-aibom v1.2.0 as an optional Helm component. It is not in the base, a mixin, or any stock overlay. To enable it, add this reference to a custom or external overlay and keep that overlay's criteria as narrow as the intended rollout:

spec:
  componentRefs:
    - name: k8s-aibom
      type: Helm
      valuesFile: components/k8s-aibom/values.yaml

A broad criteria overlay affects every matching recipe. In particular, intent: any is universal across intents; do not use it unless that injection is deliberate. The in-tree recipes/overlays/monitoring-hpa.yaml overlay shows that broad reach: its criteria: intent: any attaches to every matching intent. See Recipe Development for external data and criteria composition.

The qualified artifacts are source tag v1.2.0 at commit 4aa7638b08ab9927bfa8df85c46c80234b9996f9, OCI chart oci://ghcr.io/googlecloudplatform/charts/k8s-aibom:1.2.0, and the controller image pinned by digest in the component values. Upstream documents Kubernetes 1.27 through 1.35 support. AICR also observed the dedicated integration test passing on its Kind 1.36.1 node image; that is qualification evidence, not an extension of upstream's support statement.

Health and readiness

The deployment-phase check requires the controller Deployment to have at least one desired replica and all desired replicas available. It also requires the cluster-scoped AIBOMControllerConfig/default to report a current Ready=True condition: both top-level status and the condition must have observed the object's current generation. Missing or stale resources fail closed. Zero AIBOM objects is healthy before any namespace opts in.

readiness.strictConfig is enabled, so invalid new configuration cannot silently replace the controller's last-known-good configuration while the pod continues to report ready.

Security, privacy, and retention

Namespace discovery requires the label aibom.k8saibom.dev/enabled=true. AICR does not apply it. No external sink, endpoint, credential, or Secret access is configured by default. BOMs up to 262144 bytes are stored inline in AIBOM.status; larger output is summarized and marked truncated when no sink is configured. Inline status consumes etcd storage, so workload count and document size are part of the cluster control plane footprint. The controller runs non-root with a read-only root filesystem, RuntimeDefault seccomp, no privilege escalation, and no Linux capabilities.

The namespace label limits which workloads produce AIBOMs, not informer read scope. The controller still reads workload and pod specifications cluster-wide—including image references, arguments, and inline environment values—into memory. With the default empty sink list that data does not leave the cluster, but cluster-wide visibility remains part of the privacy boundary. The controller is not read-only: its bounded RBAC permits writes to its own AIBOM API resources and required status subresources, configuration status, and Kubernetes Events. It has no Secret access while sinks are disabled.

An AIBOM is owned by its top-level workload and is garbage-collected when that owner is deleted. Helm does not delete CRDs from a chart's crds/ directory on uninstall; consequently AIBOM resources for owners that still exist may remain after controller removal. External sinks, if an operator configures one, have their own retention policy outside AICR.

Upgrade, uninstall, and troubleshooting

Upgrade the component by qualifying a new chart and image together, then regenerate the custom recipe and bundle. Do not change only the controller image: chart, CRDs, status API, and image are one qualified set. Quiesce configuration changes during rollback and confirm that AIBOMControllerConfig/default returns to a current Ready=True state.

Apply CRDs before the bundle upgrade. The chart ships its CRDs under crds/. Helm installs that directory on first install and never touches it again on upgrade, so a chart bump whose CRDs changed leaves the previous schema in place and the API server silently prunes the new controller's writes to added fields. Apply the CRDs from the exact qualified chart first, then upgrade:

helm show crds oci://ghcr.io/googlecloudplatform/charts/k8s-aibom \
  --version <qualified-version> | kubectl apply --server-side -f -

Use --server-side because the CRDs exceed the annotation size limit that client-side apply depends on.

Which deployers need that step differs, so check yours:

Deployer CRD behavior on upgrade Pre-upgrade step needed
helm helm upgrade skips crds/ Yes
helmfile helmfile apply upgrades through Helm, so it also skips crds/ Yes
flux The generated HelmRelease sets no spec.upgrade.crds, and the helm-controller default is Skip Yes
argocd, argocd-helm Argo CD renders the chart with CRDs included and applies them as ordinary manifests each sync No

Flux can be made to handle this itself by setting spec.upgrade.crds: CreateReplace on the HelmRelease, but AICR does not generate that field for any component today, and CreateReplace is a cluster-wide behavior change that should be decided per chart rather than assumed. Until then, treat Flux like helm.

Uninstall in this order. Removing the component from the overlay and applying a regenerated bundle does not remove the previously installed release: the helm and helmfile deployers install releases by name, and a release the new bundle no longer mentions is simply left alone. Skipping the explicit uninstall leaves the controller running while the next step deletes the CRs and CRDs underneath it, so it reconciles against resources that are disappearing.

  1. Remove the component reference from the custom overlay and regenerate the recipe and bundle.
  2. Uninstall the release with the deployer-appropriate procedure in Bundle Uninstall — helm uninstall k8s-aibom -n k8s-aibom-system for the helm deployer, helmfile destroy for Helmfile bundles, deleting the owning Application for Argo CD, and the HelmRelease for Flux. Confirm the controller Deployment is gone before continuing.
  3. Only then delete retained AIBOMs and, last, the CRDs.

Deleting the CRDs cascades to every AIBOM stored cluster-wide, including any belonging to a namespace or release you did not intend to touch. Enumerate before deleting rather than passing --all:

# Review what exists and who owns it; delete only what this release should own.
kubectl get aiboms.aibom.k8saibom.dev --all-namespaces \
  -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].name
kubectl -n <namespace> delete aiboms.aibom.k8saibom.dev <name>

# CRDs last, and only once no other release uses them — deletion removes every
# stored custom resource of these kinds cluster-wide.
kubectl delete crd \
  aiboms.aibom.k8saibom.dev \
  aibomcontrollerconfigs.aibom.k8saibom.dev

If health validation fails, inspect the Deployment and configuration before looking for AIBOMs:

kubectl rollout status deployment/k8s-aibom -n k8s-aibom-system
kubectl get aibomcontrollerconfig default -o yaml
kubectl logs deployment/k8s-aibom -n k8s-aibom-system

A current Ready=False or stale observedGeneration means the active configuration was not accepted. If the controller is healthy but produces no inventory, verify the namespace label and the workload kind. A truncated BOM with no sink is expected once its canonical document exceeds the inline threshold; configure retention and credentials explicitly before enabling an external sink.

Adding Components

New components are added declaratively in recipes/registry.yaml — no Go code required. See the Contributing Guide and Components docs for details.

Upgrade Notes

Migration steps when upgrading from a prior AICR-generated bundle to a newer one that changes how a component delivers its Kubernetes resources.

A generated recipe is a point-in-time artifact of the AICR binary that produced it: the embedded registry, overlays, manifest paths, and chart pins are part of that binary's surface. When upgrading AICR, regenerate the recipe from scratch with the new binary (aicr recipe ...) before re-bundling. aicr bundle --recipe <old-file> against a newer binary may fail if the saved recipe references manifest paths the new release has moved or removed (see Bundle Generation Fails for the specific error).

gpu-operator: dcgm-exporter ConfigMap moved into the main release

Earlier bundles shipped the dcgm-exporter ConfigMap as a post-manifest in a separate Helm release named gpu-operator-post. The in-cluster ConfigMap therefore carries ownership annotations pointing at that release:

meta.helm.sh/release-name: gpu-operator-post
meta.helm.sh/release-namespace: gpu-operator

Newer bundles render the ConfigMap directly from the main gpu-operator chart's dcgmExporter.config.data values. On upgrade, Helm 3 refuses to claim the existing ConfigMap because its annotations point at a different release:

Error: ConfigMap "dcgm-exporter" in namespace "gpu-operator" exists and cannot be
imported into the current release: invalid ownership metadata; annotation
validation error: key "meta.helm.sh/release-name" must equal "gpu-operator":
current value is "gpu-operator-post"

Fresh installs are not affected. To migrate an existing cluster, remove the stale gpu-operator-post release before applying the new bundle.

Raw Helm (per-component bundle / deploy.sh):

helm uninstall gpu-operator-post --namespace gpu-operator

helm uninstall removes the ConfigMap it owns; the next gpu-operator upgrade re-creates it from values.

Helmfile — the new bundle no longer references gpu-operator-post, so helmfile apply will not prune it on its own. Run the helm uninstall above first, then helmfile apply.

Argo CD — delete the stale Application (it will not self-prune unless an ApplicationSet was managing it), then sync the updated gpu-operator application:

argocd app delete gpu-operator-post --cascade

Flux — delete the stale HelmRelease so Flux uninstalls the release and removes the ConfigMap, then reconcile the updated gpu-operator HelmRelease. The example below assumes the Flux control plane runs in flux-system; substitute the namespace where your Flux installation lives:

kubectl delete helmrelease gpu-operator-post --namespace flux-system

After migration, confirm the ConfigMap is owned by the gpu-operator release:

kubectl get configmap dcgm-exporter -n gpu-operator \
  -o jsonpath='{.metadata.annotations.meta\.helm\.sh/release-name}'
# Expected: gpu-operator