Audit and upgrade pinned component versions across the recipe catalog

[yuanchen8911]

Problem

Component versions across recipes/ are pinned individually and bumped
opportunistically, usually when a specific recipe needs something. There is no
periodic sweep that asks whether the catalog as a whole is current. The result
is drift that only surfaces when a user hits it — an old GPU driver against a
new CUDA workload, a GPU Operator behind the chart its values assume, or a
component pinned to a release that upstream has since superseded.

recipes/registry.yaml carries 35 defaultVersion pins today. The ones that
matter most for GPU correctness:

Component	Pinned	Where
GPU driver	580.173.02	recipes/components/gpu-operator/values.yaml
gpu-operator	v26.3.3	recipes/registry.yaml
nvidia-dra-driver-gpu	0.4.1	recipes/registry.yaml
nvidia-dra-driver-gpu-ocp	0.4.1	recipes/registry.yaml
network-operator	26.4.1	recipes/registry.yaml

Scope

Audit every pinned component version in the catalog against its current
upstream release, then bump what should be bumped and record why anything is
deliberately held back. Concretely:

• Every defaultVersion in recipes/registry.yaml.
• Version pins carried in overlays and mixins, which override the registry.
• The GPU driver version in recipes/components/gpu-operator/values.yaml.
• Image tags pinned inside component values and manifests.

For each, one of three outcomes: bumped, held with a recorded reason, or filed
as its own issue because the bump is not mechanical.

The DRA driver is expected to fold into GPU Operator

nvidia-dra-driver-gpu is currently a separate component (0.4.1, chart
oci://registry.k8s.io/dra-driver-nvidia/charts/dra-driver-nvidia-gpu), with a
second OCP-specific variant. Upstream is expected to integrate the DRA driver
into the GPU Operator.

When that lands, the sweep is not a version bump — it is a component
consolidation: the standalone component and its OCP sibling would be removed
or become values-gated inside gpu-operator, every overlay referencing them
would change, and the health checks and validators that assume a standalone
DaemonSet would need revisiting. That should be its own issue with its own
migration plan, not folded into a routine bump. Part of this issue is
determining whether it has landed in a release AICR can adopt, and filing
that follow-up if so.

Why this is not covered by existing automation

Renovate manages some images but deliberately excludes others. Vendor-coupled
stacks are excluded on purpose and documented as "a deliberate, human-driven
change" (.github/renovate.json5) — the EFA device plugin and the GKE TCPXO
stack among them. Those are exactly the pins that go stale silently, because
nothing prompts a human to look.

The BOM has partial coverage: TestCommittedBOMVersionsMatchRegistry fails CI
when a pinned version drifts from the registry, and the weekly BOM-refresh
workflow catches rendered-image drift inside charts. Neither answers "is this
pin still the version we want?" — they only enforce internal consistency.

Constraints

• A driver bump is not free on every family. On GKE the driver is
  GKE-managed (driver.enabled: false in the gke-cos values), so the version
  comes from the node pool or the driver-installer component, not from
  values.yaml. On GKE COS a driver version must additionally be COS-qualified
  for the GPU type. See feat(recipes)!: GKE bundle-installer replaces driver-installer #2360, which reworks that path.
• Coupled pairs must move together. Several pins are one half of a vendor
  pair; bumping either half alone creates an unsupported combination. gke-nccl-tcpxo: installer/sidecar version pair has drifted and is not enforced #2378
  records one that already drifted.
• Performance floors are calibrated against specific stacks. A driver or
  operator bump can shift measured bandwidth and throughput, so the validation
  gates need a look after any significant move.

Acceptance criteria

• Every pinned component version is compared against its current upstream release, with the result recorded
• GPU driver, gpu-operator, nvidia-dra-driver-gpu, and network-operator are explicitly addressed
• Each pin is bumped, or held with a stated reason
• Bumps that are not mechanical are split into their own issues rather than bundled
• make bom-docs is run and the regenerated BOM committed with any chart-touching change
• The DRA-into-GPU-Operator status is determined, and a migration issue filed if it has landed

Related

• feat(recipes)!: GKE bundle-installer replaces driver-installer #2360 — GKE driver-installer rework; changes how the driver version is set on that path
• gke-nccl-tcpxo: installer/sidecar version pair has drifted and is not enforced #2378 — a coupled vendor pair that drifted, illustrating the failure mode

[yuanchen8911]

Add to the audit: the Dynamo vLLM worker runtime

vllm-runtime is a pinned runtime version that the audit above does not list, because it lives in Go rather than in registry.yaml:

validators/performance/model_cache.go                                  cacheWorkerImage = ...vllm-runtime:1.2.1
validators/performance/testdata/inference/dynamo-deployment.yaml       (2 refs)
validators/performance/testdata/inference/dynamo-deployment-gateway-epp.yaml  (2 refs)

The three are kept in sync by TestCacheWorkerImageMatchesTemplate. The current pin ships CUDA 12.9.

What is available

Tag	CUDA	Dynamo version	amd64 / arm64
1.2.1 (current)	12.9.1	1.2.1	both
1.2.1-cuda13	13.0.2	1.2.1	both
1.3.1, 1.4.0, 1.4.1	13.0.2	newer	both

Version coupling constrains the choice

The worker image is versioned with the dynamo-platform chart, pinned at 1.2.1 in registry.yaml and deployed as dynamo-platform-1.2.1. Moving to 1.4.x means bumping the chart in lockstep — a user-facing component change touching the BOM and the dynamo-platform health check. 1.2.1-cuda13 avoids that entirely: same Dynamo version, CUDA 13 base.

Measured: no performance justification either way

Tested directly on GB300 (nhensley-gb300, 2x p6e-gb300r.36xlarge, Qwen/Qwen3-8B, 4 workers):

Image	throughput	TTFT p99
1.2.1 (CUDA 12.9)	20,769 tok/s	33,320 ms
1.2.1-cuda13 (CUDA 13.0.2)	21,100 tok/s	33,362 ms

Within 1.6% and 0.1%. The poor numbers in both cases turned out to be an unrelated benchmarking issue (running past the accelerator's saturation knee), not the runtime — at a sub-knee concurrency the same CUDA 12.9 image delivers 84,788 tok/s at 677 ms on GB300.

So there is no measured benefit to the CUDA 13 bump today. It belongs here as routine currency maintenance, not as a fix for anything.

Blast radius when it is picked up

The pin drives inference-perf for all nine Dynamo leaves — b200-gke-cos (x2), gb200-eks, gb200-oke, gb300-eks, h100-aks, h100-eks, h100-gke-cos, rtx-pro-6000-eks. Each carries inference-throughput / inference-ttft-p99 floors measured on 1.2.1, so a bump needs those re-measured per leaf. Nightly UAT covers only GKE/H100 and AKS/H100 in practice, so the rest would need manual runs.

Two things that make it lower risk than it sounds: every cluster already runs a CUDA 13-capable driver (580.x, verified on GB300 and AKS; GKE COS m125 ships 580.159.04), and 1.4.1 is smaller than 1.2.1 (9.5 GB vs 12.9 GB compressed, amd64), so the UAT cold-pull budgets improve rather than degrade.

Separate upstream gap, noted for the record

Neither image ships sm_103 cubins:

1.2.1         _C.abi3.so: sm_52 sm_80 sm_87 sm_89 sm_90 sm_90a sm_100 sm_100a sm_120 sm_120a
1.2.1-cuda13  _C.abi3.so: sm_75 sm_80 sm_87 sm_89 sm_90 sm_90a sm_100 sm_110 sm_120

GB300 is compute capability 10.3, so it runs via PTX JIT forward-compat from sm_100. That works — GB300 still outperforms the GB200 reference — but there may be headroom once upstream builds sm_103 kernels. Nothing actionable in this repo; upstream image matrix only.