Merge branch 'main' into fix/fern-mdx-sanitization

Author: pdmack

.github/RENOVATE.md

@@ -9,7 +9,7 @@ Self-hosted Renovate keeps the project's dependencies up to date across `go.mod`
 
 - Configuration: [`.github/renovate.json5`](renovate.json5)
 - Workflow: [`.github/workflows/renovate.yaml`](workflows/renovate.yaml)
-- Companion script: [`tools/update-chainsaw-checksums`](../tools/update-chainsaw-checksums)
+- Companion scripts: [`tools/update-chainsaw-checksums`](../tools/update-chainsaw-checksums), [`tools/update-helmfile-checksums`](../tools/update-helmfile-checksums), [`tools/update-helm-diff-checksums`](../tools/update-helm-diff-checksums)
 
 Policy choices (schedule, cooldown, auto-merge scope, group consolidation) are documented inline in `renovate.json5`. This doc covers what's covered, how to extend coverage, and the known gotchas.
 
@@ -25,6 +25,8 @@ Policy choices (schedule, cooldown, auto-merge scope, group consolidation) are d
 | `.settings.yaml` (28 tool entries) | custom regex manager (`# renovate:` annotations) |
 | `.settings.yaml` `nvkind` SHA | dedicated git-refs digest customManager (`# renovate-digest:`) |
 | `.settings.yaml` `chainsaw_checksums` | `postUpgradeTasks` → `tools/update-chainsaw-checksums` |
+| `.settings.yaml` `helmfile_checksums` | `postUpgradeTasks` → `tools/update-helmfile-checksums` |
+| `.settings.yaml` `helm_diff_checksums` | `postUpgradeTasks` → `tools/update-helm-diff-checksums` |
 | `.go-version` (Go toolchain) | dedicated `golang-version` customManager (`go-toolchain` group) |
 
 The `go` directive in `go.mod` is intentionally not bumped — the Go toolchain version is owned by `.go-version`. Makefile (`GOTOOLCHAIN`), the `load-versions` composite action, `install-karpenter-kwok`, and validator Dockerfiles (`--build-arg GO_VERSION`) all read from that single file.

.github/renovate.json5

@@ -319,6 +319,16 @@
       },
     },
 
+    // ---- helm-diff bumps refresh per-arch SHA256 checksums via post-upgrade script ----
+    {
+      matchDepNames: ["databus23/helm-diff"],
+      postUpgradeTasks: {
+        commands: ["./tools/update-helm-diff-checksums {{{newVersion}}}"],
+        fileFilters: [".settings.yaml"],
+        executionMode: "update",
+      },
+    },
+
     // ---- aiperf-bench base image is capped below python 3.14 ----
     // PR #1906 auto-bumped this to 3.14 and broke the UAT validator image
     // build (aiperf's pyzmq/uvloop had no cp314 wheels, and the single-stage

.github/workflows/codeql.yaml

@@ -39,7 +39,7 @@ jobs:
       - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e  # v7.0.0
         with:
           go-version: ${{ steps.versions.outputs.go }}
-          cache: false  # vendor/ provides deps; disable to save disk on constrained runners
+          cache: true
       - uses: github/codeql-action/init@ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd  # v4.37.7
         with:
           languages: go

.github/workflows/merge-gate.yaml

@@ -296,7 +296,7 @@ jobs:
       - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e  # v7.0.0
         with:
           go-version: ${{ steps.versions.outputs.go }}
-          cache: false
+          cache: true
       - uses: github/codeql-action/init@ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd  # v4.37.7
         with:
           languages: go
@@ -528,7 +528,7 @@ jobs:
       - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e  # v7.0.0
         with:
           go-version: ${{ steps.versions.outputs.go }}
-          cache: false
+          cache: true
       - name: Verify committed BOM versions and variants match the recipes
         env:
           GOFLAGS: -mod=vendor
@@ -579,7 +579,7 @@ jobs:
       - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e  # v7.0.0
         with:
           go-version: ${{ steps.versions.outputs.go }}
-          cache: false
+          cache: true
       - name: Verify committed tuning-status table is up to date
         env:
           GOFLAGS: -mod=vendor
@@ -661,7 +661,7 @@ jobs:
       - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e  # v7.0.0
         with:
           go-version: ${{ steps.versions.outputs.go }}
-          cache: false
+          cache: true
       - name: Verify committed coverage matrix is up to date
         env:
           GOFLAGS: -mod=vendor

.github/workflows/renovate.yaml

@@ -122,5 +122,18 @@ jobs:
           # checksums getting committed alongside a version bump (the version
           # changes but the helmfile_checksums/chainsaw_checksums block does
           # not), which then breaks E2E/CLI E2E at the sha256 verify step.
-          RENOVATE_ALLOWED_POST_UPGRADE_COMMANDS: '["^\\./tools/update-[a-z]+-checksums "]'
+          # `[a-z-]+` (not `[a-z]+`) so hyphenated tool names match too, e.g.
+          # tools/update-helm-diff-checksums.
+          #
+          # The stale-checksum symptom above is not uniform across the three
+          # tools. A hook that runs but *fails* (renamed asset, egress blip)
+          # leaves the new version pinned against the old checksums, and only
+          # helmfile/chainsaw fail a PR-gating job for it — qualification.yaml
+          # feeds their sha256 into setup-build-tools, which verifies it.
+          # helm_diff_checksums is read solely by tests/uat/lib/phases.sh, so a
+          # stale helm-diff pin stays green on every PR check and first surfaces
+          # in nightly UAT, across all clouds. Renovate's "Artifact update
+          # problem" warning in the PR body is the signal to heed; helm-diff is
+          # deliberately not auto-merged, so a human sees it.
+          RENOVATE_ALLOWED_POST_UPGRADE_COMMANDS: '["^\\./tools/update-[a-z-]+-checksums "]'
           LOG_LEVEL: ${{ inputs.logLevel || 'info' }}

.settings.yaml

@@ -84,6 +84,15 @@ testing_tools:
   helm: 'v4.2.4'
   # renovate: datasource=github-releases depName=databus23/helm-diff depType=testing_tools
   helm_diff: 'v3.15.11'
+  # helm_diff_checksums are refreshed automatically by tools/update-helm-diff-checksums
+  # via Renovate postUpgradeTasks whenever the helm-diff version above is bumped.
+  # The darwin_* keys name GOOS (matching the sibling blocks); upstream labels
+  # those release assets "macos".
+  helm_diff_checksums:
+    linux_amd64: 'ed54aa5a14a51a6e44cf8ab6020aaa9e56779b4d33bc2adf07abea6b240a5e7c'
+    linux_arm64: '1c38c487ec348fe5ef2b10c61deddff79f0fa5bfbab762b302874578e6dc2d2c'
+    darwin_amd64: '833e1054d0cfe28bbd6f2edc4c1d5b786968b7732f8df6fe99b67c3bf12bf7bb'
+    darwin_arm64: 'ba489cc4a5998ad259fbc18c454f2319da26ba4c8f469e7acb587a07e61072cd'
   # renovate: datasource=github-releases depName=helmfile/helmfile depType=testing_tools
   helmfile: 'v1.7.4'
   # helmfile_checksums are refreshed automatically by tools/update-helmfile-checksums

Makefile

@@ -34,6 +34,13 @@ CTLPTL_CONFIG_FILE = .ctlptl.yaml
 REGISTRY_PORT = 5001
 REGISTRY_NAME = ctlptl-registry
 
+# Kind node image (single source of truth: .settings.yaml testing.kind_node_image).
+# .ctlptl.yaml intentionally does not hardcode this — cluster-create injects it so
+# local dev and CI can never pin a second, drifting copy of the version. No
+# hardcoded fallback here: cluster-create fails closed on an empty read
+# instead of silently degrading to a stale image (see its yq-missing check).
+KIND_NODE_IMAGE ?= $(shell yq -r '.testing.kind_node_image' .settings.yaml 2>/dev/null)
+
 # Default target
 all: help
 
@@ -819,7 +826,19 @@ cluster-create: ## Creates local Kind cluster with registry
 		echo "     or: go install sigs.k8s.io/kind@latest"; \
 		exit 1; \
 	fi
-	ctlptl apply -f $(CTLPTL_CONFIG_FILE)
+	@if ! command -v yq >/dev/null 2>&1; then \
+		echo "Error: yq is not installed."; \
+		echo "Install: brew install yq"; \
+		echo "     or: go install github.com/mikefarah/yq/v4@latest"; \
+		exit 1; \
+	fi
+	@if [ -z "$(KIND_NODE_IMAGE)" ]; then \
+		echo "Error: could not resolve testing.kind_node_image from .settings.yaml."; \
+		echo "Check the key exists and .settings.yaml is valid YAML."; \
+		exit 1; \
+	fi
+	@echo "Pinning Kind node image: $(KIND_NODE_IMAGE) (from .settings.yaml)"
+	img="$(KIND_NODE_IMAGE)" yq eval-all '(select(.kind == "Cluster") | .kindV1Alpha4Cluster.nodes[]).image = strenv(img)' $(CTLPTL_CONFIG_FILE) | ctlptl apply -f -
 	@echo "Waiting for nodes to be ready..."
 	@kubectl wait --for=condition=ready nodes --all --timeout=300s
 	@echo "Cluster created. Registry at localhost:$(REGISTRY_PORT)"
@@ -854,10 +873,6 @@ KWOK_VERSION ?= $(shell yq -r '.testing_tools.kwok' .settings.yaml 2>/dev/null)
 ifeq ($(KWOK_VERSION),)
 KWOK_VERSION := v0.7.0
 endif
-KIND_NODE_IMAGE ?= $(shell yq -r '.testing.kind_node_image' .settings.yaml 2>/dev/null)
-ifeq ($(KIND_NODE_IMAGE),)
-KIND_NODE_IMAGE := kindest/node:v1.32.0
-endif
 CTLPTL_KWOK_CONFIG_FILE := .ctlptl-kwok.yaml
 
 .PHONY: kwok-cluster

api/aicr/v1/server.yaml

@@ -2622,6 +2622,13 @@ components:
                     mode:
                       type: string
                       enum: [disabled, customer-managed, aicr-provided]
+            runtimeInventory:
+              type: object
+              required: [mode]
+              properties:
+                mode:
+                  type: string
+                  enum: [enabled, disabled]
         componentRefs:
           type: array
           items:
@@ -2812,7 +2819,10 @@ components:
       allOf:
         - $ref: "#/components/schemas/RecipeResponseBase/properties/configuration"
         - type: object
-          required: [slurm]
+          # At least one section, but not any particular one: a recipe may
+          # record a runtime-inventory selection without Slurm accounting, or
+          # the reverse. Requiring `slurm` rejected the former outright.
+          minProperties: 1
           additionalProperties: false
           properties:
             slurm:
@@ -2828,6 +2838,14 @@ components:
                     mode:
                       type: string
                       enum: [disabled, customer-managed, aicr-provided]
+            runtimeInventory:
+              type: object
+              required: [mode]
+              additionalProperties: false
+              properties:
+                mode:
+                  type: string
+                  enum: [enabled, disabled]
 
     ProfileRecipeResponse:
       allOf:

docs/contributor/recipe.md

@@ -345,6 +345,22 @@ spec:
           value: ">= v25.10.0"
 ```
 
+Host-managed driver floors (GKE COS / A4X Max and similar platforms where
+`driver.enabled: false`) use a separate deployment constraint,
+`Deployment.gpu-driver.version` (e.g. `">= 580.95.05"`).
+`check-nvidia-smi` evaluates it against the nvidia-smi banner on each
+verified node; when the constraint is absent the check keeps its
+banner-presence behavior and does not invent a floor (#1995).
+When the constraint is set but the host driver cannot be measured —
+unreadable nvidia-smi banner, no GPU nodes, all GPU nodes cordoned, or
+GPU nodes busy with workloads — the check fails closed rather than
+Skip. Skip on those paths is preserved only when no floor is
+configured. The value must carry a comparison operator (`>=`, `>`,
+`<=`, `<`) to behave as a floor; a bare version is exact string match,
+so a newer driver would fail. The constraint name is an exact match;
+a typo silently disables the floor (shared with
+`Deployment.gpu-operator.version`).
+
 For a query `{service: eks, accelerator: gb200, intent: training}`,
 the resolver returns three independent maximal leaves —
 `gb200-eks-training` (matched by explicit criteria), `gb200-any`

docs/contributor/validator.md

@@ -611,6 +611,11 @@ tagged with its cordon state, and:
    schedulable count: it is `0` on the all-cordoned and busy-skip
    paths (nothing was attempted yet) and the successful-node count on
    the failure path (a partial pass is not conflated with a full one).
+   Exception: when the recipe declares `Deployment.gpu-driver.version`,
+   those same all-cordoned and busy paths (and the no-GPU-nodes path)
+   fail closed instead of Skip — a declared host-driver floor that
+   cannot be measured must not PASS (#1995). Skip remains only when
+   the floor constraint is absent.
    The `RESULT:` prefix is `pkg/validator/validator.go`'s
    `resultSummaryPrefix` convention: the validator runtime echoes the
    trailing text of any such stdout line into live CLI output via