Merge branch 'main' into fix/1779-e2e-kind-node-caching

Author: njhensley

.claude/CLAUDE.md

@@ -569,11 +569,11 @@ Process and unique findings below; the rule sections above (Error Wrapping, Cont
 - **Once the PR is not a draft — whether you flipped it or opened it that way — append commits instead of rewriting history.** Inline comments anchor to SHAs, and any rewrite outdates every one of them. Each appended commit dismisses approvals (`dismiss_stale_reviews_on_push` is on); that is the intended cost, since a force-push dismisses them too *and* destroys the anchors. Returning to draft stops paging reviewers and restores the draft phase's freedom to rewrite; if inline comments already exist, say on the PR that you rewrote and name the old and new SHA so reviewers know to restart.
 - **Do not hand-squash before merge.** `NVIDIA/aicr` is squash-merge-only, so GitHub composes the commit on `main` from the PR title; by default only the title and trailers reach `main`. Durable wording belongs in the PR title.
 - **Once the PR is not a draft, rebase only when the merge gate requires it** — but it does require it: the repo enforces up-to-date branches, so a PR behind `main` cannot merge. `git fetch origin main && git rebase origin/main`, never GitHub's "Update branch" button or a merge commit. A rebase is itself a force-push and can outdate anchors even when the content is unchanged, so treat it as one. To confirm it replayed your work cleanly, compare `git diff origin/main...HEAD` before and after, or use `git range-diff <old-sha>...HEAD`.
-- **Once the PR is not a draft, force-push only when you must** — a gate-required rebase, a missing signature or sign-off, a wrong base, a committed secret. Nothing else qualifies at that point; while it is still a draft the bullets above apply. Use exactly one of these two, never both:
-  - `--force-with-lease=<refname>:<sha-you-observed>` — `<refname>` is the branch name as it exists on the remote, unprefixed, e.g. `git push origin my-branch --force-with-lease=my-branch:abc1234`; or
-  - `--force-with-lease --force-if-includes` (or `push.useForceIfIncludes=true` once, globally), which adds a reflog check.
+- **Once the PR is not a draft, force-push only when you must** — a gate-required rebase, a missing signature or sign-off, a wrong base, a committed secret. Nothing else qualifies at that point; while it is still a draft the bullets above apply. **Default to the first form below**; never combine the two forms:
+  - `--force-with-lease --force-if-includes` — adds a reflog check that the remote tip was integrated locally, which is what closes the fail-open hole below. Enable it once per clone with `git config push.useForceIfIncludes true` — **per-repo, not `--global`**: the check also rejects in a fresh clone (the only reflog entry is the clone itself), and that friction belongs only where it is needed; or
+  - `--force-with-lease=<refname>:<sha-you-observed>` — pins the lease to a head you read and confirmed first. Use it where the reflog check cannot pass, such as a fresh clone. `<refname>` is the branch name as it exists on the remote, unprefixed, e.g. `git push origin my-branch --force-with-lease=my-branch:abc1234`.
 
-  `--force-if-includes` is a documented no-op when the lease already names an expected SHA, so pairing them leaves you trusting a guard git has disabled. A bare `--force-with-lease` alone can silently pass, because a background fetch or a concurrent session sharing the clone may have refreshed the remote-tracking ref its lease reads. Never plain `--force`. Say on the PR that you force-pushed, naming the old and new SHA.
+  `--force-if-includes` is a documented no-op when the lease already names an expected SHA, so pairing them leaves you trusting a guard git has disabled. A bare `--force-with-lease` alone can silently pass, because a background fetch or a concurrent session sharing the clone may have refreshed the remote-tracking ref its lease reads. Never plain `--force`. Say on the PR that you force-pushed, naming the old and new SHA. Both guards are a backstop, not the primary protection — they do not replace the rule against rebasing or force-pushing a branch you do not own.
 - Verify what you are about to push: `git log --oneline origin/main..HEAD` and `git diff --stat origin/main...HEAD`.
 - Sign every commit both ways: `git commit -S -s` (see Git Configuration above).
 
@@ -610,7 +610,8 @@ CI also posts per-package deltas post-push via `go-coverage-report` (`on-push-co
 - Area labels are auto-assigned by `.github/labeler.yml` based on changed file paths (e.g., `area/recipes`, `area/ci`, `area/api`, `area/cli`, `area/bundler`, `area/collector`, `area/validator`, `area/docs`, `area/infra`, `area/tests`). You may also add them manually when the auto-labeler wouldn't match (e.g., issue-only PRs or cross-cutting changes).
 - Do NOT add issue priority labels `P0`, `P1`, or `P2` to PRs; they are reserved for issues and automation removes them from pull requests
 - Do NOT add `size/*` labels (auto-assigned by bot)
-- Keep the PR title under 70 characters; use the description for details
+- **PR titles are linted.** CI enforces Conventional Commits format — `type: subject`, `type(scope): subject`, `type!: subject`, or `type(scope)!: subject`, where `!` marks a breaking change. Valid types: `build`, `chore`, `ci`, `docs`, `feat`, `fix`, `perf`, `refactor`, `revert`, `style`, `test`. Scopes may be mixed case (`fix(GB200):`). A malformed title fails the check; editing the title re-runs it automatically. The title is the whole commit message on `main` (`squash_merge_commit_message: BLANK`) and cannot be corrected after merge
+- Keep the PR title to 70 characters or fewer; use the description for details. Over 70 warns but does not block — dependency-bot titles embed pseudo-versions that cannot be shortened
 
 **Issue policy:**
 - Set an **org issue type** on new issues. This is a GitHub-native field (shown in the standard issue view, distinct from repo `area/*`/`theme/*` labels) that categorizes the issue. Valid types: `Task`, `Bug`, `Enhancement`, `Epic`, `Initiative`, `Documentation`.

.github/RENOVATE.md

@@ -9,7 +9,7 @@ Self-hosted Renovate keeps the project's dependencies up to date across `go.mod`
 
 - Configuration: [`.github/renovate.json5`](renovate.json5)
 - Workflow: [`.github/workflows/renovate.yaml`](workflows/renovate.yaml)
-- Companion script: [`tools/update-chainsaw-checksums`](../tools/update-chainsaw-checksums)
+- Companion scripts: [`tools/update-chainsaw-checksums`](../tools/update-chainsaw-checksums), [`tools/update-helmfile-checksums`](../tools/update-helmfile-checksums), [`tools/update-helm-diff-checksums`](../tools/update-helm-diff-checksums)
 
 Policy choices (schedule, cooldown, auto-merge scope, group consolidation) are documented inline in `renovate.json5`. This doc covers what's covered, how to extend coverage, and the known gotchas.
 
@@ -25,6 +25,8 @@ Policy choices (schedule, cooldown, auto-merge scope, group consolidation) are d
 | `.settings.yaml` (28 tool entries) | custom regex manager (`# renovate:` annotations) |
 | `.settings.yaml` `nvkind` SHA | dedicated git-refs digest customManager (`# renovate-digest:`) |
 | `.settings.yaml` `chainsaw_checksums` | `postUpgradeTasks` → `tools/update-chainsaw-checksums` |
+| `.settings.yaml` `helmfile_checksums` | `postUpgradeTasks` → `tools/update-helmfile-checksums` |
+| `.settings.yaml` `helm_diff_checksums` | `postUpgradeTasks` → `tools/update-helm-diff-checksums` |
 | `.go-version` (Go toolchain) | dedicated `golang-version` customManager (`go-toolchain` group) |
 
 The `go` directive in `go.mod` is intentionally not bumped — the Go toolchain version is owned by `.go-version`. Makefile (`GOTOOLCHAIN`), the `load-versions` composite action, `install-karpenter-kwok`, and validator Dockerfiles (`--build-arg GO_VERSION`) all read from that single file.

.github/actions/e2e/action.yml

@@ -43,38 +43,98 @@ runs:
     - name: Install E2E testing tools
       uses: ./.github/actions/install-e2e-tools
 
-    - name: Create Kind cluster
-      shell: bash
-      run: make cluster-create
-
-    - name: Deploy aicrd to Kind
+    - name: Create Kind cluster and deploy aicrd
       shell: bash
       env:
         GOFLAGS: -mod=vendor
       run: |
-        # Build and push aicrd image (replaces Tilt custom_build)
+        # If any `wait` below fails, `set -e` (GitHub's default shell flags)
+        # exits this script immediately - any background job not yet reached
+        # by its own `wait` would otherwise keep running. Most consequential
+        # for pid_cluster specifically: the job's always()-run Cleanup step
+        # (`make cluster-delete`) could then race a still-in-flight
+        # `make cluster-create`. Kill whatever is still alive on any exit.
+        cleanup_background_jobs() {
+          local rc=$?
+          for pid in "$pid_cluster" "$pid_dep" "$pid_perf" "$pid_conf" "$pid_cli" "$pid_aicr" "$pid_validators"; do
+            [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null && kill "$pid" 2>/dev/null
+          done
+          exit "$rc"
+        }
+        trap cleanup_background_jobs EXIT
+
+        # This step folds what used to be two separate steps (Create Kind
+        # cluster / Deploy aicrd) into one, so GitHub's per-step red-X no
+        # longer says at a glance which phase failed. Group markers fold the
+        # log into the same phases instead.
+        echo "::group::cluster-and-builds"
+        # cluster-create (~35s locally; Kind bring-up + registry) and the
+        # host compiles below touch none of each other's inputs/outputs, so
+        # launch everything in the background up front. Each later command
+        # waits only on the specific PID(s) its own inputs actually depend
+        # on: aicrd's ko build needs just the cluster/registry (not the
+        # validator/CLI binaries), so it must not be gated behind builds it
+        # doesn't read from - an earlier version of this step wait'ed on all
+        # four build PIDs before it, which measured ~19s slower in CI than
+        # the original sequential script for no reason: nothing here reads
+        # dist/validator/* or dist/e2e/aicr until the docker build step below.
+        pid_cluster=""; pid_dep=""; pid_perf=""; pid_conf=""; pid_cli=""; pid_aicr=""; pid_validators=""
+        make cluster-create > /tmp/cluster-create.log 2>&1 &
+        pid_cluster=$!
+
+        mkdir -p dist/validator dist/e2e
+        for phase in deployment performance conformance; do
+          mkdir -p "validators/${phase}/testdata"
+        done
+
+        # Compile validator binaries + the host aicr CLI on the runner (Go
+        # build cache hit) while the cluster comes up. This is also much
+        # faster than building inside Docker (no cache sharing there). Go's
+        # build cache is safe for concurrent use by simultaneous go
+        # commands, including ko's own internal compile of aicrd below.
+        CGO_ENABLED=0 go build -trimpath -o dist/validator/deployment ./validators/deployment &
+        pid_dep=$!
+        CGO_ENABLED=0 go build -trimpath -o dist/validator/performance ./validators/performance &
+        pid_perf=$!
+        CGO_ENABLED=0 go build -trimpath -o dist/validator/conformance ./validators/conformance &
+        pid_conf=$!
+        go build -o dist/e2e/aicr ./cmd/aicr &
+        pid_cli=$!
+
+        # aicrd only needs the cluster/registry - build and push it (replaces
+        # Tilt custom_build) as soon as that's ready, not gated behind the
+        # unrelated validator/CLI builds above.
+        #
+        # cluster-create's output was redirected to a file above so it
+        # wouldn't interleave with the concurrent builds; print it on
+        # failure too (not just success), since a failed `wait` here is the
+        # most common, most opaque failure on this step and would otherwise
+        # exit via the EXIT trap with zero diagnostic output.
+        wait $pid_cluster || { cat /tmp/cluster-create.log; exit 1; }
+        cat /tmp/cluster-create.log
+        echo "::endgroup::"
+
+        echo "::group::deploy-aicrd"
         KO_DOCKER_REPO=localhost:5001/aicrd ko build --bare --tags=tilt ./cmd/aicrd
 
         # Apply namespace first (must exist before deployment references it)
         kubectl apply -f tilt/k8s/namespace.yaml
         kubectl apply -f tilt/k8s/
-
-        # Build images + CLI binary in parallel while deployment starts
+        echo "::endgroup::"
+
+        echo "::group::validator-images"
+        # The validator images COPY the binaries built above, so wait for
+        # those specifically (not pid_cli, which nothing here needs) before
+        # building them.
+        wait $pid_dep
+        wait $pid_perf
+        wait $pid_conf
+
+        # Build + push the remaining images. Validator binaries are already
+        # compiled above; this is just the COPY-only image build + push.
         KO_DOCKER_REPO=localhost:5001/aicr ko build --bare --tags=local ./cmd/aicr &
         pid_aicr=$!
-        # Compile validator binaries on host (Go build cache hit) then COPY-only images.
-        # This is much faster than building inside Docker (no cache sharing).
         (
-          mkdir -p dist/validator
-          CGO_ENABLED=0 go build -trimpath -o dist/validator/deployment ./validators/deployment &
-          CGO_ENABLED=0 go build -trimpath -o dist/validator/performance ./validators/performance &
-          CGO_ENABLED=0 go build -trimpath -o dist/validator/conformance ./validators/conformance &
-          wait
-          # Build per-phase COPY-only images and push to local registry.
-          # Ensure testdata dirs exist (conformance has none).
-          for phase in deployment performance conformance; do
-            mkdir -p "validators/${phase}/testdata"
-          done
           for phase in deployment performance conformance; do
             docker build -t "localhost:5001/aicr-validators/${phase}:latest" -f - . <<DOCKERFILE
         FROM nvcr.io/nvidia/distroless/static:v4.0.0@sha256:d90158b69e250d2018f32622b5c622925202ee97224a990a54b63811cb1e3d69
@@ -99,10 +159,14 @@ runs:
           docker push "localhost:5001/aicr-validators/aiperf-bench:latest"
         ) &
         pid_validators=$!
-        go build -o dist/e2e/aicr ./cmd/aicr &
-        pid_cli=$!
-        wait $pid_aicr $pid_validators $pid_cli
-
+        # Separate waits, not `wait $pid_aicr $pid_validators`: bash returns
+        # only the LAST pid's exit status from a combined wait, so a failed
+        # pid_aicr would be silently masked whenever pid_validators succeeds.
+        wait $pid_aicr
+        wait $pid_validators
+        echo "::endgroup::"
+
+        echo "::group::rollout-and-verify"
         # Wait for deployment rollout (may already be ready)
         kubectl rollout status deployment/aicrd -n aicr --timeout=120s
 
@@ -112,6 +176,13 @@ runs:
         curl -sf http://localhost:5001/v2/aicr-validators/performance/tags/list
         curl -sf http://localhost:5001/v2/aicr-validators/conformance/tags/list
         curl -sf http://localhost:5001/v2/aicr-validators/aiperf-bench/tags/list
+        echo "::endgroup::"
+
+        # Not needed by anything above, but its output (dist/e2e/aicr) is
+        # used by a later step (AICR_BIN) - wait now so a build failure here
+        # fails this step instead of surfacing as a confusing missing-binary
+        # error two steps later, and so the step's own exit code reflects it.
+        wait $pid_cli
 
     - name: Set up fake GPU environment
       shell: bash
@@ -151,6 +222,7 @@ runs:
       shell: bash
       run: |
         mkdir -p /tmp/debug-artifacts
+        cp /tmp/cluster-create.log /tmp/debug-artifacts/cluster-create.log 2>/dev/null || true
         kubectl get all --all-namespaces > /tmp/debug-artifacts/all-resources.txt || true
         kubectl get events --all-namespaces --sort-by='.lastTimestamp' > /tmp/debug-artifacts/events.txt || true
         kubectl logs -n aicr -l app.kubernetes.io/name=aicrd --tail=500 > /tmp/debug-artifacts/aicrd-logs.txt || true

.github/actions/runtime-install/generate-bundle.sh

@@ -20,13 +20,13 @@ rm -rf bundle
 BUNDLE_ARGS=(
   --recipe recipe.yaml
   --accelerated-node-toleration nvidia.com/gpu:NoSchedule
-  # Kind recipes are deliberately host-installed (nvkind: driver.enabled=false,
-  # driver preinstalled on the host, no driver pod for the NVSentinel labeler
-  # to observe), so bundling without the documented remedy is a blocking error
-  # (CheckNVSentinelDriverLabelDetectable, issue #2175). This is the
-  # detect-don't-fix model: the CI caller passes the remedy the gate names,
-  # exactly as an operator would; the recipe itself stays unchanged (#2179).
-  --set nv-sentinel:labeler.assumeDriverInstalled=true
+  # No NVSentinel override: the kind overlay assigns
+  # labeler.assumeDriverInstalled itself (#2181). Kind recipes are
+  # deliberately host-installed (nvkind: driver.enabled=false, driver
+  # preinstalled on the host, no driver pod for the NVSentinel labeler to
+  # observe), so the value is required — it is now supplied by the recipe
+  # rather than by this caller, and CheckNVSentinelDriverLabelDetectable
+  # verifies it.
   --output bundle
 )
 

.github/renovate.json5

@@ -319,6 +319,16 @@
       },
     },
 
+    // ---- helm-diff bumps refresh per-arch SHA256 checksums via post-upgrade script ----
+    {
+      matchDepNames: ["databus23/helm-diff"],
+      postUpgradeTasks: {
+        commands: ["./tools/update-helm-diff-checksums {{{newVersion}}}"],
+        fileFilters: [".settings.yaml"],
+        executionMode: "update",
+      },
+    },
+
     // ---- aiperf-bench base image is capped below python 3.14 ----
     // PR #1906 auto-bumped this to 3.14 and broke the UAT validator image
     // build (aiperf's pyzmq/uvloop had no cp314 wheels, and the single-stage

.github/workflows/codeql.yaml

@@ -39,7 +39,7 @@ jobs:
       - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e  # v7.0.0
         with:
           go-version: ${{ steps.versions.outputs.go }}
-          cache: false  # vendor/ provides deps; disable to save disk on constrained runners
+          cache: true
       - uses: github/codeql-action/init@ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd  # v4.37.7
         with:
           languages: go

.github/workflows/merge-gate.yaml

@@ -296,7 +296,7 @@ jobs:
       - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e  # v7.0.0
         with:
           go-version: ${{ steps.versions.outputs.go }}
-          cache: false
+          cache: true
       - uses: github/codeql-action/init@ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd  # v4.37.7
         with:
           languages: go
@@ -528,7 +528,7 @@ jobs:
       - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e  # v7.0.0
         with:
           go-version: ${{ steps.versions.outputs.go }}
-          cache: false
+          cache: true
       - name: Verify committed BOM versions and variants match the recipes
         env:
           GOFLAGS: -mod=vendor
@@ -579,7 +579,7 @@ jobs:
       - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e  # v7.0.0
         with:
           go-version: ${{ steps.versions.outputs.go }}
-          cache: false
+          cache: true
       - name: Verify committed tuning-status table is up to date
         env:
           GOFLAGS: -mod=vendor
@@ -661,7 +661,7 @@ jobs:
       - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e  # v7.0.0
         with:
           go-version: ${{ steps.versions.outputs.go }}
-          cache: false
+          cache: true
       - name: Verify committed coverage matrix is up to date
         env:
           GOFLAGS: -mod=vendor

.github/workflows/on-push.yaml

@@ -93,7 +93,7 @@ jobs:
           persist-credentials: false
 
       - name: Setup Docker Buildx
-        uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c  # v4.2.0
+        uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e  # v4.3.0
 
       - name: Authenticate to registry
         uses: ./.github/actions/ghcr-login
@@ -154,7 +154,7 @@ jobs:
         uses: ./.github/actions/ghcr-login
 
       - name: Setup Docker Buildx
-        uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c  # v4.2.0
+        uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e  # v4.3.0
 
       - name: Create multi-arch manifests for validator images
         env: