docs: correct agent label coverage, name prefixes, and validate flag targets

Four documentation claims did not match the code.

Label coverage. pkg/k8s/agent/doc.go and docs/user/agent-deployment.md both
asserted that every created object — explicitly counting the staging ConfigMap
— carries managed-by=aicr, component=snapshot-agent and aicr.run/run-id. The
staging ConfigMap carries none of those: it is written by
serializer.ConfigMapWriter, which stamps only app.kubernetes.io/name,
app.kubernetes.io/component (the header kind) and app.kubernetes.io/version.
An earlier review decided not to add the labels at the serializer, because that
writer also produces the user's delivered cm:// artifact and stamping the sweep
key on a never-delete object is a hazard — so the documentation is corrected
and the serializer is left alone. The agent-deployment troubleshooting recipe
now addresses the staging ConfigMap by its run-scoped name.

Shipped config examples. ADR-020 decision 8 requires examples pinning
jobName/serviceAccountName to be updated; four blocks still pinned them (two in
cli-config.md, two in cli-reference.md), and one contradicted a neighbouring
row this branch already updated — pinning serviceAccountName: aicr for validate
while the reference gives the default as aicr-validate. The pins are dropped
and both agent.* schema rows now state the fields are optional prefixes with
the run ID appended.

NameBase godoc. The pkg/client/v1 copy repeated the claim that NameBase "has no
effect once either of those is set". The fallback is per name.

Validate flag targets. The aicr validate rows for --job-name /
--service-account-name attributed them to "the validation Job" and its
ServiceAccount. Both feed only the optional live snapshot-capture agent: the
validator Jobs are named aicr-<validator>-<hash> and their ServiceAccount
aicr-validator-<run-id>, neither influenced by these flags, and both flags are
inert when --snapshot is supplied.

Refs: ADR-020
Signed-off-by: Alex Yuskauskas <ayuskauskas@nvidia.com>

Author: ayuskauskas

docs/user/agent-deployment.md

@@ -333,8 +333,12 @@ Check RBAC permissions. The ServiceAccount name is run-scoped (`aicr-<run-id>`),
 Select it by run ID, not by position: with concurrent snapshot runs the namespace
 holds one ServiceAccount per run, and `.items[0]` would pick an arbitrary one — so
 the checks below could report on a healthy run while you are debugging a failed one.
-The CLI logs the run ID when it starts (`snapshot agent run: runID=...`), and every
-resource the run created carries it as the `aicr.run/run-id` label:
+The CLI logs the run ID when it starts (`snapshot agent run: runID=...`), and the
+Job, its pods, and its RBAC resources all carry it as the `aicr.run/run-id`
+label. (The staging ConfigMap is the exception — it is written from inside the
+pod and carries only `app.kubernetes.io/name`, `app.kubernetes.io/component` and
+`app.kubernetes.io/version`; find it by its run-scoped name instead, see
+[Job Completes but No Output](#job-completes-but-no-output).)
 
 ```shell
 # From the failing Job (or use the runID the CLI logged at start).
@@ -381,10 +385,18 @@ Check ConfigMap and container logs:
 # "-o cm://<namespace>/<name>", the agent stages its result in a run-scoped
 # ConfigMap named "aicr-agent-snapshot-<run-id>" that cleanup deletes when
 # the run ends — pass --no-cleanup to keep it around for inspection.
-kubectl get configmap -n gpu-operator -l app.kubernetes.io/name=aicr
+#
+# This object is written by the in-pod agent, not by the CLI, so it does NOT
+# carry the aicr.run/run-id label the Job and RBAC resources do. Address it by
+# name: with concurrent runs, the label selector below matches every run's
+# staging ConfigMap plus any delivered cm:// artifact in the namespace.
+kubectl get configmap -n gpu-operator aicr-agent-snapshot-$RUN_ID
 
 # View ConfigMap contents
-kubectl get configmap -n gpu-operator -l app.kubernetes.io/name=aicr -o yaml
+kubectl get configmap -n gpu-operator aicr-agent-snapshot-$RUN_ID -o yaml
+
+# Or list every aicr-written ConfigMap in the namespace (all runs)
+kubectl get configmap -n gpu-operator -l app.kubernetes.io/name=aicr
 
 # View pod logs for errors
 kubectl logs -n gpu-operator -l app.kubernetes.io/name=aicr,app.kubernetes.io/component=snapshot-agent

docs/user/cli-config.md

@@ -88,8 +88,8 @@ spec:
       namespace: aicr-validation
       image: ""                      # default: ghcr.io/nvidia/aicr:latest
       imagePullSecrets: []
-      jobName: aicr
-      serviceAccountName: aicr
+      # jobName / serviceAccountName are optional PREFIXES, not names — the
+      # run ID is always appended. Omit them to take the defaults.
       nodeSelector:
         nodeGroup: gpu-worker
       tolerations:
@@ -170,8 +170,8 @@ spec:
       namespace: aicr-validation
       image: ""
       imagePullSecrets: []
-      jobName: aicr
-      serviceAccountName: aicr
+      # Optional prefixes; omitted here so the defaults apply
+      # (jobName: aicr-validate, serviceAccountName: aicr).
       nodeSelector:
         nodeGroup: gpu-worker
       tolerations:
@@ -218,7 +218,7 @@ produced from the live cluster.
 | `output.path` | string | Output file path (same as `-o`) |
 | `output.format` | string | `yaml` \| `json` \| `table` |
 | `output.template` | string | Optional Go template path |
-| `agent.*` | object | In-cluster capture Job pod: `namespace`, `image`, `imagePullSecrets`, `jobName`, `serviceAccountName`, `nodeSelector`, `tolerations`, `requireGpu`, `runtimeClassName` (mutually exclusive with `requireGpu`), `os`, `requests`, `limits`. Mirrors `spec.validate.agent` so one file pins matching placement for both |
+| `agent.*` | object | In-cluster capture Job pod: `namespace`, `image`, `imagePullSecrets`, `jobName`, `serviceAccountName`, `nodeSelector`, `tolerations`, `requireGpu`, `runtimeClassName` (mutually exclusive with `requireGpu`), `os`, `requests`, `limits`. `jobName` and `serviceAccountName` are optional **prefixes**, not exact names — the run ID is appended (`<prefix>-<run-id>`), so omit them unless you need a custom prefix. Mirrors `spec.validate.agent` so one file pins matching placement for both |
 | `execution.timeout` | duration string | e.g. `5m` |
 | `execution.noCleanup` | bool | Keep the capture Job after completion |
 | `execution.privileged` | bool (tri-state) | Set `false` for PSS-restricted namespaces |
@@ -278,7 +278,7 @@ Inputs to `aicr validate`.
 | Field | Type | Notes |
 |-------|------|-------|
 | `input.recipe` / `.snapshot` | string | Recipe + snapshot to validate |
-| `agent.*` | object | In-cluster validation Job pod; same fields and nil-vs-empty semantics as `spec.snapshot.agent` (minus `runtimeClassName`/`os`/`requests`/`limits`) |
+| `agent.*` | object | The **live snapshot-capture** Job pod `aicr validate` deploys when `input.snapshot` is empty; same fields and nil-vs-empty semantics as `spec.snapshot.agent` (minus `runtimeClassName`/`os`/`requests`/`limits`). `jobName` and `serviceAccountName` are optional prefixes with the run ID appended (defaults: both `aicr-validate`); they do not name the validator Jobs |
 | `execution.phases` | []string | e.g. `[deployment, conformance, performance]` |
 | `execution.failOnError` | bool (tri-state) | Absent = CLI default (`true`); explicit `false` opts out |
 | `execution.failFast` | bool (tri-state) | Stop after the first failed phase |

docs/user/cli-reference.md

@@ -201,8 +201,8 @@ spec:
       namespace: aicr-validation
       image: ""                    # default: ghcr.io/nvidia/aicr:latest
       imagePullSecrets: []
-      jobName: aicr
-      serviceAccountName: aicr
+      # jobName / serviceAccountName are optional PREFIXES, not names — the
+      # run ID is always appended. Omit them to take the defaults.
       nodeSelector:
         nodeGroup: gpu-worker
       tolerations:
@@ -963,8 +963,8 @@ aicr validate [flags]
 | `--namespace` | `-n` | string | aicr-validation | Kubernetes namespace for validation Job deployment |
 | `--image` | | string | ghcr.io/nvidia/aicr:latest | Container image for validation Job |
 | `--image-pull-secret` | | string[] | | Image pull secrets for private registries (repeatable) |
-| `--job-name` | | string | aicr-validate | Prefix for the validation Job name; the run ID is always appended (`<prefix>-<run-id>`) |
-| `--service-account-name` | | string | aicr-validate | Prefix for the validation Job's ServiceAccount name; the run ID is always appended (`<prefix>-<run-id>`) |
+| `--job-name` | | string | aicr-validate | Prefix for the **live snapshot-capture agent's** Job name; the run ID is always appended (`<prefix>-<run-id>`). Inert when `--snapshot` is supplied — no agent is deployed. Does not name the validator Jobs (`aicr-<validator>-<hash>`) |
+| `--service-account-name` | | string | aicr-validate | Prefix for the **live snapshot-capture agent's** ServiceAccount, Role, and RoleBinding; the run ID is always appended (`<prefix>-<run-id>`). Inert when `--snapshot` is supplied. Does not name the validator Jobs' ServiceAccount (`aicr-validator-<run-id>`) |
 | `--node-selector` | | string[] | | Override GPU node selection for the live snapshot agent (when `--snapshot` is omitted) and inner validation workloads. Replaces platform-specific selectors (e.g., `cloud.google.com/gke-accelerator`, `node.kubernetes.io/instance-type`) on inner workloads like NCCL benchmark pods. Use when GPU nodes have non-standard labels. Does not affect the validator orchestrator Job. (format: key=value, repeatable) |
 | `--toleration` | | string[] | | Override tolerations for the live snapshot agent (when `--snapshot` is omitted) and inner validation workloads. When omitted, the snapshot agent tolerates all taints. Does not affect the validator orchestrator Job. (format: key=value:effect, repeatable) |
 | `--timeout` | | duration | 5m | Timeout for validation Job completion |
@@ -1170,8 +1170,9 @@ spec:
       namespace: aicr-validation
       image: ghcr.io/nvidia/aicr:v0.19.0
       imagePullSecrets: [registry-secret]
-      jobName: aicr-validate
-      serviceAccountName: aicr
+      # Optional prefixes for the live-capture agent, not names — the run ID
+      # is always appended. Omitted here so the defaults apply
+      # (both aicr-validate).
       nodeSelector:
         my-org/gpu-pool: "true"
       tolerations:                         # [] clears the live snapshot agent's tolerate-all default

pkg/client/v1/types.go

@@ -186,9 +186,13 @@ type AgentConfig struct {
 	// snapshot agent and its validator Jobs the same RunID.
 	RunID string
 
-	// NameBase prefixes generated Job/ServiceAccount/RBAC names when
-	// JobName and ServiceAccountName are left empty; it has no effect
-	// once either of those is set. Defaults to "aicr" when also empty.
+	// NameBase prefixes generated Job/ServiceAccount/RBAC names. The
+	// fallback is per name, not all-or-nothing: the Job uses JobName when
+	// set and NameBase otherwise, while the ServiceAccount, Role and
+	// RoleBinding use ServiceAccountName when set and NameBase otherwise.
+	// Setting only one of the two therefore leaves NameBase governing the
+	// other. Defaults to "aicr" when also empty.
+	//
 	// JobName and ServiceAccountName themselves are optional prefixes,
 	// not required names — RunID is appended to whichever prefix
 	// applies, so the deployed object names are always run-scoped.

pkg/k8s/agent/doc.go

@@ -46,25 +46,42 @@ Two objects are deliberately NOT run-scoped:
     artifact. It is written on purpose and never deleted
     (Config.OwnsOutputConfigMap is false for it).
 
-Every created object carries app.kubernetes.io/name=aicr,
-app.kubernetes.io/managed-by=aicr,
+Every object this package itself creates — the Job, ServiceAccount, Role,
+RoleBinding, ClusterRole, and ClusterRoleBinding — carries
+app.kubernetes.io/name=aicr, app.kubernetes.io/managed-by=aicr,
 app.kubernetes.io/component=snapshot-agent, and aicr.run/run-id=<RunID>, on the
 Job's pod template as well as the Job itself. Select agent pods across runs
 with the component label; the Job name changes every run.
 
+The staging ConfigMap is the exception: it is written from inside the pod by
+pkg/serializer's ConfigMap writer, which stamps app.kubernetes.io/name=aicr,
+app.kubernetes.io/component=<snapshot kind> and app.kubernetes.io/version — not
+managed-by and not the run-ID label. That writer also produces the user's
+delivered cm:// artifact, so it deliberately does not stamp the run-ID sweep
+key onto an object this package must never delete. Run scoping for the staging
+ConfigMap comes from its name (see StagingConfigMapName), which is what both
+Cleanup paths key on.
+
 Job and Pod lifecycle waits use the Kubernetes watch API (not polling) for
 efficiency. Pod selection narrows by label and then authorizes the candidate
 against the controlling ownerReference carrying the recorded Job UID, since pod
 labels are writable by anything that can update pods in the namespace.
 
 # Cleanup
 
-The Deployer records (kind, name, UID) for each object it successfully creates.
-Cleanup deletes exactly that set, passing the recorded UID as a
-metav1.Preconditions so a same-named object belonging to another run is never
-collected; a UID mismatch (Conflict) and a NotFound are both treated as
-success. Cleanup also runs on the Deploy failure path, which is why it is
-scoped to what was created rather than to configured names.
+The Deployer records (kind, name) immediately before each Create and writes the
+returned UID onto that entry on success. Cleanup deletes exactly that set,
+passing the recorded UID as a metav1.Preconditions so a same-named object
+belonging to another run is never collected; a UID mismatch (Conflict) and a
+NotFound are both treated as success. Cleanup also runs on the Deploy failure
+path, which is why it is scoped to what was created rather than to configured
+names.
+
+Recording before the Create is what keeps a lost Create response from orphaning
+an object forever: if the apiserver commits the create but the response never
+arrives, the entry is already in the set and Cleanup deletes it by its
+(run-unique) name with no UID precondition. The one response that proves the
+object is not ours — AlreadyExists — discards the entry again.
 
 The staging ConfigMap is written by the in-pod agent, so its UID is observed
 when GetSnapshot reads it. When the run owns that ConfigMap and failed before