Skip to content

docs(user): document the machine type each accelerator criterion targets #2380

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
yuanchen8911 wants to merge 8 commits into
base: main
Choose a base branch
from
Open

docs(user): document the machine type each accelerator criterion targets #2380

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
4f78a6c
docs(user): document the machine type each accelerator criterion targets
yuanchen8911 Aug 25, 2026
f9a5d90
docs(user): correct issue references and section placement
yuanchen8911 Aug 25, 2026
25b0c7a
docs(user): separate criteria resolution from deployment outcome
yuanchen8911 Aug 25, 2026
a7486d8
docs(user): scope the GKE H100 hard failure to the training lineage
yuanchen8911 Aug 25, 2026
7d12eff
docs(user): state family-level rows explicitly instead of glob patterns
yuanchen8911 Aug 25, 2026
e42e972
docs(recipes): correct the transport named in the B200 GKE comment
yuanchen8911 Aug 25, 2026
d48a78f
docs(user): distinguish plain GKE H100 inference from the dynamo variant
yuanchen8911 Aug 25, 2026
3f0ee2e
docs(user): AKS H100 training hard-fails deployment on non-IB NCads s…
yuanchen8911 Aug 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions docs/user/cli-reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -466,6 +466,13 @@ Generate recipes using direct system parameters:
| `--data` | | string | External data directory to overlay on embedded data (see [External Data](#external-data-directory)) |
| `--criteria-strict` | | bool | Reject criteria values not in the embedded OSS catalog; ignores values registered from `--data`. Also honored via `AICR_CRITERIA_STRICT=1` or `spec.recipe.criteriaStrict: true` in `--config`. Intended for OSS CI gates. |

**Accelerator values name a GPU model, not a machine type.** A provider
usually offers several machine types for the same GPU, and the machine type —
not the GPU — determines the fabric, the NIC count, and which components a
recipe can use. `--accelerator h100` therefore does not, on its own, say which
node shape the resolved recipe targets. See
[Qualified machine types](#qualified-machine-types) below.

> **Service / Accelerator / OS / Intent / Platform value listings above are the OSS-embedded set.** When `--data` registers additional values (e.g., undisclosed providers, proprietary platforms), the CLI admits them at runtime through the criteria registry — see [Data Extension](../integrator/data-extension.md). `--criteria-strict` restores the OSS-only set regardless of what `--data` contributes.

**Examples:**
Expand Down Expand Up @@ -510,6 +517,56 @@ proceeds for compatibility. Use a current snapshot before deployment when you
need conflict detection. See
[Conflict detection requires snapshot evidence](slinky-slurm-accounting.md#conflict-detection-requires-snapshot-evidence).

#### Qualified machine types

@njhensley njhensley Aug 25, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Nitpick — Heading capitalization diverges from immediate siblings

#### Qualified machine types is sentence case while its two peers under ### aicr recipe are Title Case (#### Query Mode, #### Snapshot Mode). The file mixes both and CLAUDE.md doesn't mandate Title Case, so this is defensible. The anchor stays #qualified-machine-types under GitHub's lowercasing either way, so the intra-page link is unaffected.

Fix: Optional: #### Qualified Machine Types for local consistency.

@yuanchen8911 yuanchen8911 Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to #### Qualified Machine Types in 3f0ee2e to match its #### Query Mode / #### Snapshot Mode siblings. As you note the anchor is unaffected — GitHub lowercases either way — so the intra-page link needed no change, and I verified it still points at #qualified-machine-types.


Each recipe is qualified against a specific node shape. Criteria resolution
does not reject another machine type of the same GPU model — there is no axis
to reject it on — so a recipe always resolves. What differs by family is what
happens afterwards: on some, deployment validation fails; on others it succeeds
and only the performance gates are affected.

| Accelerator | Service / intent | Qualified machine type | On other shapes of the same GPU |
|---|---|---|---|
| `h100` | `gke`, `training` | `a3-megagpu-8g` | **Components do not schedule.** The GPUDirect-TCPXO DaemonSets pin node affinity to `cloud.google.com/gke-accelerator: nvidia-h100-mega-80gb`, so on `a3-highgpu-*` / `a3-edgegpu-8g` nothing rolls out and the deployment health check fails. AICR ships no GPUDirect-TCPX component for the shapes that need one — tracked in [#2290](https://github.com/NVIDIA/aicr/issues/2290). |
| `h100` | `gke`, `inference` | `a3-megagpu-8g` | Deploys. The inference lineage carries no `gke-nccl-tcpxo` component, so the hard failure above does not apply. Plain `inference` declares no performance gates at all; the `dynamo` variant adds floors calibrated on the 8-GPU node, so smaller shapes such as `a3-highgpu-1g/2g/4g` can false-fail there. |

@njhensley njhensley Aug 25, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Nitpick — "Qualified = a3-megagpu-8g" is over-precise for the plain GKE H100 inference row

For the plain gke, inference half, the recipe carries no gke-nccl-tcpxo pin and no performance floor (base gke-cos-inference declares no validation phase), so it behaves identically on every H100 shape — a3-megagpu-8g is only meaningful for the dynamo sub-lineage. The adjacent "On other shapes" cell already makes the plain-vs-dynamo split, so this is internally consistent; the single-value column is just slightly over-precise.

Fix: Optional: annotate the value as a3-megagpu-8g (dynamo variant only), or footnote that plain inference is shape-agnostic.

@yuanchen8911 yuanchen8911 Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed — fixed in 3f0ee2e. The plain gke, inference half depends on no machine type, so naming one implied a constraint that isn't there. The cell now reads "not machine-type-bound (dynamo floors calibrated on a3-megagpu-8g)", which keeps the shape visible for the sub-lineage that does depend on it without asserting it for the plain path.

| `h100` | `eks` | `p5.48xlarge` (8× H100 SXM, 32× EFA) | Deploys, but performance floors are calibrated on the full node; smaller shapes such as `p5.4xlarge` can false-fail a healthy run. |
| `h100` | `aks` | `Standard_ND96isr_H100_v5` (8× H100 SXM) | Deploys, but as above; `Standard_NC80adis_H100_v5` (2 GPUs) and `Standard_NC40ads_H100_v5` (1 GPU) can false-fail performance gates. |

@njhensley njhensley Aug 25, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Major — AKS H100 training hard-fails deployment on the NCads shapes, not a perf soft-fail

This row says Standard_NC80adis_H100_v5 (2 GPU) and Standard_NC40ads_H100_v5 (1 GPU) "Deploys, but … can false-fail performance gates." The same claim is in recipes/overlays/h100-aks-training.yaml:27 and is generalized in the summary bullet at line 546 ("Only the GKE H100 training lineage … carries a machine-type-bound component"). It's the opposite for those shapes.

h100-aks-training → aks-training → aks; aks.yaml:175-182 wires network-operator with nic-cluster-policy-aks.yaml unconditionally (RDMA on by default, aks.yaml:171). The expected-resources deployment check this recipe runs (h100-aks-training.yaml:85) calls verifyRDMAFabricReady whenever a network-operator ref declares that NicClusterPolicy manifest (validators/deployment/expected_resources.go:434, recipeDeclaresRDMAFabric). rdmaFabricProbeCoverage fails closed: empty Mellanox-RDMA cohort → ErrCodeNotFound (expected_resources.go:1194-1198); present-but-zero allocatable → error (:1208-1213). NC40ads/NC80adis are PCIe H100 with no InfiniBand, so they never advertise rdma/hca_shared_devices_a — the deployment phase times out and hard-fails before the performance gate ever runs.

The EKS row by contrast is genuinely correct (aws-efa has no requiredDuringScheduling affinity and no Go gate), so the GKE↔EKS asymmetry holds — but AKS is a second hard-fail path, routed through a Go readiness gate instead of a Chainsaw one, that this table presents as a clean deploy.

Blast radius: A user on a 1-/2-GPU NCads H100 AKS cluster is told deployment will succeed and only a perf gate might false-fail; in reality aicr validate hard-fails the deployment phase at expected-resources after the full RDMA-readiness timeout. Also weakens the load-bearing "only GKE H100 training hard-fails" framing.

Fix: State that the non-IB NCads shapes hard-fail the deployment RDMA-fabric readiness gate (network-operator NicClusterPolicy + verifyRDMAFabricReady), and qualify the line-546 bullet to acknowledge the AKS RDMA fabric gate — or scope the AKS "deploys normally" claim to the IB-capable ND shapes only. Mirror the wording into h100-aks-training.yaml:24-28.

@yuanchen8911 yuanchen8911 Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed — this was a real defect in the table, and I traced the chain before changing it:

  1. aks.yaml:175-182 wires network-operator with nic-cluster-policy-aks.yaml unconditionally.
  2. recipeDeclaresRDMAFabric (expected_resources.go:948-955) matches on that manifest name, so verifyRDMAFabricReady runs in the deployment phase (:434).
  3. It fails closed — empty Mellanox cohort returns ErrCodeNotFound (:1194-1198); present-but-zero errors (:1208-1213).

NCads are PCIe H100 with no InfiniBand, so no node carries the Mellanox label, the cohort is empty, and deployment fails before any performance gate runs.

Fixed in 3f0ee2e: the AKS row now states the deployment RDMA-fabric gate and names the --set networkoperator:enabled=false opt-out, the h100-aks-training.yaml comment says the same, and the summary bullet now describes two hard-fail families by different mechanisms — GKE through a Chainsaw check on unschedulable DaemonSets, AKS through a Go readiness gate on an absent fabric. The soft-failure bullet is scoped to EKS/GB200, noting that on AKS the deployment gate bites first.

Worth recording why this was missed: an earlier review pass did check the AKS path, but only the Chainsaw health check — which indeed has no desiredNumberScheduled > 0 assert and passes. The gate is in Go, on a different mechanism entirely.

| `gb200` | `eks` | `p6e-gb200.36xlarge` (4 GPUs per K8s node) | Deploys; floors are sized for this shape and are themselves provisional pending production NVL72 data. |
| `a100` | `gke` | the whole `a2` family (`a2-highgpu-*`, `a2-ultragpu-*`) | Family-level by construction, not per-shape: GPUDirect-TCPXO targets H100 `a3-megagpu-8g`, so the `gke-nccl-tcpxo` component is inapplicable to every `a2` shape and is intentionally omitted. No shape in the family carries a machine-type-bound component. |

@njhensley njhensley Aug 25, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Nitpick — a2 (lowercase) vs A4 (uppercase) in the same column reads as an inconsistency

The a100 row says "the whole a2 family" (lowercase) while the b200 row (line 536) says "the A4 family" (uppercase), both in code font in the same column. This tracks real GCP branding (lowercase a2/a4 machine-type identifiers vs uppercase A2/A4 series names), so it's arguably correct — but side by side it reads as a typo. The b200-gke-cos-training.yaml comment uses the same A4, so doc and overlay agree.

Fix: Optional: lowercase A4 → a4 for visual parity, or add a one-word note distinguishing series from machine type.

@yuanchen8911 yuanchen8911 Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made both lowercase in 3f0ee2e. Since the column names machine-type families rather than marketing series, the lowercase identifier form (a2, a4) is the consistent choice — and it matches how the a3-megagpu-8g / p5.48xlarge rows are written in the same column.

| `b200` | `gke` | the `A4` family — **specific machine type not recorded** | No separate NCCL plugin installer; multi-node NCCL comes from GPU Operator `gdrcopy` plus GKE A4's GCP-managed multi-NIC, so nothing here is machine-type-bound. The overlay records a production reference cluster but no machine type, so this row cannot name one. |

A row that names no intent applies to every intent for that accelerator and
service. Where a row names a family rather than a machine type, the entry is a
family-level statement — either because no component in that family binds to a
machine type, or because the specific shape is not recorded in-repo. The row
says which.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

Two distinct failure modes are worth separating:

- **Component-level (hard).** Only the GKE H100 **training** lineage pins
artifacts to a machine type today — `h100-gke-cos-training` and the leaves
that inherit from it. On a non-matching shape the DaemonSets have nowhere to
land and deployment validation fails loudly rather than degrading. No other
family, and no inference recipe, carries a machine-type-bound component.
- **Performance-gate (soft).** Elsewhere the recipe deploys normally, but the
NCCL and inference floors are fixed absolute values calibrated on full,
high-bandwidth nodes. They are not normalized for GPU count or fabric class,
so a smaller shape can fail a gate while being perfectly healthy. See
[Validation › Node-shape assumption](./validation.md). Normalizing these
floors per GPU or per fabric class was considered and declined
([#1256](https://github.com/NVIDIA/aicr/issues/1256),
[#1254](https://github.com/NVIDIA/aicr/issues/1254), both closed as not
planned) — the floors are deliberately fixed absolute full-node values, so
running a qualified shape is the supported way to pass them.

The table lists the accelerator/service pairs that have a qualified shape;
a pair or a shape absent from it is **undocumented rather than known-broken**.
It has not been qualified, and the criteria model has no axis that would
distinguish it from one that has.
Whether AICR should gain one — finer-grained accelerator values, a machine-type
axis, or a fabric class — is tracked in
[#2377](https://github.com/NVIDIA/aicr/issues/2377).

#### Snapshot Mode

Generate recipes from captured snapshots:
Expand Down
6 changes: 4 additions & 2 deletions recipes/overlays/b200-gke-cos-training.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,10 @@ spec:
# enabled, combined with GKE A4's native multi-NIC infrastructure
# managed by GCP. The `gke-nccl-tcpxo` component is intentionally
# omitted because its DaemonSets pin to `cloud.google.com/gke-
# accelerator: nvidia-h100-mega-80gb` (a3-megagpu / H100) and a
# different transport (TCPX), so they would not run on A4 nodes.
# accelerator: nvidia-h100-mega-80gb` (a3-megagpu / H100) and carry a
# transport A4 does not use (GPUDirect-TCPXO), so they would not run on A4

@njhensley njhensley Aug 26, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Nitpick — a4 (doc) vs A4 (b200 overlay) casing drift

The prior-round P4 fix lowercased the doc table to a4, but this overlay comment still says A4 (here and lines 40, 44, 49, 94) — and the PR body states doc and overlay agree.

Fix: Optional. Defensible as-is (uppercase A4 = series name in prose; lowercase a4 = machine-type family in code font). Align only if doc↔overlay parity is the goal.

# nodes. (TCPXO is the a3-megagpu transport; TCPX is the separate
# a3-highgpu/edge one. Neither applies here.)
- name: gpu-operator
type: Helm
dependencyRefs:
Expand Down
5 changes: 5 additions & 0 deletions recipes/overlays/gb200-eks-ubuntu-training.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,11 @@ spec:
# This overlay adds Ubuntu-specific configurations
base: gb200-eks-training

# Machine type: qualified on p6e-gb200.36xlarge (4 GPUs per Kubernetes node).
# The performance floors inherited from gb200-eks-training are sized for that
# shape and are not normalized for GPU count, so another GB200 shape can
# false-fail a healthy run. Those thresholds are themselves provisional --
# see the parent overlay's note.
criteria:
service: eks
accelerator: gb200
Expand Down
7 changes: 7 additions & 0 deletions recipes/overlays/h100-aks-training.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,13 @@ spec:
# Inherits from aks-training recipe (AKS + training settings)
base: aks-training

# Machine type: qualified on Standard_ND96isr_H100_v5 (8x H100 SXM). These
# criteria carry no machine-type axis, so they also match the smaller NCads
# H100 shapes (Standard_NC80adis_H100_v5 = 2 GPUs, Standard_NC40ads_H100_v5 =
# 1 GPU). Those deploy normally, but the performance floors below are fixed
# absolute full-node values and can false-fail on them. Normalizing them was
# declined (#1256, #1254 — both closed not-planned); the qualified shape is
# the supported way to pass them.
criteria:
service: aks
accelerator: h100
Expand Down
8 changes: 8 additions & 0 deletions recipes/overlays/h100-eks-training.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,14 @@ spec:
# Inherits from eks-training recipe (EKS + training settings)
base: eks-training

# Machine type: qualified on p5.48xlarge (8x H100 SXM, 32x EFA). These
# criteria carry no machine-type axis, so they also match smaller H100 shapes
# (e.g. p5.4xlarge). Those deploy normally — unlike the GKE leaf, no component
# here pins node affinity to a shape — but the performance floors below are
# fixed absolute values calibrated on the full node, so a smaller shape can
# false-fail a healthy run. Normalizing them per GPU/fabric class was declined
# (#1256, #1254 — both closed not-planned); the floors are deliberately
# absolute, so the qualified shape is the supported way to pass them.
criteria:
service: eks
accelerator: h100
Expand Down
9 changes: 9 additions & 0 deletions recipes/overlays/h100-gke-cos-training-slurm.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,15 @@ spec:
# `aicr bundle ... --set slinkyslurm:...` or a valuesFile.
base: h100-gke-cos-training

# Machine type: qualified on a3-megagpu-8g only, same as the parent.
#
# Two different things carry over from `base`, and only one of them is
# inheritance: the gke-nccl-tcpxo componentRef is inherited by the base
# mechanism, while the machine-type qualification is a documented assumption
# that holds here because those same artifacts do. There is no machine-type
# axis in the criteria contract to inherit -- these criteria add none, so the
# parent's TCPXO-vs-TCPX selection gap applies unchanged. See the parent
# overlay's note and #2290.
criteria:
service: gke
accelerator: h100
Expand Down
15 changes: 15 additions & 0 deletions recipes/overlays/h100-gke-cos-training.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,21 @@ spec:
# Inherits from gke-cos-training recipe (GKE COS + training settings)
base: gke-cos-training

# Machine type: this leaf is qualified on a3-megagpu-8g only.
#
# `accelerator: h100` names the GPU model, and GKE offers it on several A3
# machine types with different fabrics: a3-megagpu-8g uses GPUDirect-TCPXO,
# a3-highgpu-8g / a3-edgegpu-8g use GPUDirect-TCPX, and the smaller
# a3-highgpu-1g/2g/4g shapes have no GPUDirect at all. The criteria model has
# no axis that distinguishes them, so these criteria match all of them while
# the gke-nccl-tcpxo DaemonSets below pin node affinity to
# `cloud.google.com/gke-accelerator: nvidia-h100-mega-80gb`.
#
# On a non-mega shape nothing schedules and the deployment health check fails
# loudly (`desiredNumberScheduled > 0`). AICR ships no GPUDirect-TCPX
# component, so those shapes have no working alternative. Selection fix and a
# TCPX component are tracked in #2290; whether criteria should gain a
# machine-type or fabric axis is tracked in #2377.
criteria:
service: gke
accelerator: h100
Expand Down
Loading