Skip to content

docs(user): document the machine type each accelerator criterion targets #2380

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
yuanchen8911 wants to merge 8 commits into
base: main
Choose a base branch
from
Open

docs(user): document the machine type each accelerator criterion targets #2380

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
4f78a6c
docs(user): document the machine type each accelerator criterion targets
yuanchen8911 Aug 25, 2026
f9a5d90
docs(user): correct issue references and section placement
yuanchen8911 Aug 25, 2026
25b0c7a
docs(user): separate criteria resolution from deployment outcome
yuanchen8911 Aug 25, 2026
a7486d8
docs(user): scope the GKE H100 hard failure to the training lineage
yuanchen8911 Aug 25, 2026
7d12eff
docs(user): state family-level rows explicitly instead of glob patterns
yuanchen8911 Aug 25, 2026
e42e972
docs(recipes): correct the transport named in the B200 GKE comment
yuanchen8911 Aug 25, 2026
d48a78f
docs(user): distinguish plain GKE H100 inference from the dynamo variant
yuanchen8911 Aug 25, 2026
3f0ee2e
docs(user): AKS H100 training hard-fails deployment on non-IB NCads s…
yuanchen8911 Aug 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions docs/user/cli-reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -466,6 +466,13 @@ Generate recipes using direct system parameters:
| `--data` | | string | External data directory to overlay on embedded data (see [External Data](#external-data-directory)) |
| `--criteria-strict` | | bool | Reject criteria values not in the embedded OSS catalog; ignores values registered from `--data`. Also honored via `AICR_CRITERIA_STRICT=1` or `spec.recipe.criteriaStrict: true` in `--config`. Intended for OSS CI gates. |

**Accelerator values name a GPU model, not a machine type.** A provider
usually offers several machine types for the same GPU, and the machine type —
not the GPU — determines the fabric, the NIC count, and which components a
recipe can use. `--accelerator h100` therefore does not, on its own, say which
node shape the resolved recipe targets. See
[Qualified machine types](#qualified-machine-types) below.

> **Service / Accelerator / OS / Intent / Platform value listings above are the OSS-embedded set.** When `--data` registers additional values (e.g., undisclosed providers, proprietary platforms), the CLI admits them at runtime through the criteria registry — see [Data Extension](../integrator/data-extension.md). `--criteria-strict` restores the OSS-only set regardless of what `--data` contributes.

**Examples:**
Expand Down Expand Up @@ -510,6 +517,45 @@ proceeds for compatibility. Use a current snapshot before deployment when you
need conflict detection. See
[Conflict detection requires snapshot evidence](slinky-slurm-accounting.md#conflict-detection-requires-snapshot-evidence).

#### Qualified machine types

@njhensley njhensley Aug 25, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Nitpick — Heading capitalization diverges from immediate siblings

#### Qualified machine types is sentence case while its two peers under ### aicr recipe are Title Case (#### Query Mode, #### Snapshot Mode). The file mixes both and CLAUDE.md doesn't mandate Title Case, so this is defensible. The anchor stays #qualified-machine-types under GitHub's lowercasing either way, so the intra-page link is unaffected.

Fix: Optional: #### Qualified Machine Types for local consistency.

@yuanchen8911 yuanchen8911 Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to #### Qualified Machine Types in 3f0ee2e to match its #### Query Mode / #### Snapshot Mode siblings. As you note the anchor is unaffected — GitHub lowercases either way — so the intra-page link needed no change, and I verified it still points at #qualified-machine-types.


Each recipe is qualified against a specific node shape. Running it on another
machine type of the same GPU model is not blocked, but it is not qualified, and
the consequence differs by family.

| Accelerator | Service | Qualified machine type | On other shapes of the same GPU |
|---|---|---|---|
| `h100` | `gke` | `a3-megagpu-8g` | **Components do not schedule.** The GPUDirect-TCPXO DaemonSets pin node affinity to `cloud.google.com/gke-accelerator: nvidia-h100-mega-80gb`, so on `a3-highgpu-*` / `a3-edgegpu-8g` nothing rolls out and the deployment health check fails. GPUDirect-TCPX is not shipped — tracked in [#2290](https://github.com/NVIDIA/aicr/issues/2290). |
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
| `h100` | `eks` | `p5.48xlarge` (8× H100 SXM, 32× EFA) | Deploys, but performance floors are calibrated on the full node; smaller shapes such as `p5.4xlarge` can false-fail a healthy run. |
| `h100` | `aks` | `Standard_ND96isr_H100_v5` (8× H100 SXM) | Deploys, but as above; `Standard_NC80adis_H100_v5` (2 GPUs) and `Standard_NC40ads_H100_v5` (1 GPU) can false-fail performance gates. |

@njhensley njhensley Aug 25, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Major — AKS H100 training hard-fails deployment on the NCads shapes, not a perf soft-fail

This row says Standard_NC80adis_H100_v5 (2 GPU) and Standard_NC40ads_H100_v5 (1 GPU) "Deploys, but … can false-fail performance gates." The same claim is in recipes/overlays/h100-aks-training.yaml:27 and is generalized in the summary bullet at line 546 ("Only the GKE H100 training lineage … carries a machine-type-bound component"). It's the opposite for those shapes.

h100-aks-training → aks-training → aks; aks.yaml:175-182 wires network-operator with nic-cluster-policy-aks.yaml unconditionally (RDMA on by default, aks.yaml:171). The expected-resources deployment check this recipe runs (h100-aks-training.yaml:85) calls verifyRDMAFabricReady whenever a network-operator ref declares that NicClusterPolicy manifest (validators/deployment/expected_resources.go:434, recipeDeclaresRDMAFabric). rdmaFabricProbeCoverage fails closed: empty Mellanox-RDMA cohort → ErrCodeNotFound (expected_resources.go:1194-1198); present-but-zero allocatable → error (:1208-1213). NC40ads/NC80adis are PCIe H100 with no InfiniBand, so they never advertise rdma/hca_shared_devices_a — the deployment phase times out and hard-fails before the performance gate ever runs.

The EKS row by contrast is genuinely correct (aws-efa has no requiredDuringScheduling affinity and no Go gate), so the GKE↔EKS asymmetry holds — but AKS is a second hard-fail path, routed through a Go readiness gate instead of a Chainsaw one, that this table presents as a clean deploy.

Blast radius: A user on a 1-/2-GPU NCads H100 AKS cluster is told deployment will succeed and only a perf gate might false-fail; in reality aicr validate hard-fails the deployment phase at expected-resources after the full RDMA-readiness timeout. Also weakens the load-bearing "only GKE H100 training hard-fails" framing.

Fix: State that the non-IB NCads shapes hard-fail the deployment RDMA-fabric readiness gate (network-operator NicClusterPolicy + verifyRDMAFabricReady), and qualify the line-546 bullet to acknowledge the AKS RDMA fabric gate — or scope the AKS "deploys normally" claim to the IB-capable ND shapes only. Mirror the wording into h100-aks-training.yaml:24-28.

@yuanchen8911 yuanchen8911 Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed — this was a real defect in the table, and I traced the chain before changing it:

  1. aks.yaml:175-182 wires network-operator with nic-cluster-policy-aks.yaml unconditionally.
  2. recipeDeclaresRDMAFabric (expected_resources.go:948-955) matches on that manifest name, so verifyRDMAFabricReady runs in the deployment phase (:434).
  3. It fails closed — empty Mellanox cohort returns ErrCodeNotFound (:1194-1198); present-but-zero errors (:1208-1213).

NCads are PCIe H100 with no InfiniBand, so no node carries the Mellanox label, the cohort is empty, and deployment fails before any performance gate runs.

Fixed in 3f0ee2e: the AKS row now states the deployment RDMA-fabric gate and names the --set networkoperator:enabled=false opt-out, the h100-aks-training.yaml comment says the same, and the summary bullet now describes two hard-fail families by different mechanisms — GKE through a Chainsaw check on unschedulable DaemonSets, AKS through a Go readiness gate on an absent fabric. The soft-failure bullet is scoped to EKS/GB200, noting that on AKS the deployment gate bites first.

Worth recording why this was missed: an earlier review pass did check the AKS path, but only the Chainsaw health check — which indeed has no desiredNumberScheduled > 0 assert and passes. The gate is in Go, on a different mechanism entirely.

| `gb200` | `eks` | `p6e-gb200.36xlarge` (4 GPUs per K8s node) | Deploys; floors are sized for this shape and are themselves provisional pending production NVL72 data. |
| `a100` | `gke` | `a2-highgpu-*` / `a2-ultragpu-*` | GPUDirect is not applicable; the `gke-nccl-tcpxo` component is intentionally omitted. |
| `b200` | `gke` | A4 shapes | No separate NCCL plugin installer; multi-node NCCL comes from GPU Operator `gdrcopy` plus GKE A4 native multi-NIC. |
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated

Two distinct failure modes are worth separating:

- **Component-level (hard).** Only GKE H100 pins artifacts to a machine type
today. On a non-matching shape the DaemonSets have nowhere to land and
deployment validation fails loudly rather than degrading.
- **Performance-gate (soft).** Elsewhere the recipe deploys normally, but the
NCCL and inference floors are fixed absolute values calibrated on full,
high-bandwidth nodes. They are not normalized for GPU count or fabric class,
so a smaller shape can fail a gate while being perfectly healthy. See
[Validation › Node-shape assumption](./validation.md). Normalizing these
floors per GPU or per fabric class was considered and declined
([#1256](https://github.com/NVIDIA/aicr/issues/1256),
[#1254](https://github.com/NVIDIA/aicr/issues/1254), both closed as not
planned) — the floors are deliberately fixed absolute full-node values, so
running a qualified shape is the supported way to pass them.

The table lists the accelerator/service pairs that have a qualified shape;
a pair or a shape absent from it is **undocumented rather than known-broken**.
It has not been qualified, and the criteria model has no axis that would
distinguish it from one that has.
Whether AICR should gain one — finer-grained accelerator values, a machine-type
axis, or a fabric class — is tracked in
[#2377](https://github.com/NVIDIA/aicr/issues/2377).

#### Snapshot Mode

Generate recipes from captured snapshots:
Expand Down
5 changes: 5 additions & 0 deletions recipes/overlays/gb200-eks-ubuntu-training.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,11 @@ spec:
# This overlay adds Ubuntu-specific configurations
base: gb200-eks-training

# Machine type: qualified on p6e-gb200.36xlarge (4 GPUs per Kubernetes node).
# The performance floors inherited from gb200-eks-training are sized for that
# shape and are not normalized for GPU count, so another GB200 shape can
# false-fail a healthy run. Those thresholds are themselves provisional --
# see the parent overlay's note.
criteria:
service: eks
accelerator: gb200
Expand Down
7 changes: 7 additions & 0 deletions recipes/overlays/h100-aks-training.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,13 @@ spec:
# Inherits from aks-training recipe (AKS + training settings)
base: aks-training

# Machine type: qualified on Standard_ND96isr_H100_v5 (8x H100 SXM). These
# criteria carry no machine-type axis, so they also match the smaller NCads
# H100 shapes (Standard_NC80adis_H100_v5 = 2 GPUs, Standard_NC40ads_H100_v5 =
# 1 GPU). Those deploy normally, but the performance floors below are fixed
# absolute full-node values and can false-fail on them. Normalizing them was
# declined (#1256, #1254 — both closed not-planned); the qualified shape is
# the supported way to pass them.
criteria:
service: aks
accelerator: h100
Expand Down
8 changes: 8 additions & 0 deletions recipes/overlays/h100-eks-training.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,14 @@ spec:
# Inherits from eks-training recipe (EKS + training settings)
base: eks-training

# Machine type: qualified on p5.48xlarge (8x H100 SXM, 32x EFA). These
# criteria carry no machine-type axis, so they also match smaller H100 shapes
# (e.g. p5.4xlarge). Those deploy normally — unlike the GKE leaf, no component
# here pins node affinity to a shape — but the performance floors below are
# fixed absolute values calibrated on the full node, so a smaller shape can
# false-fail a healthy run. Normalizing them per GPU/fabric class was declined
# (#1256, #1254 — both closed not-planned); the floors are deliberately
# absolute, so the qualified shape is the supported way to pass them.
criteria:
service: eks
accelerator: h100
Expand Down
4 changes: 4 additions & 0 deletions recipes/overlays/h100-gke-cos-training-slurm.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ spec:
# `aicr bundle ... --set slinkyslurm:...` or a valuesFile.
base: h100-gke-cos-training

# Machine type: a3-megagpu-8g only, inherited from the parent leaf along with
# its gke-nccl-tcpxo componentRef. These criteria add no machine-type axis, so
# the parent's TCPXO-vs-TCPX selection gap applies here unchanged. See the
# parent overlay's note and #2290.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
criteria:
service: gke
accelerator: h100
Expand Down
14 changes: 14 additions & 0 deletions recipes/overlays/h100-gke-cos-training.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,20 @@ spec:
# Inherits from gke-cos-training recipe (GKE COS + training settings)
base: gke-cos-training

# Machine type: this leaf is qualified on a3-megagpu-8g only.
#
# `accelerator: h100` names the GPU model, and GKE offers it on several A3
# machine types with different fabrics: a3-megagpu-8g uses GPUDirect-TCPXO,
# a3-highgpu-8g / a3-edgegpu-8g use GPUDirect-TCPX, and the smaller
# a3-highgpu-1g/2g/4g shapes have no GPUDirect at all. The criteria model has
# no axis that distinguishes them, so these criteria match all of them while
# the gke-nccl-tcpxo DaemonSets below pin node affinity to
# `cloud.google.com/gke-accelerator: nvidia-h100-mega-80gb`.
#
# On a non-mega shape nothing schedules and the deployment health check fails
# loudly (`desiredNumberScheduled > 0`). TCPX is not shipped. Selection fix
# and a TCPX component are tracked in #2290; whether criteria should gain a
# machine-type or fabric axis is tracked in #2377.
criteria:
service: gke
accelerator: h100
Expand Down
Loading