Skip to content
Open
42 changes: 42 additions & 0 deletions docs/user/cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -466,8 +466,50 @@ Generate recipes using direct system parameters:
| `--data` | | string | External data directory to overlay on embedded data (see [External Data](#external-data-directory)) |
| `--criteria-strict` | | bool | Reject criteria values not in the embedded OSS catalog; ignores values registered from `--data`. Also honored via `AICR_CRITERIA_STRICT=1` or `spec.recipe.criteriaStrict: true` in `--config`. Intended for OSS CI gates. |

**Accelerator values name a GPU model, not a machine type.** A provider
usually offers several machine types for the same GPU, and the machine type —
not the GPU — determines the fabric, the NIC count, and which components a
recipe can use. `--accelerator h100` therefore does not, on its own, say which
node shape the resolved recipe targets. See
[Qualified machine types](#qualified-machine-types) below.

> **Service / Accelerator / OS / Intent / Platform value listings above are the OSS-embedded set.** When `--data` registers additional values (e.g., undisclosed providers, proprietary platforms), the CLI admits them at runtime through the criteria registry — see [Data Extension](../integrator/data-extension.md). `--criteria-strict` restores the OSS-only set regardless of what `--data` contributes.

#### Qualified machine types

Each recipe is qualified against a specific node shape. Running it on another
machine type of the same GPU model is not blocked, but it is not qualified, and
the consequence differs by family.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated

| Accelerator | Service | Qualified machine type | On other shapes of the same GPU |
|---|---|---|---|
| `h100` | `gke` | `a3-megagpu-8g` | **Components do not schedule.** The GPUDirect-TCPXO DaemonSets pin node affinity to `cloud.google.com/gke-accelerator: nvidia-h100-mega-80gb`, so on `a3-highgpu-*` / `a3-edgegpu-8g` nothing rolls out and the deployment health check fails. GPUDirect-TCPX is not shipped — tracked in [#2290](https://github.com/NVIDIA/aicr/issues/2290). |
| `h100` | `eks` | `p5.48xlarge` (8× H100 SXM, 32× EFA) | Deploys, but performance floors are calibrated on the full node; smaller shapes such as `p5.4xlarge` can false-fail a healthy run. |
| `h100` | `aks` | `Standard_ND96isr_H100_v5` (8× H100 SXM) | Deploys, but as above; `Standard_NC80adis_H100_v5` (2 GPUs) and `Standard_NC40ads_H100_v5` (1 GPU) can false-fail performance gates. |
| `gb200` | `eks` | `p6e-gb200.36xlarge` (4 GPUs per K8s node) | Deploys; floors are sized for this shape. |
| `a100` | `gke` | `a2-highgpu-*` / `a2-ultragpu-*` | GPUDirect is not applicable; the `gke-nccl-tcpxo` component is intentionally omitted. |
| `b200` | `gke` | A4 shapes | No separate NCCL plugin installer; multi-node NCCL comes from GPU Operator `gdrcopy` plus GKE A4 native multi-NIC. |

Two distinct failure modes are worth separating:

- **Component-level (hard).** Only GKE H100 pins artifacts to a machine type
today. On a non-matching shape the DaemonSets have nowhere to land and
deployment validation fails loudly rather than degrading.
- **Performance-gate (soft).** Elsewhere the recipe deploys normally, but the
NCCL and inference floors are fixed absolute values calibrated on full,
high-bandwidth nodes. They are not normalized for GPU count or fabric class,
so a smaller shape can fail a gate while being perfectly healthy. See
[Validation › Node-shape assumption](./validation.md). Making these gates
fabric/transport-class aware is tracked in
[#1256](https://github.com/NVIDIA/aicr/issues/1256) and
[#1254](https://github.com/NVIDIA/aicr/issues/1254).

Shapes not listed above are undocumented rather than known-broken: they have
not been qualified, and the criteria model has no axis that distinguishes them.
Whether AICR should gain one — finer-grained accelerator values, a machine-type
axis, or a fabric class — is tracked in
[#2377](https://github.com/NVIDIA/aicr/issues/2377).

**Examples:**
```shell
# Basic recipe for Ubuntu on EKS with H100
Expand Down
3 changes: 3 additions & 0 deletions recipes/overlays/gb200-eks-ubuntu-training.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ spec:
# This overlay adds Ubuntu-specific configurations
base: gb200-eks-training

# Machine type: qualified on p6e-gb200.36xlarge (4 GPUs per Kubernetes node).
# The performance floors below are sized for that shape and are not normalized
# for GPU count, so another GB200 shape can false-fail a healthy run.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
criteria:
service: eks
accelerator: gb200
Expand Down
5 changes: 5 additions & 0 deletions recipes/overlays/h100-aks-training.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,11 @@ spec:
# Inherits from aks-training recipe (AKS + training settings)
base: aks-training

# Machine type: qualified on Standard_ND96isr_H100_v5 (8x H100 SXM). These
# criteria carry no machine-type axis, so they also match the smaller NCads
# H100 shapes (Standard_NC80adis_H100_v5 = 2 GPUs, Standard_NC40ads_H100_v5 =
# 1 GPU). Those deploy normally, but the performance floors below are fixed
# absolute full-node values and can false-fail on them. See #1256/#1254.
criteria:
service: aks
accelerator: h100
Expand Down
6 changes: 6 additions & 0 deletions recipes/overlays/h100-eks-training.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,12 @@ spec:
# Inherits from eks-training recipe (EKS + training settings)
base: eks-training

# Machine type: qualified on p5.48xlarge (8x H100 SXM, 32x EFA). These
# criteria carry no machine-type axis, so they also match smaller H100 shapes
# (e.g. p5.4xlarge). Those deploy normally — unlike the GKE leaf, no component
# here pins node affinity to a shape — but the performance floors below are
# fixed absolute values calibrated on the full node, so a smaller shape can
# false-fail a healthy run. Normalizing the gates is tracked in #1256/#1254.
criteria:
service: eks
accelerator: h100
Expand Down
4 changes: 4 additions & 0 deletions recipes/overlays/h100-gke-cos-training-slurm.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ spec:
# `aicr bundle ... --set slinkyslurm:...` or a valuesFile.
base: h100-gke-cos-training

# Machine type: a3-megagpu-8g only, inherited from the parent leaf along with
# its gke-nccl-tcpxo componentRef. These criteria add no machine-type axis, so
# the parent's TCPXO-vs-TCPX selection gap applies here unchanged. See the
# parent overlay's note and #2290.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
criteria:
service: gke
accelerator: h100
Expand Down
14 changes: 14 additions & 0 deletions recipes/overlays/h100-gke-cos-training.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,20 @@ spec:
# Inherits from gke-cos-training recipe (GKE COS + training settings)
base: gke-cos-training

# Machine type: this leaf is qualified on a3-megagpu-8g only.
#
# `accelerator: h100` names the GPU model, and GKE offers it on several A3
# machine types with different fabrics: a3-megagpu-8g uses GPUDirect-TCPXO,
# a3-highgpu-8g / a3-edgegpu-8g use GPUDirect-TCPX, and the smaller
# a3-highgpu-1g/2g/4g shapes have no GPUDirect at all. The criteria model has
# no axis that distinguishes them, so these criteria match all of them while
# the gke-nccl-tcpxo DaemonSets below pin node affinity to
# `cloud.google.com/gke-accelerator: nvidia-h100-mega-80gb`.
#
# On a non-mega shape nothing schedules and the deployment health check fails
# loudly (`desiredNumberScheduled > 0`). TCPX is not shipped. Selection fix
# and a TCPX component are tracked in #2290; whether criteria should gain a
# machine-type or fabric axis is tracked in #2377.
criteria:
service: gke
accelerator: h100
Expand Down
Loading