Skip to content

Commit f9a5d90

Browse files
committed
docs(user): correct issue references and section placement
Review found three defects in the previous commit. #1256 and #1254 are closed as not-planned, so describing the normalized performance floors as 'tracked in' them asserted follow-up work that was in fact declined. Reword to state the decision: the floors are deliberately fixed absolute full-node values, and running a qualified shape is how they pass. The new section was inserted at the same heading level as 'Query Mode' and before its examples block, so those examples rendered under the new heading. Move the section after them, keeping it a sibling before 'Snapshot Mode'. The gb200-eks-ubuntu-training comment said 'the performance floors below' but that overlay carries none — they are inherited from gb200-eks-training, and are provisional. Say so, and note the same on the table row. Signed-off-by: Yuan Chen <yuanchen97@gmail.com>
1 parent 4f78a6c commit f9a5d90

4 files changed

Lines changed: 49 additions & 39 deletions

File tree

docs/user/cli-reference.md

Lines changed: 39 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -475,41 +475,6 @@ node shape the resolved recipe targets. See
475475

476476
> **Service / Accelerator / OS / Intent / Platform value listings above are the OSS-embedded set.** When `--data` registers additional values (e.g., undisclosed providers, proprietary platforms), the CLI admits them at runtime through the criteria registry — see [Data Extension](../integrator/data-extension.md). `--criteria-strict` restores the OSS-only set regardless of what `--data` contributes.
477477

478-
#### Qualified machine types
479-
480-
Each recipe is qualified against a specific node shape. Running it on another
481-
machine type of the same GPU model is not blocked, but it is not qualified, and
482-
the consequence differs by family.
483-
484-
| Accelerator | Service | Qualified machine type | On other shapes of the same GPU |
485-
|---|---|---|---|
486-
| `h100` | `gke` | `a3-megagpu-8g` | **Components do not schedule.** The GPUDirect-TCPXO DaemonSets pin node affinity to `cloud.google.com/gke-accelerator: nvidia-h100-mega-80gb`, so on `a3-highgpu-*` / `a3-edgegpu-8g` nothing rolls out and the deployment health check fails. GPUDirect-TCPX is not shipped — tracked in [#2290](https://github.com/NVIDIA/aicr/issues/2290). |
487-
| `h100` | `eks` | `p5.48xlarge` (8× H100 SXM, 32× EFA) | Deploys, but performance floors are calibrated on the full node; smaller shapes such as `p5.4xlarge` can false-fail a healthy run. |
488-
| `h100` | `aks` | `Standard_ND96isr_H100_v5` (8× H100 SXM) | Deploys, but as above; `Standard_NC80adis_H100_v5` (2 GPUs) and `Standard_NC40ads_H100_v5` (1 GPU) can false-fail performance gates. |
489-
| `gb200` | `eks` | `p6e-gb200.36xlarge` (4 GPUs per K8s node) | Deploys; floors are sized for this shape. |
490-
| `a100` | `gke` | `a2-highgpu-*` / `a2-ultragpu-*` | GPUDirect is not applicable; the `gke-nccl-tcpxo` component is intentionally omitted. |
491-
| `b200` | `gke` | A4 shapes | No separate NCCL plugin installer; multi-node NCCL comes from GPU Operator `gdrcopy` plus GKE A4 native multi-NIC. |
492-
493-
Two distinct failure modes are worth separating:
494-
495-
- **Component-level (hard).** Only GKE H100 pins artifacts to a machine type
496-
today. On a non-matching shape the DaemonSets have nowhere to land and
497-
deployment validation fails loudly rather than degrading.
498-
- **Performance-gate (soft).** Elsewhere the recipe deploys normally, but the
499-
NCCL and inference floors are fixed absolute values calibrated on full,
500-
high-bandwidth nodes. They are not normalized for GPU count or fabric class,
501-
so a smaller shape can fail a gate while being perfectly healthy. See
502-
[Validation › Node-shape assumption](./validation.md). Making these gates
503-
fabric/transport-class aware is tracked in
504-
[#1256](https://github.com/NVIDIA/aicr/issues/1256) and
505-
[#1254](https://github.com/NVIDIA/aicr/issues/1254).
506-
507-
Shapes not listed above are undocumented rather than known-broken: they have
508-
not been qualified, and the criteria model has no axis that distinguishes them.
509-
Whether AICR should gain one — finer-grained accelerator values, a machine-type
510-
axis, or a fabric class — is tracked in
511-
[#2377](https://github.com/NVIDIA/aicr/issues/2377).
512-
513478
**Examples:**
514479
```shell
515480
# Basic recipe for Ubuntu on EKS with H100
@@ -552,6 +517,45 @@ proceeds for compatibility. Use a current snapshot before deployment when you
552517
need conflict detection. See
553518
[Conflict detection requires snapshot evidence](slinky-slurm-accounting.md#conflict-detection-requires-snapshot-evidence).
554519

520+
#### Qualified machine types
521+
522+
Each recipe is qualified against a specific node shape. Running it on another
523+
machine type of the same GPU model is not blocked, but it is not qualified, and
524+
the consequence differs by family.
525+
526+
| Accelerator | Service | Qualified machine type | On other shapes of the same GPU |
527+
|---|---|---|---|
528+
| `h100` | `gke` | `a3-megagpu-8g` | **Components do not schedule.** The GPUDirect-TCPXO DaemonSets pin node affinity to `cloud.google.com/gke-accelerator: nvidia-h100-mega-80gb`, so on `a3-highgpu-*` / `a3-edgegpu-8g` nothing rolls out and the deployment health check fails. GPUDirect-TCPX is not shipped — tracked in [#2290](https://github.com/NVIDIA/aicr/issues/2290). |
529+
| `h100` | `eks` | `p5.48xlarge` (8× H100 SXM, 32× EFA) | Deploys, but performance floors are calibrated on the full node; smaller shapes such as `p5.4xlarge` can false-fail a healthy run. |
530+
| `h100` | `aks` | `Standard_ND96isr_H100_v5` (8× H100 SXM) | Deploys, but as above; `Standard_NC80adis_H100_v5` (2 GPUs) and `Standard_NC40ads_H100_v5` (1 GPU) can false-fail performance gates. |
531+
| `gb200` | `eks` | `p6e-gb200.36xlarge` (4 GPUs per K8s node) | Deploys; floors are sized for this shape and are themselves provisional pending production NVL72 data. |
532+
| `a100` | `gke` | `a2-highgpu-*` / `a2-ultragpu-*` | GPUDirect is not applicable; the `gke-nccl-tcpxo` component is intentionally omitted. |
533+
| `b200` | `gke` | A4 shapes | No separate NCCL plugin installer; multi-node NCCL comes from GPU Operator `gdrcopy` plus GKE A4 native multi-NIC. |
534+
535+
Two distinct failure modes are worth separating:
536+
537+
- **Component-level (hard).** Only GKE H100 pins artifacts to a machine type
538+
today. On a non-matching shape the DaemonSets have nowhere to land and
539+
deployment validation fails loudly rather than degrading.
540+
- **Performance-gate (soft).** Elsewhere the recipe deploys normally, but the
541+
NCCL and inference floors are fixed absolute values calibrated on full,
542+
high-bandwidth nodes. They are not normalized for GPU count or fabric class,
543+
so a smaller shape can fail a gate while being perfectly healthy. See
544+
[Validation › Node-shape assumption](./validation.md). Normalizing these
545+
floors per GPU or per fabric class was considered and declined
546+
([#1256](https://github.com/NVIDIA/aicr/issues/1256),
547+
[#1254](https://github.com/NVIDIA/aicr/issues/1254), both closed as not
548+
planned) — the floors are deliberately fixed absolute full-node values, so
549+
running a qualified shape is the supported way to pass them.
550+
551+
The table lists the accelerator/service pairs that have a qualified shape;
552+
a pair or a shape absent from it is **undocumented rather than known-broken**.
553+
It has not been qualified, and the criteria model has no axis that would
554+
distinguish it from one that has.
555+
Whether AICR should gain one — finer-grained accelerator values, a machine-type
556+
axis, or a fabric class — is tracked in
557+
[#2377](https://github.com/NVIDIA/aicr/issues/2377).
558+
555559
#### Snapshot Mode
556560

557561
Generate recipes from captured snapshots:

recipes/overlays/gb200-eks-ubuntu-training.yaml

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,8 +23,10 @@ spec:
2323
base: gb200-eks-training
2424

2525
# Machine type: qualified on p6e-gb200.36xlarge (4 GPUs per Kubernetes node).
26-
# The performance floors below are sized for that shape and are not normalized
27-
# for GPU count, so another GB200 shape can false-fail a healthy run.
26+
# The performance floors inherited from gb200-eks-training are sized for that
27+
# shape and are not normalized for GPU count, so another GB200 shape can
28+
# false-fail a healthy run. Those thresholds are themselves provisional --
29+
# see the parent overlay's note.
2830
criteria:
2931
service: eks
3032
accelerator: gb200

recipes/overlays/h100-aks-training.yaml

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,9 @@ spec:
2525
# criteria carry no machine-type axis, so they also match the smaller NCads
2626
# H100 shapes (Standard_NC80adis_H100_v5 = 2 GPUs, Standard_NC40ads_H100_v5 =
2727
# 1 GPU). Those deploy normally, but the performance floors below are fixed
28-
# absolute full-node values and can false-fail on them. See #1256/#1254.
28+
# absolute full-node values and can false-fail on them. Normalizing them was
29+
# declined (#1256, #1254 — both closed not-planned); the qualified shape is
30+
# the supported way to pass them.
2931
criteria:
3032
service: aks
3133
accelerator: h100

recipes/overlays/h100-eks-training.yaml

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,9 @@ spec:
2626
# (e.g. p5.4xlarge). Those deploy normally — unlike the GKE leaf, no component
2727
# here pins node affinity to a shape — but the performance floors below are
2828
# fixed absolute values calibrated on the full node, so a smaller shape can
29-
# false-fail a healthy run. Normalizing the gates is tracked in #1256/#1254.
29+
# false-fail a healthy run. Normalizing them per GPU/fabric class was declined
30+
# (#1256, #1254 — both closed not-planned); the floors are deliberately
31+
# absolute, so the qualified shape is the supported way to pass them.
3032
criteria:
3133
service: eks
3234
accelerator: h100

0 commit comments

Comments
 (0)