Skip to content

Commit 25b0c7a

Browse files
committed
docs(user): separate criteria resolution from deployment outcome
Review feedback: the section opened with 'running it on another machine type is not blocked', which blurred two different things. Criteria resolution never rejects another machine type of the same GPU model -- there is no axis to reject on -- so a recipe always resolves; what differs by family is what happens after. Say that instead. Likewise 'TCPX is not shipped' read as a statement about GPUDirect-TCPX itself. Google ships it; AICR ships no component that integrates it. Say that in both the table and the GKE overlay comment. The Slurm overlay comment put the inherited componentRef and the machine-type qualification in one 'inherited' clause. Only the componentRef is inherited by the base mechanism; the qualification is a documented assumption that holds because those same artifacts do. There is no machine-type axis to inherit -- which is the distinction this change exists to make. Signed-off-by: Yuan Chen <yuanchen97@gmail.com>
1 parent f9a5d90 commit 25b0c7a

3 files changed

Lines changed: 18 additions & 10 deletions

File tree

docs/user/cli-reference.md

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -519,13 +519,15 @@ need conflict detection. See
519519

520520
#### Qualified machine types
521521

522-
Each recipe is qualified against a specific node shape. Running it on another
523-
machine type of the same GPU model is not blocked, but it is not qualified, and
524-
the consequence differs by family.
522+
Each recipe is qualified against a specific node shape. Criteria resolution
523+
does not reject another machine type of the same GPU model — there is no axis
524+
to reject it on — so a recipe always resolves. What differs by family is what
525+
happens afterwards: on some, deployment validation fails; on others it succeeds
526+
and only the performance gates are affected.
525527

526528
| Accelerator | Service | Qualified machine type | On other shapes of the same GPU |
527529
|---|---|---|---|
528-
| `h100` | `gke` | `a3-megagpu-8g` | **Components do not schedule.** The GPUDirect-TCPXO DaemonSets pin node affinity to `cloud.google.com/gke-accelerator: nvidia-h100-mega-80gb`, so on `a3-highgpu-*` / `a3-edgegpu-8g` nothing rolls out and the deployment health check fails. GPUDirect-TCPX is not shipped — tracked in [#2290](https://github.com/NVIDIA/aicr/issues/2290). |
530+
| `h100` | `gke` | `a3-megagpu-8g` | **Components do not schedule.** The GPUDirect-TCPXO DaemonSets pin node affinity to `cloud.google.com/gke-accelerator: nvidia-h100-mega-80gb`, so on `a3-highgpu-*` / `a3-edgegpu-8g` nothing rolls out and the deployment health check fails. AICR ships no GPUDirect-TCPX component for the shapes that need one — tracked in [#2290](https://github.com/NVIDIA/aicr/issues/2290). |
529531
| `h100` | `eks` | `p5.48xlarge` (8× H100 SXM, 32× EFA) | Deploys, but performance floors are calibrated on the full node; smaller shapes such as `p5.4xlarge` can false-fail a healthy run. |
530532
| `h100` | `aks` | `Standard_ND96isr_H100_v5` (8× H100 SXM) | Deploys, but as above; `Standard_NC80adis_H100_v5` (2 GPUs) and `Standard_NC40ads_H100_v5` (1 GPU) can false-fail performance gates. |
531533
| `gb200` | `eks` | `p6e-gb200.36xlarge` (4 GPUs per K8s node) | Deploys; floors are sized for this shape and are themselves provisional pending production NVL72 data. |

recipes/overlays/h100-gke-cos-training-slurm.yaml

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -25,10 +25,15 @@ spec:
2525
# `aicr bundle ... --set slinkyslurm:...` or a valuesFile.
2626
base: h100-gke-cos-training
2727

28-
# Machine type: a3-megagpu-8g only, inherited from the parent leaf along with
29-
# its gke-nccl-tcpxo componentRef. These criteria add no machine-type axis, so
30-
# the parent's TCPXO-vs-TCPX selection gap applies here unchanged. See the
31-
# parent overlay's note and #2290.
28+
# Machine type: qualified on a3-megagpu-8g only, same as the parent.
29+
#
30+
# Two different things carry over from `base`, and only one of them is
31+
# inheritance: the gke-nccl-tcpxo componentRef is inherited by the base
32+
# mechanism, while the machine-type qualification is a documented assumption
33+
# that holds here because those same artifacts do. There is no machine-type
34+
# axis in the criteria contract to inherit -- these criteria add none, so the
35+
# parent's TCPXO-vs-TCPX selection gap applies unchanged. See the parent
36+
# overlay's note and #2290.
3237
criteria:
3338
service: gke
3439
accelerator: h100

recipes/overlays/h100-gke-cos-training.yaml

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,8 +32,9 @@ spec:
3232
# `cloud.google.com/gke-accelerator: nvidia-h100-mega-80gb`.
3333
#
3434
# On a non-mega shape nothing schedules and the deployment health check fails
35-
# loudly (`desiredNumberScheduled > 0`). TCPX is not shipped. Selection fix
36-
# and a TCPX component are tracked in #2290; whether criteria should gain a
35+
# loudly (`desiredNumberScheduled > 0`). AICR ships no GPUDirect-TCPX
36+
# component, so those shapes have no working alternative. Selection fix and a
37+
# TCPX component are tracked in #2290; whether criteria should gain a
3738
# machine-type or fabric axis is tracked in #2377.
3839
criteria:
3940
service: gke

0 commit comments

Comments
 (0)