Skip to content

Commit 777e44f

Browse files
committed
fix(recipes): make GB300 NCCL checks run and the nodewright assert name-agnostic
Two declared gates did not do what the recipes implied. The nodewright-customizations health check hardcoded the Skyhook CR name "tuning", but no-op.yaml renders a CR named "no-op". The render-aware suppression only skips the assert when a component renders zero CRs, so any component wiring a differently-named manifest asserted a CR that never exists. The assert is now name-agnostic: the executor's list-and-match path fails closed on an empty collection rather than passing vacuously, and verifyNodewrightReady still enforces per-CR completeness by name. The GB300 NCCL net and nvls checks skipped rather than ran, because the validator's applicability matrix registers only gb200 for EKS on both variants and an unsupported combination returns Skip — leaving the declared thresholds inert and the performance phase vacuously green. Declare nccl-benchmark-profile: gb200/eks, the documented borrow for an existing service with a new accelerator. Also corrects two overlay comments: the fabric claim now states that the overlays assume EFA to match the reference cluster rather than asserting GB300 was qualified on it, and the Dynamo gate is marked as live from the first run, since inference-perf has no accelerator gating. Signed-off-by: Yuan Chen <yuanchen97@gmail.com>
1 parent 4ee8fc0 commit 777e44f

6 files changed

Lines changed: 98 additions & 68 deletions

File tree

pkg/bundler/testdata/stock_render_golden.yaml

Lines changed: 27 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -3,51 +3,51 @@
33
#
44
# One entry per leaf overlay: a digest over its fully rendered helm-deployer
55
# bundle tree (sorted relative paths paired with per-file content hashes).
6-
a100-aks-ubuntu-training-kubeflow: c372718509da1b12167fb3b9dd73d5ab50cbd3e196bc1ccc907d760719dc070e
6+
a100-aks-ubuntu-training-kubeflow: d9e713041b0ecafb1f9bddcb9edce87d9a43bcc7e732a1cedd6226e79a4d1652
77
a100-any: 443a3195639f0354c05d41c34e84310184e77fe19a1db4e4157248b423eda8be
8-
a100-eks-ubuntu-training-kubeflow: 2f95f78419ec6b24837a89b3168922b0c1cd0291234616b51977227ed0bc82ae
9-
a100-gke-cos-training-kubeflow: 98ae8ea5b594164b59aaec2b3abbd2d626ae94b1630f47562e49bfccb2d6958a
8+
a100-eks-ubuntu-training-kubeflow: efc43236df64d337bba806d9ecf1fee04749242d1c538e645f0b35125c66a111
9+
a100-gke-cos-training-kubeflow: f3744d9387bf227c38373674a0fe9a0812b561ff943ca55b9f94ab57393d2518
1010
a100-oke-ubuntu-training-kubeflow: 1539a64bf1eb692e759101fd13115083c657779060f40719d0d2f3b70d801d31
1111
b200-any: e64aac932ee176d835eee0a326ec4b77c30b2e97afd0dc5f22e8c89640c37da2
12-
b200-gke-cos-inference-dynamo: a19cac53536564f9cec76aa65723eaf1eb2c21095565c7d23809be3ca1b97ac2
13-
b200-gke-cos-training-kubeflow: 9fd82f03b9700181f1efd16f6c572d5b3a5f44874c9bcf255f43dc55f63c1e65
14-
bcm-inference: 3adcec67f5e7745ce21c2fcec49143b76b29f66abe207f47d42a7575d772ba27
12+
b200-gke-cos-inference-dynamo: 3bac9410068fa1f73e2dc61799d5cbedbfc8b721e564f7334b8b054b6e171b5c
13+
b200-gke-cos-training-kubeflow: 094c79ca01541a09c183cbc1d35c8f25408f015a8519b67d1c2370d4e4bb8fa4
14+
bcm-inference: eba0c9554ee16f0b79f169c2f69017233c69d7b418ea0b23d54cda98d3d0eec6
1515
gb200-any: 5ceba4eb7316ca448369bcb4dd20240247cd15c3d3215b354c738dec8f359162
16-
gb200-eks-ubuntu-inference-dynamo: 4dd15671a1126562b30389b73d93f6357aa30df13465d4b44a76cf834dbef52b
17-
gb200-eks-ubuntu-training-kubeflow: dcad2ea2c6b3742670d4f50925715d7a5cbd3cf0526d94789659b099fe75e444
18-
gb200-eks-ubuntu-training-slurm: 44538377243945fefb4b037fee2c9b316a74894bfeacdd3bc0bde13188c62930
16+
gb200-eks-ubuntu-inference-dynamo: b3e7aa31e71b6a40ac9be1e5c4320e2943ad64f2f19a000d4a1b60c45ac3aa24
17+
gb200-eks-ubuntu-training-kubeflow: 6d22eac954455791a7f5dfce7b1f301f6c57624838a717031c388539990b9c52
18+
gb200-eks-ubuntu-training-slurm: 22a4cccff15a46ba774ef981ea5f6035dab9283c57028b77219c2f7cfa95f12b
1919
gb200-oke-ubuntu-inference-dynamo: 0148baa4731d28f0251a8dc121117085ce445b638f3b27f257c6d7bc9ee35ab9
2020
gb200-oke-ubuntu-training-kubeflow: e6bcb925712d588a1596bb9825169a944609a7f6b0ab8981810877652d678f5d
2121
gb300-any: 2cd40fe86553cc08b3bfdf3f8f5c11e3c50e84a4d0b22494f274fa62e9ab01e9
22-
gb300-eks-ubuntu-inference-dynamo: 51fa43bea7ae52ec6d75a618e12672800d293105a7303adc8c87a8fb0237c799
23-
gb300-eks-ubuntu-training-kubeflow: 8f0bfaf20e300af5f3417fd5e1f0f2fa65e27841cba81d254360f09c47a9821b
24-
h100-aks-ubuntu-inference-dynamo: 5464797e850b026cf6b1750c5dfca195809f7d1d9a4e8550c6048139743e281e
25-
h100-aks-ubuntu-training-kubeflow: 56166fa188acd730492e67d8f0bf7b82bd47ec7ec3a9bdbf69ee66ae307c5241
26-
h100-aks-ubuntu-training-slurm: 41ef1f36d370c064e8b452ec2dffe6b9dae5193de3d073c766e97c24bc5b130a
22+
gb300-eks-ubuntu-inference-dynamo: 8f4a35471d993926d3877d22ee3a379755d5e8a7c37580a37bdf808fd6287187
23+
gb300-eks-ubuntu-training-kubeflow: 51d4cfa9314323d79c152f88157c2d589123417403d7c805c5908f1e78e9fb38
24+
h100-aks-ubuntu-inference-dynamo: 60106eca2412d6fad922a4524a09d2d68b7526eaa7f3ff49fd43d0cd7a26c4ce
25+
h100-aks-ubuntu-training-kubeflow: 231fb345f168a21fc81a5b2ee6fb3a072dd88ac5a64f4b83368a47c0f016bc15
26+
h100-aks-ubuntu-training-slurm: 4e358a78b920a7abe8f29562ee74c3ca93d9ed9cd89150ca4bf7ceb73ac680d7
2727
h100-any: b96b596863029f47ee8759d159e12db7c9863589379c62ab5d859287bb353667
28-
h100-bcm-ubuntu-training: cc6ec7a4dec6c0eb28fde65a92c64f4e1a225ef1b2f7d421e2829676e92157b8
29-
h100-eks-ubuntu-inference-dynamo: 865c0cde4d8a6bc2b37a097a21b016039db923e0c6a4f791b58bf94647af6662
30-
h100-eks-ubuntu-inference-nim: ef6db54bfd14fff7db0553524600071cf22315c0a049d2d86ef0d07b51ce20db
31-
h100-eks-ubuntu-training-kubeflow: bb1c03beb00233862006e4d7b741c756f4bcfcde89831c0a77a7dc4f25ecfb20
32-
h100-eks-ubuntu-training-slurm: 3188554d15b20ce5e5a042e8231bbb865a3c9bd236eade52902804d2bcbbd535
33-
h100-gke-cos-inference-dynamo: d3d3f1aab57c4204d00c2aa06b4e367d11372bf9d94762207c8d906d142c7a2c
34-
h100-gke-cos-training-kubeflow: ae4f3d89a6662adc47be27108cd5d1d211be0d9f02260bc8c5510c6059a8bff5
35-
h100-gke-cos-training-slurm: c5b1bc34d751cf7ef641df5bcea3643f5fb37945fef0dc3483ec7a6065681ab7
28+
h100-bcm-ubuntu-training: aa4c2bd1b395fd29b57dd9226ac0b04eaf3edcd39a853a113a379dfea2f31cb9
29+
h100-eks-ubuntu-inference-dynamo: 71d5ea9ccb46dd4cb1cf1685faa6d3f8b8404dacf38935f6ce5ca319c6c8b99b
30+
h100-eks-ubuntu-inference-nim: 1d79a4e312bc47486493bf835ac2264d8786c849b6e6c88c355d7c9084381df6
31+
h100-eks-ubuntu-training-kubeflow: a121f87063837539f4c26710d37e587d8c3c3be95a7446b6e80088273c1e7625
32+
h100-eks-ubuntu-training-slurm: d37883a04011b600fc1a24350c20044f08c317138f40bfe674cb0a7ba235d726
33+
h100-gke-cos-inference-dynamo: fbae6625cfea5c37079cae55bfbd05d177f59c9c8c766b098f3291065b0c660c
34+
h100-gke-cos-training-kubeflow: 4486d628426f471c3b1ec087947e798ae6998cbe339bf8188570fddb9450a135
35+
h100-gke-cos-training-slurm: fa6d26058170f02e3ae099c57ba6cb2afa8d890298d6efc52cc21d929d34c325
3636
h100-kind-inference-dynamo: 3b5c3aa86fe396d355fc77a6a7bd2fb6b87d7b7d9a6abf272b2ed41023920673
3737
h100-kind-training-kubeflow: c2a29960328fc03e6ce67e5e8f25d6fe750886909ca31ee63750b5150a1e6a76
3838
h100-kind-training-slurm: 3cfbdfb63490de808042b535b8ad92ba45e1dd63d02401ecf57163765f36a4fe
3939
h200-any: c84e22d03d87a174b5215d883e9131e8a6c5f1befedc179a2cd2ca2bded39144
40-
h200-eks-inference: 17d5b7b4ee2d82443a89e9c403625aaa38a1f70ec6f4fee3a34d02c3210fc648
41-
h200-eks-training: c7b713eb05c1fb221e7e85c4e84f6845e34197a6375077942800e033199faeab
40+
h200-eks-inference: bbae5686447c4b4c0dec4b8a5e263710fe67f6fce61b2635bab3a9a335502430
41+
h200-eks-training: 57b10be6b4878f1b9aabfa2df09721cb7fa362cba411335ad3052f50ab3822c2
4242
l40s-any: 384bb817ae61f86388cb87e8ff28287cc518988c3e0f65468d64f4e488be58ca
4343
l40s-oke-inference: 4fd3eaa4c3b3b9f9aae2597abe450c8c6f69ed44c8a25c858d38d264e175d377
4444
l40s-oke-training: b18da6c239e730f5e11e8da9a109e9e95fc5ddd5f1e947666b8119db5b517d3f
4545
monitoring-hpa: b35a6c42d73788facd17b104b554bf6d2ae32bf73b0193424a72e566f1b6ceaf
4646
ocp-inference-nim: bb00cdb191823b32da334bea70826c8a92c11b62d0e096918d43a8e1b043c361
4747
ocp-training: d7a213263630f2c25982d6f4a144df7d9d5784428d296ccda1b4dab5a42b98bb
4848
rtx-pro-6000-any: 5500f71ae8812ed1541588222a1dc61f0d4e1c8f8cd7fd9c6a21e29f4d647507
49-
rtx-pro-6000-eks-ubuntu-inference-dynamo: 112dea463beb0f2328a4c61f908c934fe0944b066b6eb650d69e160f3c187cee
50-
rtx-pro-6000-eks-ubuntu-inference-nim: 9f675d1c39282b7aa4eb884a1f2b2512a1186dfa529c8b5200967726f4be65dd
51-
rtx-pro-6000-eks-ubuntu-training-kubeflow: 87207e073c6c9234f655d997d4f605b76ddc7d04905686b94a224774af2f6843
49+
rtx-pro-6000-eks-ubuntu-inference-dynamo: 16a4de23f7d145e56f8da9ee3e067c47cd354e37b26cfd1f3c52ba4ff7728a93
50+
rtx-pro-6000-eks-ubuntu-inference-nim: eb765d6af75d71c346a3ce1f8bd8c6c8ab9f3282327988a2eb8a100386764b62
51+
rtx-pro-6000-eks-ubuntu-training-kubeflow: bc961a695a0e639e6fe79727f0284c9635e26c208824c63bac8526648446b53c
5252
rtx-pro-6000-lke-ubuntu-inference: d08362d22a96a8992c4669d6bf334fc06f6cbf050a883f456fe10674d9d925eb
5353
rtx-pro-6000-lke-ubuntu-training: 5b0b3d9cc55eea2f60bcc80a54a250e5a03a204dc506863d137d75b8a70a4072

pkg/recipe/testdata/catalog_parity_golden.yaml

Lines changed: 27 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -3,51 +3,51 @@
33
#
44
# One entry per leaf overlay: sha256 of its deterministically-marshalled
55
# resolved recipe. A moved digest means that recipe's resolved bytes changed.
6-
a100-aks-ubuntu-training-kubeflow: ffbf217dc9fd1b596e621db0c59d8ea5b5e6866caf263dbc232e250c9f1d3368
6+
a100-aks-ubuntu-training-kubeflow: 3dfa350aa517fa401b8474629659be7146aacbc25805e70de241e287f4b2e659
77
a100-any: 62d1581b21fecd3465b69659bac3dc9e5438eb2612c98c1acdd7b19cc9237084
8-
a100-eks-ubuntu-training-kubeflow: 8221fea2ed20a162728161059975571733630817a9c517988cc747ce2652cf45
9-
a100-gke-cos-training-kubeflow: 56853465e3f4922b64cf4306b9d991cf5dd1d4d9070afa52df03dc27dfef8555
8+
a100-eks-ubuntu-training-kubeflow: 974a3c313452457605a42ae6e4669ca15ba747e91a4b8d0da85944323ff45291
9+
a100-gke-cos-training-kubeflow: 17df2d55ab7b62e3d1357dba60a20289881c2f647797dd5efb8b34b999e790fa
1010
a100-oke-ubuntu-training-kubeflow: 310a6ee2abdd7f8508882429d657c01bd23458ba635236c410934825e4f617dc
1111
b200-any: 63dbc4fe27b84321e8b1991cd98c13f85395a3919d13e0c6891599eaa1f1e729
12-
b200-gke-cos-inference-dynamo: 51431391a0d6d470709174ab9be8103fa64baa21f9bd1fc6c0373959b6bfe21c
13-
b200-gke-cos-training-kubeflow: bffdd1b939344ea9a8f40abdfcc00edea3908304e0e9f668d5b07d113fc1224b
14-
bcm-inference: 1871d4b08c3d7221d3eb72a70c765f696fff0c7e96b6d4111440882901d2287b
12+
b200-gke-cos-inference-dynamo: 6efe04d7048b6d9848c07bd90a315ed0a0641905d9c6794b22271c49323d6fdb
13+
b200-gke-cos-training-kubeflow: e151a51e1ad7ec31f77a7ec7c24961b8fdce94d569127cf5114864ba40ba1462
14+
bcm-inference: 43ab64c51d714450cbdcdb6b16022b9a89ddab55cab0c56e6de7a5d8d0f1e4a1
1515
gb200-any: 9ee6ab187b4c1b93cd859d86ada35e922d0873ffc6fe892cb084ad98baf09727
16-
gb200-eks-ubuntu-inference-dynamo: 77b97f2841e6c58754b22d45b7cbfc20ffc0582dc804ccdb7d72776429c53c20
17-
gb200-eks-ubuntu-training-kubeflow: 4329cd8773d0bd00428bbf6bdc53d2695742d7bd7bc16f07fd3019ed7a5854e4
18-
gb200-eks-ubuntu-training-slurm: cf2147f07cae42f8c4b171ab79022dac223e7c102e46469b202ee6ba2b27efbe
16+
gb200-eks-ubuntu-inference-dynamo: c0de54a7ade04e826a1f7cb1ef9c88c28b60c4aad6357eb0ee8fe9063112a3ec
17+
gb200-eks-ubuntu-training-kubeflow: 19220d5a2e9ec32954bd1b32bb4e874cc9f57ca1119d7e04990330ea2e3a59c7
18+
gb200-eks-ubuntu-training-slurm: 86a9358483ebb372d6a62d4702a585c6302f56518c509f2f42e414915fac8d28
1919
gb200-oke-ubuntu-inference-dynamo: 138c28fdd64d4a08a2c1349b5b998f2ee68cdf8de452724510d00f7296b1a5cc
2020
gb200-oke-ubuntu-training-kubeflow: 3959f47474edfac9b53a65939117d4364551fde0c9324246f195f144fe3beb12
2121
gb300-any: 7f68607dffbbfc912f19b00ffa833caa81ca675d25bc22cf612dbd0b76f39475
22-
gb300-eks-ubuntu-inference-dynamo: f83c02d93b81c1c774527755ad79d0c520fc9e569e0659389415eb5a2241b0ee
23-
gb300-eks-ubuntu-training-kubeflow: e6a6d171d7511a4fc9ca950cf48b20ac68f8230af36d1e6294565402ead55d21
24-
h100-aks-ubuntu-inference-dynamo: 61a6cdb6c3f1a5460332cef094fd76e14da87dc1b39bc7229fbbb8ccae0e5aac
25-
h100-aks-ubuntu-training-kubeflow: b4bf459485d8f5882393220de96e8ea4aef65fadae6b86f395b26fa6b3767c1b
26-
h100-aks-ubuntu-training-slurm: cd9488c6ae6eed9c43a08eeaaaa4ff5e9eb378f19256ac0c02d478706efb6048
22+
gb300-eks-ubuntu-inference-dynamo: 44bdad10e2340b164d1b2f906cdc61791f4eb5d2ab7ac83257ec725284f401e4
23+
gb300-eks-ubuntu-training-kubeflow: 23ff8f6bc4ac65d877ae79215ed7e22b817c62e2fbc974793e8fe67f7a1de89c
24+
h100-aks-ubuntu-inference-dynamo: 07117b4d11ec6042258f42fb08781b218880f58c301ec71f641beedef8d62806
25+
h100-aks-ubuntu-training-kubeflow: 2b23859469f426f76d39136a561eacee0e12cfe43a56c2abe72b0501bf9236ef
26+
h100-aks-ubuntu-training-slurm: 9776f8101cc5d7755681d6424f8b3f74f159af8f5c1436847bfa2c72ccf0c3ea
2727
h100-any: f5a55f03948075adde9b1b24fd650d98fa4d35b85f394d071ad3365ba05588a5
28-
h100-bcm-ubuntu-training: 22d3283e415997b5cfd747a842db026bff1a297afcce6b23247c12a384097d59
29-
h100-eks-ubuntu-inference-dynamo: dd981d3b52fcf02e2c24b1a940b236b02fd96a57e6e020e091090d7d3643c3af
30-
h100-eks-ubuntu-inference-nim: c75f6502b92431467692a77e5eb372f0633385c52e0c46fd8d8cdc49a1c1d72c
31-
h100-eks-ubuntu-training-kubeflow: 67b3879633e1d47fc932a4292719e692848aaba1ac6bb07a6ab52c3e5b9548e1
32-
h100-eks-ubuntu-training-slurm: 9b69a63af76b3595af773368147b73dc0f6ab3708eb21aa2a4dac93b0bce3e1d
33-
h100-gke-cos-inference-dynamo: a236779e81f74303bbc13c36fafb71158d05b6a9398d27fd5de9c5c013a183a9
34-
h100-gke-cos-training-kubeflow: 10db012bf1053759ef2610e7cc46eeee242a61f92c745e5d8175829b7b1cda6d
35-
h100-gke-cos-training-slurm: b5fd06a9b6764c596e2e33e7e79c193f01004a17af0ecf4901c64399ff2547a2
28+
h100-bcm-ubuntu-training: a4c56385b4f0908ebfb022899d5075555e3179d5edb55f08cc062548c196c318
29+
h100-eks-ubuntu-inference-dynamo: b79ae736a7d9bb20ca2ddb8117aae7e6cdd955f89b8face685b614701a7d64c9
30+
h100-eks-ubuntu-inference-nim: 9b52a271c4afd205b5b6c0346513e5980f052bb3d6b8a5c8c04c1ef9d2d101e1
31+
h100-eks-ubuntu-training-kubeflow: 501e3007f9d1b2391b6f196c5fdd769e9a2f180f33c25a9b4592c01dc4dc282a
32+
h100-eks-ubuntu-training-slurm: 4f5a721af39c8ea372fcdb85e5e740065b99ed4f1287d355d39174d09d023657
33+
h100-gke-cos-inference-dynamo: 3ba2ec40aecbd3910785df7343896773e07c563901373fdff34fcfb4bca0aba6
34+
h100-gke-cos-training-kubeflow: 52a6517d46a6be029a12115d8d0f7ccc0a4516270cc03ce9b172f5b3c03b1cd5
35+
h100-gke-cos-training-slurm: 6899367250adb995f0383eaee6cf0762673495b2b7442ff98e7296731edfb917
3636
h100-kind-inference-dynamo: 0e2d552b62b55b91f960c9721ed128b1d07f68be8517799473fbaa3baa1cc6e5
3737
h100-kind-training-kubeflow: 4bb03659faaa6a9a66d58dfc95572bdb54b75010c219c4bbefbd0eafeb8ed867
3838
h100-kind-training-slurm: 0245c786638ade1023ba4dc49e0f2efd56d2f789bdf447e946bbc3a3e652e734
3939
h200-any: acf1986d76037dddd025125eaa34eb7ecf3d31fe62cb80e7dc89dd042ebb42f5
40-
h200-eks-inference: 02fc22eeddf62c684fdc58d35b2728ed5b0aaf147d63424a96b2814764dd21b5
41-
h200-eks-training: 57b28d6401d77a2443462876fd39dd1dd85b57c73c797ee4549cc8cbed3b2ada
40+
h200-eks-inference: badc48bec4aa553f3b9aad5e89ee0d8fcbae3ee5d393b885f1aee0377024d764
41+
h200-eks-training: 3b7d225d76c247691cfbb0fe91bd98f5e4bb2863f4403df89ff2daa0a49f7b8f
4242
l40s-any: 89210ba5815f93ce3d7b1a2a78f7ba75d083556e940603a1bc812eac9aeccd0a
4343
l40s-oke-inference: 6c75ef93e86fb21dbdf73ac97089b894ca8da832164dc807e94592493ab7f7cc
4444
l40s-oke-training: 2a35011bd59cf3c727cea158d504c51bd59cba069620d3d6a4988d68972dee74
4545
monitoring-hpa: 15e93304e7e68997e7b15a0be4af360a29f10c869502e6dccb6a6cefb6519fd5
4646
ocp-inference-nim: 49245ecfc91f67fcc2ed0b305b00046a4164628cafc75ab98542e6dafa626821
4747
ocp-training: aaa8cd7f45ef42235b075936461d715e2e017ea2802df9e8017de38304a17915
4848
rtx-pro-6000-any: 9639d67639256a44be8566f3f8eda49ceb4025b6b140a3bc2a4b6d4768d07ffa
49-
rtx-pro-6000-eks-ubuntu-inference-dynamo: 941af41effe3efff910e1fe88eb0915621426a0080efb54682af30b597455580
50-
rtx-pro-6000-eks-ubuntu-inference-nim: 10cf9bcd387410a5441c9956c941238d50efcccf2f085a24e195cdd8b26753c1
51-
rtx-pro-6000-eks-ubuntu-training-kubeflow: ce9b10a01a63dc079033206a6a26e601fb7f84c8f809bc6322a0ee9dca57d9fe
49+
rtx-pro-6000-eks-ubuntu-inference-dynamo: 14c5d90b608eda5dd48cfb22a4ceabc50cc8ca50acf08df5adea5e2059f6fe60
50+
rtx-pro-6000-eks-ubuntu-inference-nim: 59cafd3a23b4cd52cdfff318fd39dcc5bd8b587ccd49146d5868bde69127d3b1
51+
rtx-pro-6000-eks-ubuntu-training-kubeflow: ff3a0be78e21b186458ebe1183742ba80077a7608b4d1d7ed1c27f9fdec486cf
5252
rtx-pro-6000-lke-ubuntu-inference: 28cdab0531f8ed6018391aba7cab1fbfad8ba551392de1b9138f3b88042f5340
5353
rtx-pro-6000-lke-ubuntu-training: 312c4cf49d0e8cec0377cd9858338d7106226a2252d81828a75c1d426a194a2f

recipes/checks/nodewright-customizations/health-check.yaml

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -50,13 +50,25 @@ spec:
5050
steps:
5151
- name: validate-skyhook-cr-complete
5252
try:
53-
# Assert the Skyhook CR (e.g. tuning) in the skyhook namespace has
54-
# status.status == "complete" per the Skyhook CRD schema.
53+
# Assert a Skyhook CR in the skyhook namespace has status.status ==
54+
# "complete" per the Skyhook CRD schema.
55+
#
56+
# Deliberately name-agnostic. The manifests do not agree on a name —
57+
# tuning.yaml / tuning-gke.yaml / tuning-generic.yaml render "tuning"
58+
# while no-op.yaml renders "no-op" — so a hardcoded name asserts a CR
59+
# that never exists for any component wiring a differently-named
60+
# manifest. A nameless assert uses the executor's list-and-match path
61+
# (pkg/chainsaw/inprocess.go): it fails closed when the collection is
62+
# empty (ErrCodeNotFound, not a vacuous pass) and passes when a live,
63+
# non-terminating Skyhook reports complete.
64+
#
65+
# Per-CR completeness is not weakened by this: the render-aware Go
66+
# readiness check (verifyNodewrightReady) derives the expected CR names
67+
# from the component's rendered manifests and requires every one of
68+
# them to reach complete, reporting each failure by name.
5569
- assert:
5670
resource:
5771
apiVersion: skyhook.nvidia.com/v1alpha1
5872
kind: Skyhook
59-
metadata:
60-
name: tuning
6173
status:
6274
status: complete

recipes/overlays/gb300-eks-inference.yaml

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,10 @@ spec:
2727
accelerator: gb300
2828
intent: inference
2929

30-
# Qualified on p6e-gb300r.36xlarge (GB300 NVL72, 4 GPUs per node, Grace
31-
# ARM64 host, EFA scale-out fabric).
30+
# Targets p6e-gb300r.36xlarge (GB300 NVL72, 4 GPUs per node, Grace ARM64
31+
# host). This overlay assumes the EFA scale-out fabric, matching the
32+
# reference EKS cluster whose GPU nodes carry an efa-only ENI. RoCE-fabric
33+
# GB300 is out of scope here; fabric-aware selection is tracked in #1410.
3234
#
3335
# Specific constraints for GB300 on EKS inference workloads.
3436
# Constraint names use fully qualified measurement paths: {type}.{subtype}.{key}

recipes/overlays/gb300-eks-training.yaml

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,10 @@ spec:
2727
accelerator: gb300
2828
intent: training
2929

30-
# Qualified on p6e-gb300r.36xlarge (GB300 NVL72, 4 GPUs per node, Grace
31-
# ARM64 host, EFA scale-out fabric).
30+
# Targets p6e-gb300r.36xlarge (GB300 NVL72, 4 GPUs per node, Grace ARM64
31+
# host). This overlay assumes the EFA scale-out fabric, matching the
32+
# reference EKS cluster whose GPU nodes carry an efa-only ENI. RoCE-fabric
33+
# GB300 is out of scope here; fabric-aware selection is tracked in #1410.
3234
#
3335
# Specific constraints for GB300 on EKS training workloads.
3436
# Constraint names use fully qualified measurement paths: {type}.{subtype}.{key}
@@ -125,6 +127,18 @@ spec:
125127
- nccl-all-reduce-bw-net
126128
- nccl-all-reduce-bw-nvls
127129
constraints:
130+
# Without this, both checks SKIP rather than run: the validator's
131+
# applicability matrix registers only gb200 for EKS on the net and nvls
132+
# variants, and an unsupported combination returns Skip — the
133+
# thresholds below would be inert and the phase would report a vacuous
134+
# pass. gb200/eks is the documented "existing service, new accelerator"
135+
# borrow (see docs/user/validation.md), and it is the right template
136+
# here because GB300 on EKS runs the same EFA scale-out fabric and the
137+
# same NVL72 MNNVL/IMEX topology the gb200/eks profile was built for.
138+
# Replace with a native gb300 matrix entry once GB300 runtime templates
139+
# exist.
140+
- name: nccl-benchmark-profile
141+
value: gb200/eks
128142
- name: nccl-all-reduce-bw-net
129143
value: ">= 40"
130144
- name: nccl-all-reduce-bw-nvls

recipes/overlays/gb300-eks-ubuntu-inference-dynamo.yaml

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -56,12 +56,14 @@ spec:
5656
performance:
5757
checks:
5858
- inference-perf
59-
# Conservative smoke gate mirrored from the GB200 EKS Dynamo overlay
60-
# until GB300-specific inference baselines are available. GB300 NVL72
61-
# exceeds GB200 per-GPU, so the GB200-derived floor should not false-fail
62-
# a healthy run — it must be re-baselined on real GB300 hardware before
63-
# it is treated as a regression gate. Model + concurrency are pinned so
64-
# the gate stays valid independent of the compiled validator defaults.
59+
# Smoke gate mirrored from the GB200 EKS Dynamo overlay until
60+
# GB300-specific inference baselines are available. Unlike the NCCL
61+
# checks, inference-perf has no accelerator gating, so this gate is live
62+
# on GB300 from the first run — it is a real threshold, not a
63+
# placeholder, and must be re-baselined on real GB300 hardware. GB300
64+
# NVL72 exceeds GB200 per-GPU, so the GB200-derived floor should not
65+
# false-fail a healthy run. Model + concurrency are pinned so the gate
66+
# stays valid independent of the compiled validator defaults.
6567
#
6668
# NOTE: this throughput floor is a fixed absolute full-node value (here a
6769
# 4-GPU GB300 node) and is not normalized for GPU count, so a smaller SKU

0 commit comments

Comments
 (0)