fix(validator): reject a terminating Nodewright CR in the readiness check

The per-name readiness check read status.status without inspecting
metadata.deletionTimestamp. Nodewright uses a deletion finalizer, so an
expected Skyhook can sit Terminating for a while while still reporting
complete. Combined with a stale or unrelated live complete Skyhook —
which the name-agnostic component assert accepts on its own — both gates
could pass while the CR the recipe actually declared was on its way out.

Reject a CR with a deletionTimestamp before reading its status, matching
the ghost-skipping the Chainsaw executor already applies to positive
assertions. Adds a regression test for the combined state and a control
proving a live complete CR still passes.

Also raises the GB300 nvls floor from the inherited GB200 value of 500 to
700, measured on 2x p6e-gb300r.36xlarge: nvls 841.17 GB/s and net
44.04 GB/s, both ~88% of their line rates. The net floor stays at 40 —
the reference cluster advertises a single EFA device, so its figure is
capped by NIC provisioning rather than by GB300 and is not a
representative scale-out baseline.

Corrects the health-check comment: Skyhook is cluster-scoped.

Signed-off-by: Yuan Chen <yuanchen97@gmail.com>

Author: yuanchen8911

pkg/bundler/testdata/stock_render_golden.yaml

@@ -3,51 +3,51 @@
 #
 # One entry per leaf overlay: a digest over its fully rendered helm-deployer
 # bundle tree (sorted relative paths paired with per-file content hashes).
-a100-aks-ubuntu-training-kubeflow: d9e713041b0ecafb1f9bddcb9edce87d9a43bcc7e732a1cedd6226e79a4d1652
+a100-aks-ubuntu-training-kubeflow: 8328fc68f6aa12c805a788f4e364919843828d9fd2f6759362de5e0a4fa406c6
 a100-any: 443a3195639f0354c05d41c34e84310184e77fe19a1db4e4157248b423eda8be
-a100-eks-ubuntu-training-kubeflow: efc43236df64d337bba806d9ecf1fee04749242d1c538e645f0b35125c66a111
-a100-gke-cos-training-kubeflow: f3744d9387bf227c38373674a0fe9a0812b561ff943ca55b9f94ab57393d2518
+a100-eks-ubuntu-training-kubeflow: 32bde42e60fa032d379095ff7221a40e27fa1e23cdb041d1a93ee7700830611e
+a100-gke-cos-training-kubeflow: 97db4756bce33b48451ccdacc0fb46aabcdefcc889749e1477fd62a4f77d65df
 a100-oke-ubuntu-training-kubeflow: 1539a64bf1eb692e759101fd13115083c657779060f40719d0d2f3b70d801d31
 b200-any: e64aac932ee176d835eee0a326ec4b77c30b2e97afd0dc5f22e8c89640c37da2
-b200-gke-cos-inference-dynamo: 3bac9410068fa1f73e2dc61799d5cbedbfc8b721e564f7334b8b054b6e171b5c
-b200-gke-cos-training-kubeflow: 094c79ca01541a09c183cbc1d35c8f25408f015a8519b67d1c2370d4e4bb8fa4
-bcm-inference: eba0c9554ee16f0b79f169c2f69017233c69d7b418ea0b23d54cda98d3d0eec6
+b200-gke-cos-inference-dynamo: 5492567683a679eb416bbad2db1ddf78ce28bde6b852b69a6331cd3f06d20471
+b200-gke-cos-training-kubeflow: ea479d8c9001d93569bc743bbfa7162a73edcd706c9b0f377a28188e9a98cb8c
+bcm-inference: a62afcf0ed10cdfd910e55c73f76f7376d434f73a995d011b2d4f206e88000f9
 gb200-any: 5ceba4eb7316ca448369bcb4dd20240247cd15c3d3215b354c738dec8f359162
-gb200-eks-ubuntu-inference-dynamo: b3e7aa31e71b6a40ac9be1e5c4320e2943ad64f2f19a000d4a1b60c45ac3aa24
-gb200-eks-ubuntu-training-kubeflow: 6d22eac954455791a7f5dfce7b1f301f6c57624838a717031c388539990b9c52
-gb200-eks-ubuntu-training-slurm: 22a4cccff15a46ba774ef981ea5f6035dab9283c57028b77219c2f7cfa95f12b
+gb200-eks-ubuntu-inference-dynamo: 2322e924b7055d5c24e0ca40ef2a8a16cd610344b11290e166456f9a4c287f07
+gb200-eks-ubuntu-training-kubeflow: 1f5559aac0bb96f5ce65c4e0a30fc11d2d22d6c29a715b312ad381f2d1f7f8b6
+gb200-eks-ubuntu-training-slurm: 5c3c5e84f3268a54ca3c4c0e230e74801a87abbbf382a80593b23d35649d2864
 gb200-oke-ubuntu-inference-dynamo: 0148baa4731d28f0251a8dc121117085ce445b638f3b27f257c6d7bc9ee35ab9
 gb200-oke-ubuntu-training-kubeflow: e6bcb925712d588a1596bb9825169a944609a7f6b0ab8981810877652d678f5d
 gb300-any: 2cd40fe86553cc08b3bfdf3f8f5c11e3c50e84a4d0b22494f274fa62e9ab01e9
-gb300-eks-ubuntu-inference-dynamo: 8f4a35471d993926d3877d22ee3a379755d5e8a7c37580a37bdf808fd6287187
-gb300-eks-ubuntu-training-kubeflow: 51d4cfa9314323d79c152f88157c2d589123417403d7c805c5908f1e78e9fb38
-h100-aks-ubuntu-inference-dynamo: 60106eca2412d6fad922a4524a09d2d68b7526eaa7f3ff49fd43d0cd7a26c4ce
-h100-aks-ubuntu-training-kubeflow: 231fb345f168a21fc81a5b2ee6fb3a072dd88ac5a64f4b83368a47c0f016bc15
-h100-aks-ubuntu-training-slurm: 4e358a78b920a7abe8f29562ee74c3ca93d9ed9cd89150ca4bf7ceb73ac680d7
+gb300-eks-ubuntu-inference-dynamo: 5cdb2d2f2c690b37cae923fb2f6839973c7c6195c9d7ff408dd6207bb9bcb556
+gb300-eks-ubuntu-training-kubeflow: aaf2e991426b5407d398398b9d76c1fd071cd4056b57deea426cfa45decb3963
+h100-aks-ubuntu-inference-dynamo: 617a5e29126a85d656fce3521d3c3bc978fd12b675a8d1201c3e160991c01d6d
+h100-aks-ubuntu-training-kubeflow: bb0d9f74c1f2e4996a61d8317842aca97a089cb7399c83d0d77c8e123588c60d
+h100-aks-ubuntu-training-slurm: d2fdb66f611e2a710ff1e5f0cabbbdb0a07504012af6ed43fe2b063e77a53732
 h100-any: b96b596863029f47ee8759d159e12db7c9863589379c62ab5d859287bb353667
-h100-bcm-ubuntu-training: aa4c2bd1b395fd29b57dd9226ac0b04eaf3edcd39a853a113a379dfea2f31cb9
-h100-eks-ubuntu-inference-dynamo: 71d5ea9ccb46dd4cb1cf1685faa6d3f8b8404dacf38935f6ce5ca319c6c8b99b
-h100-eks-ubuntu-inference-nim: 1d79a4e312bc47486493bf835ac2264d8786c849b6e6c88c355d7c9084381df6
-h100-eks-ubuntu-training-kubeflow: a121f87063837539f4c26710d37e587d8c3c3be95a7446b6e80088273c1e7625
-h100-eks-ubuntu-training-slurm: d37883a04011b600fc1a24350c20044f08c317138f40bfe674cb0a7ba235d726
-h100-gke-cos-inference-dynamo: fbae6625cfea5c37079cae55bfbd05d177f59c9c8c766b098f3291065b0c660c
-h100-gke-cos-training-kubeflow: 4486d628426f471c3b1ec087947e798ae6998cbe339bf8188570fddb9450a135
-h100-gke-cos-training-slurm: fa6d26058170f02e3ae099c57ba6cb2afa8d890298d6efc52cc21d929d34c325
+h100-bcm-ubuntu-training: 2bc1d46a22090010628a586a044fee498de3d9dce9d12cc42c72b56260e4dd2b
+h100-eks-ubuntu-inference-dynamo: 969949076b2c67c77a889203441ee35e3e31cdd0bb23372ab0df9e01d53f5ec0
+h100-eks-ubuntu-inference-nim: 90c00f509e91a3f539584bfa34a008ea7e8d3c27d076507cdee6225d94f0d55a
+h100-eks-ubuntu-training-kubeflow: 5c550f723b74d8d6cb8e7438ed9807e6188fbe113a18d1c9824f58ce3e516460
+h100-eks-ubuntu-training-slurm: 0d0b8e32ac8556fc150a57923f05eefda4a60d4d4f83f721ddfea57736d74e02
+h100-gke-cos-inference-dynamo: fbef3acb866c687f21733b853390daff448a6c85941cbefea16513ab88914402
+h100-gke-cos-training-kubeflow: 2138cef5affc58ebc49a846c01ed4746ba6fd5837031a84876d826ccf5844de2
+h100-gke-cos-training-slurm: 65ef9ab00b9214eb939aa4e09331212c807e1da6c05c158d65ec04488cf2b7b7
 h100-kind-inference-dynamo: 3b5c3aa86fe396d355fc77a6a7bd2fb6b87d7b7d9a6abf272b2ed41023920673
 h100-kind-training-kubeflow: c2a29960328fc03e6ce67e5e8f25d6fe750886909ca31ee63750b5150a1e6a76
 h100-kind-training-slurm: 3cfbdfb63490de808042b535b8ad92ba45e1dd63d02401ecf57163765f36a4fe
 h200-any: c84e22d03d87a174b5215d883e9131e8a6c5f1befedc179a2cd2ca2bded39144
-h200-eks-inference: bbae5686447c4b4c0dec4b8a5e263710fe67f6fce61b2635bab3a9a335502430
-h200-eks-training: 57b10be6b4878f1b9aabfa2df09721cb7fa362cba411335ad3052f50ab3822c2
+h200-eks-inference: 4d33ad82a19d6307d3bc602388c73a7ecd782ef7fae0138c9a2dfe94e96b94ed
+h200-eks-training: 101b06af7364d15913fbcdad731ec4b7ed3589e7cc5d6096ca4556deb34b183e
 l40s-any: 384bb817ae61f86388cb87e8ff28287cc518988c3e0f65468d64f4e488be58ca
 l40s-oke-inference: 4fd3eaa4c3b3b9f9aae2597abe450c8c6f69ed44c8a25c858d38d264e175d377
 l40s-oke-training: b18da6c239e730f5e11e8da9a109e9e95fc5ddd5f1e947666b8119db5b517d3f
 monitoring-hpa: b35a6c42d73788facd17b104b554bf6d2ae32bf73b0193424a72e566f1b6ceaf
 ocp-inference-nim: bb00cdb191823b32da334bea70826c8a92c11b62d0e096918d43a8e1b043c361
 ocp-training: d7a213263630f2c25982d6f4a144df7d9d5784428d296ccda1b4dab5a42b98bb
 rtx-pro-6000-any: 5500f71ae8812ed1541588222a1dc61f0d4e1c8f8cd7fd9c6a21e29f4d647507
-rtx-pro-6000-eks-ubuntu-inference-dynamo: 16a4de23f7d145e56f8da9ee3e067c47cd354e37b26cfd1f3c52ba4ff7728a93
-rtx-pro-6000-eks-ubuntu-inference-nim: eb765d6af75d71c346a3ce1f8bd8c6c8ab9f3282327988a2eb8a100386764b62
-rtx-pro-6000-eks-ubuntu-training-kubeflow: bc961a695a0e639e6fe79727f0284c9635e26c208824c63bac8526648446b53c
+rtx-pro-6000-eks-ubuntu-inference-dynamo: 20c5a486b2c20ad68ebcefe857babdce28bfa143c8016f9a402caf173ad3fdeb
+rtx-pro-6000-eks-ubuntu-inference-nim: 13916d712f33b442dbe749c66c8a99cfb150096d5a575f573702c6c0a692deaa
+rtx-pro-6000-eks-ubuntu-training-kubeflow: 5255dfd871c4b433d4ace1e6046434cc886716323c8fbfd67f7ce52b3af37f00
 rtx-pro-6000-lke-ubuntu-inference: d08362d22a96a8992c4669d6bf334fc06f6cbf050a883f456fe10674d9d925eb
 rtx-pro-6000-lke-ubuntu-training: 5b0b3d9cc55eea2f60bcc80a54a250e5a03a204dc506863d137d75b8a70a4072

pkg/recipe/testdata/catalog_parity_golden.yaml

@@ -3,51 +3,51 @@
 #
 # One entry per leaf overlay: sha256 of its deterministically-marshalled
 # resolved recipe. A moved digest means that recipe's resolved bytes changed.
-a100-aks-ubuntu-training-kubeflow: 3dfa350aa517fa401b8474629659be7146aacbc25805e70de241e287f4b2e659
+a100-aks-ubuntu-training-kubeflow: 3d8d77ac7ac29bf13253410e0b0486bc322d69d66d8778884df3375619566d58
 a100-any: 62d1581b21fecd3465b69659bac3dc9e5438eb2612c98c1acdd7b19cc9237084
-a100-eks-ubuntu-training-kubeflow: 974a3c313452457605a42ae6e4669ca15ba747e91a4b8d0da85944323ff45291
-a100-gke-cos-training-kubeflow: 17df2d55ab7b62e3d1357dba60a20289881c2f647797dd5efb8b34b999e790fa
+a100-eks-ubuntu-training-kubeflow: 0196adfa0c77f230d95ed3ebee2bdade8ac720f631cd3bbe2b8cec7d55094a10
+a100-gke-cos-training-kubeflow: d31373e01f42088549d6d0f60a3f4946fe921d9182cd7165780c8dd90f9ae771
 a100-oke-ubuntu-training-kubeflow: 310a6ee2abdd7f8508882429d657c01bd23458ba635236c410934825e4f617dc
 b200-any: 63dbc4fe27b84321e8b1991cd98c13f85395a3919d13e0c6891599eaa1f1e729
-b200-gke-cos-inference-dynamo: 6efe04d7048b6d9848c07bd90a315ed0a0641905d9c6794b22271c49323d6fdb
-b200-gke-cos-training-kubeflow: e151a51e1ad7ec31f77a7ec7c24961b8fdce94d569127cf5114864ba40ba1462
-bcm-inference: 43ab64c51d714450cbdcdb6b16022b9a89ddab55cab0c56e6de7a5d8d0f1e4a1
+b200-gke-cos-inference-dynamo: 8d74ebad963238c97de614bac3795922aa92a672c0a424fd84547669e2502795
+b200-gke-cos-training-kubeflow: 2e87d5c8c8b1882a1c7fe501f44199264380a2597884d448677c889f0482c55f
+bcm-inference: 92fa64d3af61026891b37cbf692d2663be7b2484078ab65e2653d18fc2bfe6f8
 gb200-any: 9ee6ab187b4c1b93cd859d86ada35e922d0873ffc6fe892cb084ad98baf09727
-gb200-eks-ubuntu-inference-dynamo: c0de54a7ade04e826a1f7cb1ef9c88c28b60c4aad6357eb0ee8fe9063112a3ec
-gb200-eks-ubuntu-training-kubeflow: 19220d5a2e9ec32954bd1b32bb4e874cc9f57ca1119d7e04990330ea2e3a59c7
-gb200-eks-ubuntu-training-slurm: 86a9358483ebb372d6a62d4702a585c6302f56518c509f2f42e414915fac8d28
+gb200-eks-ubuntu-inference-dynamo: 1c349d6a6893643c3a16dbf0af62a8e5b5bfdd3bda8c8baec65351c2fccb3ccf
+gb200-eks-ubuntu-training-kubeflow: 63b2fbd3a195bac682bb45263e5d5d0ec479a6571dd38ef0505eeb4f287b29e9
+gb200-eks-ubuntu-training-slurm: c35e33618c0a8bb548bbe3c5d6765faf90cebe1427478ef97089f7a30bb37a6c
 gb200-oke-ubuntu-inference-dynamo: 138c28fdd64d4a08a2c1349b5b998f2ee68cdf8de452724510d00f7296b1a5cc
 gb200-oke-ubuntu-training-kubeflow: 3959f47474edfac9b53a65939117d4364551fde0c9324246f195f144fe3beb12
 gb300-any: 7f68607dffbbfc912f19b00ffa833caa81ca675d25bc22cf612dbd0b76f39475
-gb300-eks-ubuntu-inference-dynamo: 44bdad10e2340b164d1b2f906cdc61791f4eb5d2ab7ac83257ec725284f401e4
-gb300-eks-ubuntu-training-kubeflow: 23ff8f6bc4ac65d877ae79215ed7e22b817c62e2fbc974793e8fe67f7a1de89c
-h100-aks-ubuntu-inference-dynamo: 07117b4d11ec6042258f42fb08781b218880f58c301ec71f641beedef8d62806
-h100-aks-ubuntu-training-kubeflow: 2b23859469f426f76d39136a561eacee0e12cfe43a56c2abe72b0501bf9236ef
-h100-aks-ubuntu-training-slurm: 9776f8101cc5d7755681d6424f8b3f74f159af8f5c1436847bfa2c72ccf0c3ea
+gb300-eks-ubuntu-inference-dynamo: 6fda59c645b0407833c0ab0a130b00881075699a8cd57fcd87f3740a71f6642a
+gb300-eks-ubuntu-training-kubeflow: 5303e5ebf8dc2ec13c2cff60c7caba2ef53d1a65bfd69d06f4f78cbb93835c49
+h100-aks-ubuntu-inference-dynamo: 9fe4d82412d250d4004efc56fa4ddd6b8b1a9166387455e96ac2c2284d323107
+h100-aks-ubuntu-training-kubeflow: 72a8ca777c333e0db82717f53a706ae9155163ff0c73fd4e5ed8d71badda5811
+h100-aks-ubuntu-training-slurm: ef48d77171817bb5b24815d128457c246e0e5dc0fe850aa6881276952e590055
 h100-any: f5a55f03948075adde9b1b24fd650d98fa4d35b85f394d071ad3365ba05588a5
-h100-bcm-ubuntu-training: a4c56385b4f0908ebfb022899d5075555e3179d5edb55f08cc062548c196c318
-h100-eks-ubuntu-inference-dynamo: b79ae736a7d9bb20ca2ddb8117aae7e6cdd955f89b8face685b614701a7d64c9
-h100-eks-ubuntu-inference-nim: 9b52a271c4afd205b5b6c0346513e5980f052bb3d6b8a5c8c04c1ef9d2d101e1
-h100-eks-ubuntu-training-kubeflow: 501e3007f9d1b2391b6f196c5fdd769e9a2f180f33c25a9b4592c01dc4dc282a
-h100-eks-ubuntu-training-slurm: 4f5a721af39c8ea372fcdb85e5e740065b99ed4f1287d355d39174d09d023657
-h100-gke-cos-inference-dynamo: 3ba2ec40aecbd3910785df7343896773e07c563901373fdff34fcfb4bca0aba6
-h100-gke-cos-training-kubeflow: 52a6517d46a6be029a12115d8d0f7ccc0a4516270cc03ce9b172f5b3c03b1cd5
-h100-gke-cos-training-slurm: 6899367250adb995f0383eaee6cf0762673495b2b7442ff98e7296731edfb917
+h100-bcm-ubuntu-training: 76746f45add1fb6b0d7b6441b4d9581acceb3b5d3d4800b943c1cf460c92fbc5
+h100-eks-ubuntu-inference-dynamo: 0a31f03010849f76b385ce81c21e499af884f41bee89de47b9bacf03f8b6e311
+h100-eks-ubuntu-inference-nim: 9c6a4839bf5b59f620093427de0e138a9fafaf1f55f3fa7f555b1ee33b914a3a
+h100-eks-ubuntu-training-kubeflow: bfcbd5e14fe43209f2401e429d39b5f7a52d049832db07e5b9595d130ab6e2fc
+h100-eks-ubuntu-training-slurm: 8d29c2be5c3c0dc8396b0df53111a485f6d57771394df8ef135e4b052d681064
+h100-gke-cos-inference-dynamo: 474f349532b9d678ac45a0b52f032c968fa058f2ccb9543ed7d4d9e0838c846d
+h100-gke-cos-training-kubeflow: 3eeb5813928b9df57346b0c1237471bea704c683cf0cb60712e760d7f376b148
+h100-gke-cos-training-slurm: ad9fb0c594ee3f41d55453eb25871fd00a4a4283f03c67f6f855960a3262c753
 h100-kind-inference-dynamo: 0e2d552b62b55b91f960c9721ed128b1d07f68be8517799473fbaa3baa1cc6e5
 h100-kind-training-kubeflow: 4bb03659faaa6a9a66d58dfc95572bdb54b75010c219c4bbefbd0eafeb8ed867
 h100-kind-training-slurm: 0245c786638ade1023ba4dc49e0f2efd56d2f789bdf447e946bbc3a3e652e734
 h200-any: acf1986d76037dddd025125eaa34eb7ecf3d31fe62cb80e7dc89dd042ebb42f5
-h200-eks-inference: badc48bec4aa553f3b9aad5e89ee0d8fcbae3ee5d393b885f1aee0377024d764
-h200-eks-training: 3b7d225d76c247691cfbb0fe91bd98f5e4bb2863f4403df89ff2daa0a49f7b8f
+h200-eks-inference: d1ff148001722e9d5e23606bd7770f8120506f08c10f0c132e0c379f535609c5
+h200-eks-training: 98d04b3c81a059a0321451a39772af8d0d3310ad1fb861837f87d82dd997c4a1
 l40s-any: 89210ba5815f93ce3d7b1a2a78f7ba75d083556e940603a1bc812eac9aeccd0a
 l40s-oke-inference: 6c75ef93e86fb21dbdf73ac97089b894ca8da832164dc807e94592493ab7f7cc
 l40s-oke-training: 2a35011bd59cf3c727cea158d504c51bd59cba069620d3d6a4988d68972dee74
 monitoring-hpa: 15e93304e7e68997e7b15a0be4af360a29f10c869502e6dccb6a6cefb6519fd5
 ocp-inference-nim: 49245ecfc91f67fcc2ed0b305b00046a4164628cafc75ab98542e6dafa626821
 ocp-training: aaa8cd7f45ef42235b075936461d715e2e017ea2802df9e8017de38304a17915
 rtx-pro-6000-any: 9639d67639256a44be8566f3f8eda49ceb4025b6b140a3bc2a4b6d4768d07ffa
-rtx-pro-6000-eks-ubuntu-inference-dynamo: 14c5d90b608eda5dd48cfb22a4ceabc50cc8ca50acf08df5adea5e2059f6fe60
-rtx-pro-6000-eks-ubuntu-inference-nim: 59cafd3a23b4cd52cdfff318fd39dcc5bd8b587ccd49146d5868bde69127d3b1
-rtx-pro-6000-eks-ubuntu-training-kubeflow: ff3a0be78e21b186458ebe1183742ba80077a7608b4d1d7ed1c27f9fdec486cf
+rtx-pro-6000-eks-ubuntu-inference-dynamo: cbe11051d979ae20e83471142c4582e1504fd24f55898a00afcededbe143bba6
+rtx-pro-6000-eks-ubuntu-inference-nim: 08b73d58d0d60681be55ee1402bdec5bf1367a0cb137a273ca27bec0788cc020
+rtx-pro-6000-eks-ubuntu-training-kubeflow: c2595e2994d56a735714103eedd30c6afd45921f458bf026c3432017b5eba60e
 rtx-pro-6000-lke-ubuntu-inference: 28cdab0531f8ed6018391aba7cab1fbfad8ba551392de1b9138f3b88042f5340
 rtx-pro-6000-lke-ubuntu-training: 312c4cf49d0e8cec0377cd9858338d7106226a2252d81828a75c1d426a194a2f

recipes/checks/nodewright-customizations/health-check.yaml

@@ -50,8 +50,8 @@ spec:
   steps:
     - name: validate-skyhook-cr-complete
       try:
-        # Assert a Skyhook CR in the skyhook namespace has status.status ==
-        # "complete" per the Skyhook CRD schema.
+        # Assert a Skyhook CR has status.status == "complete" per the Skyhook
+        # CRD schema. Skyhook is cluster-scoped.
         #
         # Deliberately name-agnostic. The manifests do not agree on a name —
         # tuning.yaml / tuning-gke.yaml / tuning-generic.yaml render "tuning"
@@ -65,7 +65,10 @@ spec:
         # Per-CR completeness is not weakened by this: the render-aware Go
         # readiness check (verifyNodewrightReady) derives the expected CR names
         # from the component's rendered manifests and requires every one of
-        # them to reach complete, reporting each failure by name.
+        # them to be live (not Terminating) and complete, reporting each failure
+        # by name. That per-name gate is what binds readiness to the CRs the
+        # recipe declared — this assert alone would accept a stale or unrelated
+        # complete Skyhook.
         - assert:
             resource:
               apiVersion: skyhook.nvidia.com/v1alpha1

recipes/overlays/gb300-eks-training.yaml

@@ -118,11 +118,22 @@ spec:
       # actually carried traffic, so a silent fallback to Socket or NET does
       # not masquerade as a pass.
       #
-      # Thresholds are mirrored from the GB200 EKS baseline as a conservative
-      # floor until GB300 numbers are measured. GB300 NVL72 exceeds GB200 on
-      # both NVLink and EFA bandwidth, so these floors should not false-fail a
-      # healthy run; they are deliberately loose and must be re-baselined on
-      # real GB300 hardware before they are treated as a regression gate.
+      # Measured on GB300 (2x p6e-gb300r.36xlarge, 8 GPUs, NCCL 2.27.7,
+      # 16 GiB message): net 44.04 GB/s, nvls 841.17 GB/s. Both land at ~88%
+      # of their respective line rates (a single 400 Gb/s EFA device; 18
+      # NVLink-5 links at 53.125 GB/s = ~956 GB/s per GPU).
+      #
+      # nvls is floored at 700 from that measurement, leaving ~17% margin
+      # before the evaluator's additional 10% tolerance. The GB200 value of
+      # 500 was far too loose for GB300's NVLink fabric.
+      #
+      # net stays at the GB200 value of 40 deliberately. The reference cluster
+      # advertises vpc.amazonaws.com/efa: 1, so 44.04 GB/s is capped by NIC
+      # provisioning rather than by GB300 — it is not a representative
+      # scale-out figure. Raising this floor now would make the recipe
+      # unusable on the only GB300 cluster available to test against. It must
+      # be re-baselined on a fully-provisioned rack, where a materially higher
+      # floor is expected.
       checks:
         - nccl-all-reduce-bw-net
         - nccl-all-reduce-bw-nvls
@@ -142,7 +153,7 @@ spec:
         - name: nccl-all-reduce-bw-net
           value: ">= 40"
         - name: nccl-all-reduce-bw-nvls
-          value: ">= 500"
+          value: ">= 700"
     conformance:
       checks:
         - platform-health

validators/deployment/expected_resources.go

@@ -610,6 +610,19 @@ func nodewrightStatusFailure(verifyCtx context.Context, dynClient dynamic.Interf
 		}
 		return fmt.Sprintf("Nodewright %s: failed to get: %v", name, getErr)
 	}
+	// Reject a CR that is on its way out even when it still reports complete.
+	// Nodewright uses a deletion finalizer, so an expected Skyhook can sit
+	// Terminating for a while with status.status untouched. Accepting it would
+	// report readiness on the strength of state that is about to disappear —
+	// the same false-PASS direction the Chainsaw executor already guards
+	// against by skipping ghosts on positive assertions (#2041). The nameless
+	// assert in the component health check cannot cover this: it is satisfied
+	// by any live complete Skyhook, including a stale or unrelated one, so this
+	// per-name check is the only gate that binds liveness to the CR the recipe
+	// actually declared.
+	if sk.GetDeletionTimestamp() != nil {
+		return fmt.Sprintf("Nodewright %s: terminating (deletionTimestamp set)", name)
+	}
 	status, found, statusErr := unstructured.NestedString(sk.Object, "status", "status")
 	if statusErr != nil {
 		return fmt.Sprintf("Nodewright %s: failed to read status.status: %v", name, statusErr)