Skip to content

Commit f721eed

Browse files
committed
ci(uat): remove AWS GB200 nightly UAT lane
GB200 hardware is being retired. Remove the aws-gb200 reservation row, its GB200 EKS cluster/test configs, the dedicated cross-account janitor job, and the broker/coverage test expectations that pinned it. The multi-arch/arm64 build machinery in uat-aws.yaml is left fully intact for a successor ARM instance type. Signed-off-by: Nathan Hensley <nhensley@nvidia.com>
1 parent 534a9f5 commit f721eed

8 files changed

Lines changed: 6 additions & 498 deletions

File tree

.github/workflows/uat-janitor.yaml

Lines changed: 0 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -163,44 +163,6 @@ jobs:
163163
GH_TOKEN: ${{ github.token }}
164164
run: ./.github/scripts/uat-janitor.sh aws
165165

166-
# GB200 lives in a SEPARATE AWS account — the cross-account Capacity Block in
167-
# tenancy 897722687756 (tests/uat/aws/cluster-config-gb200.yaml), not the
168-
# standard 615299774277 the `aws` job above covers — so it needs its own role
169-
# assumption + config, or aws-gb200 orphans are never discovered or reaped.
170-
# PREREQUISITE: infra/uat-aws-account applied in 897722687756 (GitHub OIDC
171-
# provider + github-actions-role-aicr whose trust permits this workflow's ref).
172-
# Until then the auth step fails harmlessly and the first dry-run surfaces the gap.
173-
aws-gb200:
174-
if: >-
175-
github.repository == 'nvidia/aicr' &&
176-
github.ref == 'refs/heads/main' &&
177-
(github.event_name == 'schedule' || inputs.cloud == 'all' || inputs.cloud == 'aws')
178-
runs-on: ubuntu-latest
179-
timeout-minutes: 60
180-
permissions:
181-
contents: read
182-
id-token: write
183-
actions: read
184-
env:
185-
AWS_REGION: "us-east-1"
186-
JANITOR_CONFIG: "tests/uat/aws/cluster-config-gb200.yaml"
187-
JANITOR_ACTUATOR_IMAGE: "ghcr.io/mchmarny/cluster/eks@sha256:8bc33d14e6e5659d242aa2cdbcd69fca389cfcdb9e94f7a7a1b82fd33176befa"
188-
steps:
189-
- name: Checkout
190-
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
191-
with:
192-
persist-credentials: false
193-
- name: Authenticate to AWS (GB200 account)
194-
uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3
195-
with:
196-
role-to-assume: "arn:aws:iam::897722687756:role/github-actions-role-aicr"
197-
aws-region: ${{ env.AWS_REGION }}
198-
role-session-name: GitHubActions-UAT-Janitor
199-
- name: Reconcile GB200 orphans
200-
env:
201-
GH_TOKEN: ${{ github.token }}
202-
run: ./.github/scripts/uat-janitor.sh aws
203-
204166
azure:
205167
if: >-
206168
github.repository == 'nvidia/aicr' &&

docs/design/017-uat-cluster-name-convention.md

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -105,7 +105,6 @@ reservation name is too long and too cloud-scoped to be. Launch values:
105105
| `aws-h100` | `ah1` |
106106
| `gcp-h100` | `gh1` |
107107
| `azure-h100`| `zh1` |
108-
| `aws-gb200` | `ag2` |
109108
| `kind-h100` | `kh1` |
110109

111110
The nvkind lane (`kind-h100`) gets a slug for uniformity, though it provisions

infra/uat/reservations.yaml

Lines changed: 0 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -147,51 +147,6 @@ reservations:
147147
nightly-intent-min-versions:
148148
inference: v0.18.0
149149
training: v0.18.0
150-
- name: aws-gb200
151-
slug: ag2
152-
cloud: aws
153-
# CROSS-ACCOUNT: the GB200 Capacity Block lives in a SEPARATE tenancy
154-
# (dgx-cloud-runai-prod-22, account 897722687756), not the standard UAT
155-
# account. The account is NOT a field here: the AWS pipeline derives the
156-
# tenancy it assumes the OIDC role into from the target cluster-config's
157-
# .deployment.tenancy (cluster-config-gb200.yaml sets it to 897722687756),
158-
# the single source of truth. Prereq: apply infra/uat-aws-account in
159-
# 897722687756 (GitHub OIDC provider + github-actions-role-aicr).
160-
# GB200 reservations are Capacity Blocks (marketType: capacity-block in
161-
# cluster-config-gb200.yaml), not standing on-demand CRs like the h100 rows,
162-
# so a run must fall inside the block's active window or actuator apply
163-
# fails to acquire capacity. This particular block (cr-0e2f3833a602809a6,
164-
# CapacityBlockId cbr-0a6a5543dd5c74727) is booked long: State=active,
165-
# StartDate 2025-11-15, EndDate 2029-08-22 (TotalInstanceCount 18). A 04:00
166-
# UTC nightly cron therefore always falls inside the window today, which is
167-
# why nightly enrollment below is safe without broker window-awareness. If
168-
# the block is ever re-booked to a shorter window, add a scheduling gate to
169-
# the broker (or revert to `nightly-intents: []`) before it lapses.
170-
reservation-id: cr-0e2f3833a602809a6
171-
accelerator: gb200
172-
# 4 Blackwell GPUs per p6e-gb200.36xlarge node; the cluster-config runs 2
173-
# nodes (activates the multi-node NCCL East-West fabric check).
174-
gpu-count: 4
175-
cluster-config-path: tests/uat/aws/cluster-config-gb200.yaml
176-
test-config-dir: tests/uat/aws/tests
177-
# Enrolled in the nightly batch with both intents, matching the aws-h100
178-
# row. Training runs the Kubeflow TrainJob CUJ; inference stands up and
179-
# validates the served inference stack (the vLLM serve run itself stays
180-
# disabled repo-wide, #1644). Each (version × intent) is a full
181-
# provision→CUJ→teardown cell dispatched through the shared per-reservation
182-
# lease, so the two intents serialize with no contention. Safe to enroll
183-
# because the Capacity Block is booked active through 2029 (see the block
184-
# note above) and the GB200 EKS UAT config first shipped in v0.18.0.
185-
nightly-intents: [training, inference]
186-
# Gate release cells to v0.18.0: cluster-config-gb200.yaml and the GB200 EKS
187-
# UAT lane first shipped in v0.18.0 (absent in v0.17.0 and earlier), so a
188-
# release cell below that tag has no GB200 config to run and would fail
189-
# closed. `main` always runs both intents (built from source); older
190-
# releases are skipped until v0.18.0 is the oldest cell in the window. Bump
191-
# if the actual first-supported tag differs.
192-
nightly-intent-min-versions:
193-
training: v0.18.0
194-
inference: v0.18.0
195150
- name: kind-h100
196151
slug: kh1
197152
cloud: kind

pkg/uatbroker/registry_test.go

Lines changed: 4 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -809,15 +809,14 @@ func TestDaytimeAssignmentsNone(t *testing.T) {
809809
}
810810

811811
// TestCommittedRegistryValid guards the actual checked-in registry: it must
812-
// parse, validate, and carry the launch reservations plus the aws-gb200
813-
// bring-up row. A bad data edit fails here before it can break the broker
814-
// workflows.
812+
// parse, validate, and carry the launch reservations. A bad data edit fails
813+
// here before it can break the broker workflows.
815814
func TestCommittedRegistryValid(t *testing.T) {
816815
reg, err := LoadRegistryFile(filepath.Join("..", "..", "infra", "uat", "reservations.yaml"))
817816
if err != nil {
818817
t.Fatalf("committed reservations.yaml invalid: %v", err)
819818
}
820-
want := map[string]string{"aws-h100": CloudAWS, "gcp-h100": CloudGCP, "azure-h100": CloudAzure, "aws-gb200": CloudAWS, "kind-h100": CloudKind}
819+
want := map[string]string{"aws-h100": CloudAWS, "gcp-h100": CloudGCP, "azure-h100": CloudAzure, "kind-h100": CloudKind}
821820
for name, cloud := range want {
822821
res, err := reg.Lookup(name)
823822
if err != nil {
@@ -837,7 +836,6 @@ func TestCommittedRegistryValid(t *testing.T) {
837836
"aws-h100": "ah1",
838837
"gcp-h100": "gh1",
839838
"azure-h100": "zh1",
840-
"aws-gb200": "ag2",
841839
"kind-h100": "kh1",
842840
}
843841
for name, slug := range wantSlug {
@@ -890,13 +888,6 @@ func TestCommittedRegistryValid(t *testing.T) {
890888
// acceptance run (29125390442); inference joined after a green
891889
// manual intent=inference dispatch.
892890
"azure-h100": {IntentTraining, IntentInference},
893-
// aws-gb200 enrolled with both intents once the GB200 EKS UAT lane
894-
// shipped (v0.18.0) and the Capacity Block was confirmed booked active
895-
// through 2029 (cr-0e2f3833a602809a6). Release cells are gated to
896-
// v0.18.0 via nightly-intent-min-versions (the config first shipped
897-
// there), so only `main` runs GB200 nightly until v0.18.0 ages into the
898-
// window.
899-
"aws-gb200": {IntentTraining, IntentInference},
900891
// kind-h100 (nvkind real-silicon lane, DC5 #1278) enrolled with both
901892
// intents after green manual H100 acceptance runs (training 29954092703,
902893
// inference 29965464868). Release cells are gated to v0.18.0 via
@@ -920,12 +911,10 @@ func TestCommittedRegistryValid(t *testing.T) {
920911
// a future edit cannot silently drop a gate — which would let a release cell
921912
// run a pre-fix aicr and emit failing/unusable evidence. azure-h100 gates its
922913
// AKS perf + driver-only fixes; kind-h100 gates the uat-kind lane + the
923-
// os-agnostic coordinate fix (#1851); aws-gb200 gates the GB200 EKS UAT
924-
// config, which first shipped in v0.18.0. All landed post-v0.17.0.
914+
// os-agnostic coordinate fix (#1851). All landed post-v0.17.0.
925915
wantMinVersions := map[string]map[string]string{
926916
"azure-h100": {IntentTraining: "v0.18.0", IntentInference: "v0.18.0"},
927917
"kind-h100": {IntentTraining: "v0.18.0", IntentInference: "v0.18.0"},
928-
"aws-gb200": {IntentTraining: "v0.18.0", IntentInference: "v0.18.0"},
929918
}
930919
for name, want := range wantMinVersions {
931920
res, lookupErr := reg.Lookup(name)

tests/uat/aws/cluster-config-gb200.yaml

Lines changed: 0 additions & 157 deletions
This file was deleted.

0 commit comments

Comments
 (0)