Skip to content

Commit 9f6fd4c

Browse files
committed
fix(validators): pair the TCPXO sidecar with the installer AICR ships
The performance validator's GKE H100 TrainingRuntime pinned tcpgpudmarxd-dev v1.0.20, which Google pairs with plugin installer v1.0.14. AICR's recipe deploys installer v1.0.15, whose partner is v1.0.21. That runtime is rendered and run on a live cluster during the performance phase, so the benchmark executed against a pair Google documents as unsupported. Renovate excludes **/testdata/** so the pin could never be updated automatically, and ignoreDeps listed only the installer half of the pair, leaving the daemon free to move on its own. Add the daemon so the coupling is symmetric. Also move the demo workload from the v1.0.14/v1.0.20 pair to v1.0.15/v1.0.21 so it matches the installer the recipe deploys, and update the two doc notes that described it as pinned to the earlier pair. Signed-off-by: Yuan Chen <yuanchen97@gmail.com>
1 parent 2baf43a commit 9f6fd4c

4 files changed

Lines changed: 14 additions & 8 deletions

File tree

.github/renovate.json5

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -363,9 +363,14 @@
363363
// workload-side daemon (tcpgpudmarxd-dev) must move together; an
364364
// independent Renovate bump on this image alone would create
365365
// stack version skew. See docs/integrator/gke-tcpxo-networking.md.
366+
// - tcpgpudmarxd-dev: the workload-side half of the same TCPXO pair. Listed
367+
// alongside the installer so the coupling is symmetric — excluding only
368+
// the installer would leave the daemon free to move on its own, which is
369+
// the same stack version skew from the opposite direction.
366370
ignoreDeps: [
367371
"602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/aws-efa-k8s-device-plugin",
368372
"us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpxo/nccl-plugin-gpudirecttcpx-dev",
373+
"us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpxo/tcpgpudmarxd-dev",
369374
],
370375

371376
// Paths Renovate should not scan.

demos/workloads/training/gke-nccl-test-tcpxo.yaml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -127,7 +127,7 @@ spec:
127127
cloud.google.com/gke-accelerator: nvidia-h100-mega-80gb
128128
containers:
129129
- name: tcpxo-daemon
130-
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpxo/tcpgpudmarxd-dev:v1.0.20
130+
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpxo/tcpgpudmarxd-dev:v1.0.21
131131
imagePullPolicy: Always
132132
command: ["/bin/sh", "-c"]
133133
args:
@@ -150,7 +150,7 @@ spec:
150150
- name: LD_LIBRARY_PATH
151151
value: /usr/local/nvidia/lib64
152152
- name: nccl-test
153-
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpxo/nccl-plugin-gpudirecttcpx-dev:v1.0.14
153+
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpxo/nccl-plugin-gpudirecttcpx-dev:v1.0.15
154154
imagePullPolicy: Always
155155
command:
156156
- /bin/sh
@@ -242,7 +242,7 @@ spec:
242242
cloud.google.com/gke-accelerator: nvidia-h100-mega-80gb
243243
containers:
244244
- name: tcpxo-daemon
245-
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpxo/tcpgpudmarxd-dev:v1.0.20
245+
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpxo/tcpgpudmarxd-dev:v1.0.21
246246
imagePullPolicy: Always
247247
command: ["/bin/sh", "-c"]
248248
args:
@@ -265,7 +265,7 @@ spec:
265265
- name: LD_LIBRARY_PATH
266266
value: /usr/local/nvidia/lib64
267267
- name: nccl-test
268-
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpxo/nccl-plugin-gpudirecttcpx-dev:v1.0.14
268+
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpxo/nccl-plugin-gpudirecttcpx-dev:v1.0.15
269269
imagePullPolicy: Always
270270
command:
271271
- /bin/sh

docs/integrator/gke-tcpxo-networking.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -179,7 +179,7 @@ Running a **Kubeflow TrainJob** rather than a bare Pod? A TrainJob cannot add
179179
the `tcpxo-daemon` sidecar, so the wiring must live in a `TrainingRuntime` — see
180180
[Attaching a Training Workload to the Cluster Fabric](../user/fabric-attached-training.md).
181181

182-
See [`demos/workloads/training/gke-nccl-test-tcpxo.yaml`](https://github.com/NVIDIA/aicr/blob/main/demos/workloads/training/gke-nccl-test-tcpxo.yaml) for a complete 2-node NCCL benchmark example. (pinned to an earlier coupled pair, plugin `v1.0.14` with daemon `v1.0.20`)
182+
See [`demos/workloads/training/gke-nccl-test-tcpxo.yaml`](https://github.com/NVIDIA/aicr/blob/main/demos/workloads/training/gke-nccl-test-tcpxo.yaml) for a complete 2-node NCCL benchmark example. (pinned to the same coupled pair the recipe ships, plugin `v1.0.15` with daemon `v1.0.21`)
183183

184184
## NCCL Plugin Version Matching
185185

@@ -258,8 +258,9 @@ NRI profile (recommended, no `hostNetwork`):
258258

259259
```shell
260260
kubectl create ns nccl-test
261-
# Note: this manifest is pinned to the earlier v1.0.14 / v1.0.20 pair.
262-
# Update both images to your cluster's pair before applying.
261+
# Note: this manifest is pinned to the v1.0.15 / v1.0.21 pair, matching the
262+
# installer the recipe deploys. If your cluster runs a different installer
263+
# version, update both images to that cluster's pair before applying.
263264
kubectl apply -f demos/workloads/training/gke-nccl-test-tcpxo.yaml -n nccl-test
264265
265266
# Wait for pods to be 2/2 Running

validators/performance/testdata/h100/gke/runtime.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -215,7 +215,7 @@ spec:
215215
# alongside the worker. Uses NRI device injection (devices.gke.io
216216
# annotation) for GPU device access instead of privileged mode.
217217
- name: tcpxo-daemon
218-
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpxo/tcpgpudmarxd-dev:v1.0.20
218+
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpxo/tcpgpudmarxd-dev:v1.0.21
219219
imagePullPolicy: Always
220220
restartPolicy: Always
221221
command: ["/bin/sh", "-c"]

0 commit comments

Comments
 (0)