@@ -263,8 +263,9 @@ Also TrainJob-expressible. AICR fixes the resource name and the value is always
263263primary interface. AICR's tested AKS runtime sets it, as does the EKS one — on
264264a multi-NIC node, leaving NCCL to guess can send rendezvous traffic down an
265265interface that cannot carry it. `eth0` assumes the pod's primary interface is
266- named that; under `hostNetwork` or a non-standard CNI it may not be. Confirm on
267- a running pod with `kubectl exec ... -- ip route get 1.1.1.1` and pin whatever
266+ named that; under `hostNetwork` or a non-standard CNI it may not be. Confirm
267+ on a running pod with
268+ ` kubectl exec -n <namespace> <pod> -- ip route get 1.1.1.1` and pin whatever
268269that reports.
269270
270271The same repeat-every-resource caveat applies.
@@ -307,8 +308,9 @@ kubectl logs -n <namespace> -c node --tail=-1 --prefix \
307308
308309This selects the `node` replicated job, which is where a node-only runtime such
309310as `torch-distributed` prints the banner. If your runtime uses an MPI launcher —
310- as AICR's tested AKS runtime does — the ranks are aggregated in the `launcher`
311- pod and the `node` pods run only sshd, so select
311+ as AICR's tested AKS runtime does — the rank processes still execute on the
312+ ` node` pods, but they are started over sshd and `mpirun` aggregates their
313+ stdout in the `launcher` pod, so select
312314` jobset.sigs.k8s.io/replicatedjob-name=launcher` instead. A grep against `node`
313315on such a runtime finds nothing, which is not evidence of socket fallback.
314316
0 commit comments