@@ -262,7 +262,11 @@ Also TrainJob-expressible. AICR fixes the resource name and the value is always
262262` NCCL_SOCKET_IFNAME` pins NCCL's bootstrap and out-of-band traffic to the
263263primary interface. AICR's tested AKS runtime sets it, as does the EKS one — on
264264a multi-NIC node, leaving NCCL to guess can send rendezvous traffic down an
265- interface that cannot carry it.
265+ interface that cannot carry it. `eth0` assumes the pod's primary interface is
266+ named that; under `hostNetwork` or a non-standard CNI it may not be. Confirm
267+ on a running pod with
268+ ` kubectl exec -n <namespace> <pod> -- ip route get 1.1.1.1` and pin whatever
269+ that reports.
266270
267271The same repeat-every-resource caveat applies.
268272
@@ -297,11 +301,19 @@ socket fallback:
297301# Select workers by label rather than guessing the generated pod name.
298302# --tail=-1 is required: with a selector, kubectl defaults to the last 10 lines
299303# and would omit the transport banner, which NCCL prints during init.
300- kubectl logs -n <namespace> -c node --tail=-1 \
304+ kubectl logs -n <namespace> -c node --tail=-1 --prefix \
301305 -l jobset.sigs.k8s.io/jobset-name=<trainjob-name>,jobset.sigs.k8s.io/replicatedjob-name=node \
302306 | grep -i 'NCCL INFO.*Using network'
303307` ` `
304308
309+ This selects the `node` replicated job, which is where a node-only runtime such
310+ as `torch-distributed` prints the banner. If your runtime uses an MPI launcher —
311+ as AICR's tested AKS runtime does — the rank processes still execute on the
312+ ` node` pods, but they are started over sshd and `mpirun` aggregates their
313+ stdout in the `launcher` pod, so select
314+ ` jobset.sigs.k8s.io/replicatedjob-name=launcher` instead. A grep against `node`
315+ on such a runtime finds nothing, which is not evidence of socket fallback.
316+
305317Expect the plugin name — `FasTrak` for TCPXO, `AWS Libfabric` for EFA, `IB` for
306318InfiniBand. **`Socket` means the fabric is not in use** and the job is running
307319over TCP.
0 commit comments