Skip to content

Commit 45031f5

Browse files
committed
docs: state MPI rank placement precisely, use explicit exec placeholders
Signed-off-by: Yuan Chen <yuanchen97@gmail.com>
1 parent 4c4417c commit 45031f5

1 file changed

Lines changed: 6 additions & 4 deletions

File tree

docs/user/fabric-attached-training.md

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -263,8 +263,9 @@ Also TrainJob-expressible. AICR fixes the resource name and the value is always
263263
primary interface. AICR's tested AKS runtime sets it, as does the EKS one — on
264264
a multi-NIC node, leaving NCCL to guess can send rendezvous traffic down an
265265
interface that cannot carry it. `eth0` assumes the pod's primary interface is
266-
named that; under `hostNetwork` or a non-standard CNI it may not be. Confirm on
267-
a running pod with `kubectl exec ... -- ip route get 1.1.1.1` and pin whatever
266+
named that; under `hostNetwork` or a non-standard CNI it may not be. Confirm
267+
on a running pod with
268+
`kubectl exec -n <namespace> <pod> -- ip route get 1.1.1.1` and pin whatever
268269
that reports.
269270

270271
The same repeat-every-resource caveat applies.
@@ -307,8 +308,9 @@ kubectl logs -n <namespace> -c node --tail=-1 --prefix \
307308

308309
This selects the `node` replicated job, which is where a node-only runtime such
309310
as `torch-distributed` prints the banner. If your runtime uses an MPI launcher —
310-
as AICR's tested AKS runtime does — the ranks are aggregated in the `launcher`
311-
pod and the `node` pods run only sshd, so select
311+
as AICR's tested AKS runtime does — the rank processes still execute on the
312+
`node` pods, but they are started over sshd and `mpirun` aggregates their
313+
stdout in the `launcher` pod, so select
312314
`jobset.sigs.k8s.io/replicatedjob-name=launcher` instead. A grep against `node`
313315
on such a runtime finds nothing, which is not evidence of socket fallback.
314316

0 commit comments

Comments
 (0)