Skip to content

Commit 2b2307c

Browse files
committed
docs: add --tail=-1 to the label-selector log command
Review catch, and it is a defect this PR introduced. kubectl's own help states the default plainly: 'Defaults to -1 with no selector, showing all log lines otherwise 10, if a selector is provided.' So switching from a named pod to a label selector silently capped output at the last 10 lines per worker, which would omit the 'NCCL INFO ... Using network' banner NCCL prints during init — the exact line the command greps for. The check would have reported nothing on a correctly attached job. Refs #2306 Signed-off-by: Yuan Chen <yuanchen97@gmail.com>
1 parent 890be93 commit 2b2307c

1 file changed

Lines changed: 3 additions & 1 deletion

File tree

docs/user/fabric-attached-training.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -295,7 +295,9 @@ socket fallback:
295295

296296
```shell
297297
# Select workers by label rather than guessing the generated pod name.
298-
kubectl logs -n <namespace> -c node \
298+
# --tail=-1 is required: with a selector, kubectl defaults to the last 10 lines
299+
# and would omit the transport banner, which NCCL prints during init.
300+
kubectl logs -n <namespace> -c node --tail=-1 \
299301
-l jobset.sigs.k8s.io/jobset-name=<trainjob-name>,jobset.sigs.k8s.io/replicatedjob-name=node \
300302
| grep -i 'NCCL INFO.*Using network'
301303
```

0 commit comments

Comments
 (0)