Skip to content

Commit a37d5c1

Browse files
committed
docs: AKS needs IPC_LOCK and a memory-backed /dev/shm too
The AKS section documented the RDMA resource request and the NCCL environment but stopped there. Verified against AICR's tested AKS runtime: it grants IPC_LOCK — with a comment explaining that NCCL's IB transport registers pinned buffers via ibverbs and IPC_LOCK is what lifts RLIMIT_MEMLOCK — and mounts a memory-backed dshm at /dev/shm. Allocating the RDMA device does not grant either. The failure mode is the reason this is worth stating: the job fails after the device is successfully allocated, which reads as an NCCL problem rather than a missing capability. Both are now called out, with IPC_LOCK routed through runtimePatches the same way as for EFA. The GKE requirement list also names dshm explicitly rather than leaving it inside the volumes placeholder. Refs #2295 Signed-off-by: Yuan Chen <yuanchen97@gmail.com>
1 parent ae982ef commit a37d5c1

1 file changed

Lines changed: 16 additions & 3 deletions

File tree

docs/user/fabric-attached-training.md

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,8 @@ namespace and reference it from `runtimeRef`.
5050
annotations, on the pod template metadata
5151
- the `tcpxo-daemon` native sidecar, at the version paired with the plugin your
5252
cluster runs
53-
- four hostPath volumes, and `IPC_LOCK` on the worker
53+
- four hostPath volumes plus a memory-backed `dshm` at `/dev/shm`, and `IPC_LOCK`
54+
on the worker
5455
- the NCCL configuration for your plugin release
5556

5657
**A placement sketch**`"<...>"` marks a value you must fill in. It shows
@@ -245,8 +246,20 @@ Also TrainJob-expressible. AICR fixes the resource name and the value is always
245246
value: none
246247
```
247248

248-
The same repeat-every-resource caveat applies. See
249-
[AKS GPU Setup](../integrator/aks-gpu-setup.md) for the cluster-side
249+
The same repeat-every-resource caveat applies.
250+
251+
**`IPC_LOCK` is required here too, and allocating the RDMA device does not grant
252+
it.** NCCL's IB transport registers pinned (memlocked) buffers via ibverbs, and
253+
`IPC_LOCK` is what lifts `RLIMIT_MEMLOCK` — without it the job can fail *after*
254+
the device is allocated, which reads as an NCCL bug rather than a missing
255+
capability. Add it the same way as for EFA: a `runtimePatches` entry setting
256+
`securityContext` on the `node` container.
257+
258+
**A memory-backed `/dev/shm` is also expected.** AICR's tested runtime mounts a
259+
`dshm` volume (`emptyDir: {medium: Memory}`) there; the default 64 MiB `/dev/shm`
260+
is small for multi-process NCCL.
261+
262+
See [AKS GPU Setup](../integrator/aks-gpu-setup.md) for the cluster-side
250263
prerequisites.
251264

252265
## Verifying the fabric is in use

0 commit comments

Comments
 (0)