Skip to content

Commit 709b179

Browse files
authored
docs: finish DaemonSet casing pass and bump a stale aicrd tag (#2339)
Signed-off-by: Yuan Chen <yuanchen97@gmail.com>
1 parent 06d2cbb commit 709b179

2 files changed

Lines changed: 5 additions & 5 deletions

File tree

docs/integrator/kubernetes-deployment.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -609,7 +609,7 @@ as HTTP 429 responses with the `X-RateLimit-*` headers.
609609
```shell
610610
# Update image
611611
kubectl set image deployment/aicrd \
612-
api-server=ghcr.io/nvidia/aicrd:v0.8.0 \
612+
api-server=ghcr.io/nvidia/aicrd:v0.19.0 \
613613
-n aicr
614614

615615
# Watch rollout

docs/user/cli-reference.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1526,7 +1526,7 @@ The `--accelerated-node-selector` and `--accelerated-node-toleration` flags cont
15261526

15271527
NFD (Node Feature Discovery) workers must run on **all nodes** (GPU, CPU, and system) to detect hardware features. This matches the gpu-operator default behavior where NFD workers also run on control-plane nodes. The `--accelerated-node-selector` is intentionally not applied to NFD workers so they are not restricted to GPU nodes.
15281528

1529-
> **Note:** When no `--accelerated-node-toleration` is specified, a default toleration (`operator: Exists`) is applied to both GPU daemonsets and NFD workers, allowing them to run on nodes with any taint.
1529+
> **Note:** When no `--accelerated-node-toleration` is specified, a default toleration (`operator: Exists`) is applied to both GPU DaemonSets and NFD workers, allowing them to run on nodes with any taint.
15301530
15311531
**Example:**
15321532

@@ -1544,7 +1544,7 @@ aicr bundle --recipe recipe.yaml \
15441544
> **Cluster node requirements:** This example assumes the cluster has nodes labeled `nodeGroup=system-worker` with taints `dedicated=system-workload:NoSchedule,NoExecute` for system infrastructure, and GPU nodes labeled `nodeGroup=gpu-worker` with taints `dedicated=worker-workload:NoSchedule,NoExecute`.
15451545
15461546
This results in:
1547-
- **GPU daemonsets** (driver, device-plugin, toolkit, dcgm): `nodeSelector=nodeGroup=gpu-worker` + tolerations for `dedicated=worker-workload` with both `NoSchedule` and `NoExecute`
1547+
- **GPU DaemonSets** (driver, device-plugin, toolkit, dcgm): `nodeSelector=nodeGroup=gpu-worker` + tolerations for `dedicated=worker-workload` with both `NoSchedule` and `NoExecute`
15481548
- **NFD workers**: no nodeSelector (runs on all nodes) + tolerations for `dedicated=worker-workload` with both `NoSchedule` and `NoExecute`
15491549
- **System components** (gpu-operator controller, NFD gc/master, dynamo grove, agentgateway proxy): `nodeSelector=nodeGroup=system-worker` + tolerations for `dedicated=system-workload` with both `NoSchedule` and `NoExecute`
15501550

@@ -2629,9 +2629,9 @@ Components that use operator patterns with custom resources that reconcile async
26292629

26302630
##### DRA kubelet plugin registration
26312631

2632-
After installing `nvidia-dra-driver-gpu`, the script automatically restarts the DRA kubelet plugin daemonset. This is a best-effort mitigation for a known issue: after uninstall/reinstall, the kubelet's plugin watcher (`fsnotify`) may not detect new registration sockets, causing `DRA driver gpu.nvidia.com is not registered` errors.
2632+
After installing `nvidia-dra-driver-gpu`, the script automatically restarts the DRA kubelet plugin DaemonSet. This is a best-effort mitigation for a known issue: after uninstall/reinstall, the kubelet's plugin watcher (`fsnotify`) may not detect new registration sockets, causing `DRA driver gpu.nvidia.com is not registered` errors.
26332633

2634-
If DRA pods fail with this error after redeployment, the daemonset restart alone may not be sufficient — a **node reboot** is required to reset the kubelet's plugin registration state. To reboot GPU nodes:
2634+
If DRA pods fail with this error after redeployment, the DaemonSet restart alone may not be sufficient — a **node reboot** is required to reset the kubelet's plugin registration state. To reboot GPU nodes:
26352635

26362636
```bash
26372637
# Cordon, drain, and reboot the affected node

0 commit comments

Comments
 (0)