You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+52-49Lines changed: 52 additions & 49 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,7 +6,12 @@
6
6
7
7
**NVSentinel detects and remediates GPU faults on Kubernetes nodes**
8
8
9
-
A single bad GPU can silently corrupt a training run or leave a node sitting idle for hours before anyone notices. NVSentinel catches these faults as they happen, cordons and drains the affected node, then fixes it with a GPU reset or a reboot, and puts it back into service, no paging required.
9
+
A single bad GPU can silently corrupt a training run or leave a node sitting idle for hours before anyone notices. NVSentinel **detects** faults as they happen, **protects** jobs by cordoning and draining the affected node, and **remediates** it with a GPU reset or a reboot, returning it to service with no paging required.
10
+
11
+
- 🔍 **Detect**: real-time GPU, NIC, and system-level fault detection via DCGM, syslog, and cloud provider maintenance events
12
+
- 🛡️ **Protect**: cordon and drain the affected node before a fault spreads to other jobs
13
+
- 🔧 **Remediate**: auto-repair with a targeted GPU reset or a full reboot, then bring the node back into service
14
+
- 🧩 **Extensible**: pluggable health monitors, drain strategies, and remediation actions
10
15
11
16
> [!NOTE]
12
17
> **Beta / Stable**
@@ -20,6 +25,8 @@ A single bad GPU can silently corrupt a training run or leave a node sitting idl
20
25
-[cert-manager](https://cert-manager.io/) v1.19+
21
26
- Persistent storage support for a database
22
27
28
+
The commands below get you ready for NVSentinel: the first makes sure the GPU Operator exposes DCGM as its own service, since NVSentinel queries it directly instead of going through dcgm-exporter; the second installs cert-manager, which issues the TLS certificates NVSentinel's webhooks and internal services need.
This turns on health monitoring only. NVSentinel watches your GPUs and system logs and reports faults as Kubernetes node conditions. It won't cordon a node, evict a pod or reboot a machine. Nothing here can disrupt a workload, so it's safe to run anywhere while you get a feel for what it reports. The defaults below are all you need.
49
-
50
-
> [!NOTE]
51
-
> **Host installed drivers**
52
-
> If your GPU nodes get their NVIDIA driver from the host image instead of the GPU Operator's driver DaemonSet, add `--set labeler.assumeDriverInstalled=true` to every NVSentinel install/upgrade command below.
51
+
One command works for both a first install and every later upgrade. By default it only turns on health monitoring: it won't cordon a node, evict a pod, or reboot a machine, so it's safe to run anywhere. The flags below the command are everything you can layer on later; see [Adoption](#adoption) for what each one does.
53
52
54
53
```bash
55
54
NVSENTINEL_VERSION=v1.20.0
56
55
57
-
# Drop the --set podMonitor.enabled=false flag if prometheus is installed
We recommend starting with monitoring, then enabling one step at a time as you get comfortable with how NVSentinel runs in your environment.
72
82
73
-
Once you trust what it's reporting, turn on remediation too. NVSentinel will now cordon a faulty node, drain its workloads, and fix it automatically:
83
+
### 1. Monitor
74
84
75
-
- Faults that don't need a full restart get an GPU reset, so the rest of the node's GPUs stay in service.
76
-
- Everything else gets a node reboot.
85
+
NVSentinel watches your GPUs and system logs and reports faults as Kubernetes node conditions. Nothing here can disrupt a workload, so it's the safe default to run anywhere while you get a feel for what it reports. The command above already does this; no extra flags needed.
77
86
78
-
By default, both actions run as a privileged job right on the node itself, so this works on day one with no cloud credentials to set up, on any infrastructure: on-prem, or any cloud.
87
+
### 2a. Protect: Cordon and drain
88
+
89
+
Uncomment these flags:
79
90
80
91
```bash
81
-
# Drop the --set podMonitor.enabled=false flag if prometheus is installed
To reboot nodes through your cloud provider's API instead, see the [cloud provider configuration guide](https://docs.nvidia.com/nvsentinel/configuration/janitor-provider/#cloud-provider-selection).
97
+
NVSentinel will now cordon a faulty node, so your scheduler stops placing new work on it, and drain its existing workloads. Only want to cordon, without draining yet? Drop the `nodeDrainer` line above. This is as far as NVSentinel goes unless you also enable remediation below; a cordoned (and optionally drained) node stays isolated until you (or your own tooling) repair it.
91
98
92
-
Verify it's running:
99
+
### 2b. Protect: Remediate
100
+
101
+
Remediation builds on Protect, so uncomment all of Protect's flags plus these:
93
102
94
103
```bash
95
-
kubectl get pods -n nvsentinel
104
+
--set global.faultRemediation.enabled=true \
105
+
--set global.janitor.enabled=true \
106
+
--set global.janitorProvider.enabled=true \
107
+
--set janitor-provider.csp.provider=generic
96
108
```
97
109
98
-
### Stage 3: Preflight (optional)
110
+
NVSentinel will now reboot a faulty node automatically once it's cordoned and drained. This runs as a privileged job right on the node itself, so it works on day one with no credentials to set up, regardless of whether you're running on-prem or on a CSP. To reboot through your cloud provider's API instead, see the [cloud provider configuration guide](https://docs.nvidia.com/nvsentinel/configuration/janitor-provider/#cloud-provider-selection).
99
111
100
-
Preflight is an active check that runs as an init container in the workload pod to confirm the node is ready to take that workload. A job never lands on bad hardware in the first place.
112
+
### 3. Preflight (optional)
101
113
102
-
Multi-node checks also need to know which pods belong to the same distributed job, so setup depends on the scheduler you use:
114
+
Preflight tries to keep a job from ever landing on bad hardware. It runs as an active check, an init container in the workload pod, that confirms the node is ready before the job starts.
103
115
104
-
1.**Check your scheduler.** By default, NVSentinel uses Kubernetes' native gang scheduling, covered by [values-preflight-kube.yaml](distros/kubernetes/nvsentinel/values-preflight-kube.yaml) (Note: the `GenericWorkload` and `GangScheduling` feature gates should enabled by a cluster admin). For different schedulers (KAI, Volcano, etc.), see the [gang discovery guide](https://docs.nvidia.com/nvsentinel/configuration/preflight/#gang-discovery) for configuration options.
116
+
Uncomment this flag:
105
117
106
-
2.**Enable preflight:**
107
-
108
-
```bash
109
-
# Drop the --set podMonitor.enabled=false flag if prometheus is installed
Swap in your scheduler's values file from step 1 if you're not on native Kubernetes gang scheduling.
122
+
This uses Kubernetes' native gang scheduling (the `GenericWorkload` and `GangScheduling` feature gates need to be enabled by a cluster admin). Using a different scheduler instead? See the [gang discovery guide](https://docs.nvidia.com/nvsentinel/configuration/preflight/#gang-discovery).
120
123
121
-
3.**Label the namespaces that should run it.** It's opt-in per namespace, so nothing changes until you do this:
124
+
**Label the namespaces that should run it.** It's opt-in per namespace, so nothing changes until you do this:
0 commit comments