Skip to content

Commit db52609

Browse files
dmveventsdmvevents
committed
docs(design): ADR-049 system-services-monitor scope and event taxonomy
Adds the design doc for system-services-monitor — a health-monitor for the host services behind fabric health (nvidia-fabricmanager, nvidia-persistenced, NVSwitch registration state) that sit in a layer neither gpu-health-monitor nor syslog-health-monitor observes today. Numbered 049 (next free slot on main; 030/042/043 taken). Detection mechanism, HealthEvent schema, checkName/errorCode taxonomy, and cached-state semantics are grounded in the system-services-monitor implementation (PR #1382). Signed-off-by: Anton Alexander <dmvevents@users.noreply.github.com>
1 parent 983089f commit db52609

1 file changed

Lines changed: 376 additions & 0 deletions

File tree

‎docs/designs/049-system-services-monitor-scope.md‎

Lines changed: 376 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,376 @@
1+
# ADR-049: System Services Monitor — Scope and Event Taxonomy
2+
3+
- **Status:** Proposed
4+
- **Date:** 2026-03-20 (revised 2026-07-24)
5+
- **Author:** dmvevents
6+
- **Reviewers:** XRFXLP, lalitadithya
7+
8+
## Context
9+
10+
NVSentinel has no service-level monitoring of the host daemons that keep a GPU
11+
node's fabric healthy. The existing monitors observe different layers:
12+
13+
- `gpu-health-monitor` polls DCGM via `pydcgm` for per-GPU device telemetry
14+
(PCIe, NVLink, thermal). It does not observe host process state.
15+
- `syslog-health-monitor` parses kernel logs for XID/SXID and fallen-off-bus
16+
events. It does not observe systemd unit state.
17+
18+
Neither monitor can tell whether `nvidia-fabricmanager` is running, has hung, or
19+
is crash-looping, whether `nvidia-persistenced` is up, or whether NVSwitch fabric
20+
registration is stuck "In Progress" indefinitely. On NVSwitch platforms a dead or
21+
stuck Fabric Manager silently breaks multi-GPU workloads while every device-level
22+
and log-level check still reports healthy — the failure is in the *service layer*,
23+
which nothing currently watches.
24+
25+
`system-services-monitor` fills that gap: a node-local monitor for non-DCGM,
26+
non-syslog service health. This ADR defines its scope, the boundary against
27+
`gpu-health-monitor`, and the health-event taxonomy it emits.
28+
29+
## Problem Statement
30+
31+
Service-level health signals fall into two buckets relative to what NVSentinel
32+
already collects:
33+
34+
1. **Already covered** — per-GPU device health (PCIe, NVLink, clock throttling)
35+
is owned by `gpu-health-monitor` through DCGM health watches; kernel XID/SXID
36+
events are owned by `syslog-health-monitor`.
37+
38+
2. **Not covered by any monitor** — Fabric Manager process state, FM
39+
responsiveness, NVSwitch fabric registration state, and GPU-support service
40+
lifecycle (e.g. `nvidia-persistenced`). These require active host probing
41+
(systemd queries, `nvidia-smi` fabric-state) that does not fit a passive DCGM
42+
polling loop or a syslog tail.
43+
44+
The design goal is to add the second bucket **without** re-collecting the first.
45+
Re-scraping DCGM-visible signals would create a second collection path for the
46+
same underlying data, with divergent polling intervals, thresholds, and event
47+
semantics, and would split the source of truth for GPU device health.
48+
49+
## Options Evaluated
50+
51+
### Option A: One monitor collects everything (DCGM re-scrape)
52+
53+
A single new monitor collects FM service health *and* re-derives PCIe/NVLink/clock
54+
telemetry by scraping the DCGM exporter over HTTP.
55+
56+
| Advantage | Disadvantage |
57+
|-----------|-------------|
58+
| Single component for all fabric-related checks | Re-implements DCGM polling already in `gpu-health-monitor` |
59+
| No changes to `gpu-health-monitor` | HTTP-scraped metrics are derived; `pydcgm` is authoritative |
60+
| | Divergent thresholds and event semantics |
61+
| | Redundant DCGM client code |
62+
63+
**Assessment:** Not recommended. Creates duplication and a source-of-truth conflict.
64+
65+
### Option B: Scope split — device health stays in DCGM path, services get a focused monitor
66+
67+
Keep DCGM-visible checks in `gpu-health-monitor` via `pydcgm`. Add
68+
`system-services-monitor` scoped only to non-DCGM service health.
69+
70+
| Advantage | Disadvantage |
71+
|-----------|-------------|
72+
| Aligns device checks with the existing DCGM-based monitor | Needs a clear event taxonomy across monitors |
73+
| Avoids HTTP-scraping duplication | |
74+
| Keeps the new monitor focused on what DCGM cannot see | |
75+
| Reuses the existing health-event envelope and gRPC transport | |
76+
77+
**Assessment:** Recommended.
78+
79+
### Option C: Fold service checks into `gpu-health-monitor`
80+
81+
Add FM systemd checks and fabric-state probes into `gpu-health-monitor`.
82+
83+
| Advantage | Disadvantage |
84+
|-----------|-------------|
85+
| Single binary for all node health | Active probes (`systemctl`, `nvidia-smi`) risk stalling the DCGM polling loop |
86+
| | Mixes passive telemetry with active service probing |
87+
| | `gpu-health-monitor` is not designed for host-level daemon state |
88+
89+
**Assessment:** Not recommended. Architectural mismatch.
90+
91+
## Recommendation
92+
93+
Adopt **Option B: Scope split**.
94+
95+
### Device-health signals stay in `gpu-health-monitor`
96+
97+
DCGM-visible signals — PCIe link health (`DCGM_HEALTH_WATCH_PCIE`), NVLink fabric
98+
(`DCGM_HEALTH_WATCH_NVLINK`), and thermal/clock throttling
99+
(`DCGM_HEALTH_WATCH_THERMAL`) — remain owned by `gpu-health-monitor` via the
100+
existing `pydcgm` path and are explicitly **out of scope** for
101+
`system-services-monitor`. "Future degradation-detection work" on these signals
102+
means additional policy or processing built *on top of* the existing DCGM health
103+
watches, tracked in `gpu-health-monitor`; it does **not** mean re-implementing or
104+
re-scraping those watches here. `system-services-monitor` adds no DCGM watches and
105+
opens no DCGM client. DCGM service health itself (the `nv-hostengine` process) is
106+
also excluded, because `gpu-health-monitor` already reports it as
107+
`GpuDcgmConnectivityFailure`.
108+
109+
### Service-health signals go to `system-services-monitor`
110+
111+
`system-services-monitor` is scoped to what DCGM genuinely cannot see:
112+
113+
- Whether `nvidia-fabricmanager` is running, hung, or crash-looping
114+
- Whether NVSwitch fabric registration is stuck or errored per GPU
115+
- Whether GPU-support services (e.g. `nvidia-persistenced`) are up
116+
117+
These require active systemd and `nvidia-smi` probing that does not belong in a
118+
passive DCGM polling loop.
119+
120+
## Signal Ownership
121+
122+
| Signal | Source | Owner |
123+
|--------|--------|-------|
124+
| PCIe link downtraining | `DCGM_HEALTH_WATCH_PCIE` via `pydcgm` | `gpu-health-monitor` |
125+
| NVLink degradation | `DCGM_HEALTH_WATCH_NVLINK` via `pydcgm` | `gpu-health-monitor` |
126+
| GPU clock / thermal throttling | `DCGM_HEALTH_WATCH_THERMAL` via `pydcgm` | `gpu-health-monitor` |
127+
| DCGM host-engine connectivity | `pydcgm` connect | `gpu-health-monitor` |
128+
| XID / SXID errors | kernel log parsing | `syslog-health-monitor` |
129+
| Fabric Manager up/down | `nsenter` → `systemctl show` (`ActiveState`/`SubState`) | `system-services-monitor` |
130+
| Fabric Manager flapping | `systemctl show NRestarts` + restart-window tracking | `system-services-monitor` |
131+
| Fabric Manager journal errors | `journalctl -u nvidia-fabricmanager` pattern scan | `system-services-monitor` |
132+
| NVSwitch fabric registration state | `nvidia-smi --query-gpu=fabric.state,fabric.status` | `system-services-monitor` |
133+
| GPU service lifecycle (`nvidia-persistenced`) | `nsenter` → `systemctl show` | `system-services-monitor` |
134+
135+
## Detection Mechanism
136+
137+
`system-services-monitor` runs a polling loop (`--poll-interval`, default 30s) that
138+
executes host probes via `nsenter -t 1 -m --` into PID 1's mount namespace. There
139+
is **no HTTP health endpoint** — `nvidia-fabricmanager` does not expose one — so
140+
detection is unit-state and query based:
141+
142+
- **Fabric Manager service state:** `systemctl show nvidia-fabricmanager
143+
--property=ActiveState,SubState,MainPID,ExecMainStartTimestamp`. The service is
144+
considered up when `ActiveState == "active"`. `NRestarts` is queried in a
145+
separate `systemctl show` call for compatibility with older systemd that omits
146+
it from combined property lists.
147+
- **Flap detection:** each new restart (delta in `NRestarts`) is timestamped into a
148+
per-service deque; the service is "flapping" when the number of restarts inside
149+
the window (`--flap-window`, default 600s) reaches the threshold
150+
(`--flap-threshold`, default 3).
151+
- **Journal errors:** `journalctl -u nvidia-fabricmanager --since "5 minutes ago"`
152+
is scanned for known patterns and classified as `nvswitch_error`,
153+
`initialization_failed`, `timeout`, or `general_error`.
154+
- **Fabric registration state:** `nvidia-smi --query-gpu=index,fabric.state,fabric.status
155+
--format=csv,noheader` per GPU. A GPU is healthy when `fabric.state == "Completed"`
156+
and `fabric.status == "Success"`; `fabric.state == "N/A"` (non-NVSwitch topology)
157+
is skipped.
158+
- **GPU service lifecycle:** the same `systemctl show` probe for each service in the
159+
configured list (default `nvidia-persistenced`).
160+
161+
A boot grace period (`--boot-grace-period`, default 300s) suppresses unhealthy
162+
alerts during node startup.
163+
164+
### FM readiness (Phase 1)
165+
166+
Fabric Manager is considered ready when it is **systemd `active`** *and* the
167+
per-GPU fabric-state query shows FM is responsive/progressing — i.e. not in a
168+
stuck or errored condition (`FM_REGISTRATION_STUCK` / `FM_FABRIC_ERROR`). An
169+
explicit local API/socket probe is deferred as a future refinement, not a Phase-1
170+
prerequisite, because the systemd unit exposes no health endpoint.
171+
172+
## Architecture
173+
174+
```text
175+
┌────────────────────────────────────────────────────────────────────┐
176+
│ Node │
177+
│ │
178+
│ ┌────────────────────┐ ┌────────────────────┐ ┌──────────────────┐ │
179+
│ │ gpu-health-monitor │ │ syslog-health- │ │ system-services- │ │
180+
│ │ │ │ monitor │ │ monitor │ │
181+
│ │ DCGM via pydcgm: │ │ │ │ Non-DCGM probes: │ │
182+
│ │ - PCIe / NVLink │ │ Kernel log parse: │ │ - FM systemd │ │
183+
│ │ - thermal / clock │ │ - XID / SXID │ │ - FM flap / jrnl │ │
184+
│ │ - DCGM connectivity│ │ - fallen-off-bus │ │ - fabric state │ │
185+
│ │ │ │ │ │ - GPU svc up/down│ │
186+
│ │ │ │ │ │ + state cache │ │
187+
│ └─────────┬──────────┘ └─────────┬──────────┘ └────────┬─────────┘ │
188+
│ │ │ │ │
189+
│ └──────────────────────┼──────────────────────┘ │
190+
│ ▼ │
191+
│ ┌────────────────────────────────────────────────────────────────┐ │
192+
│ │ platform-connector │ │
193+
│ │ - Aggregates HealthEvents from all node monitors │ │
194+
│ │ - gRPC over a node-local Unix domain socket (unix://…) │ │
195+
│ │ - Node health state reconciliation │ │
196+
│ └────────────────────────────────────────────────────────────────┘ │
197+
└────────────────────────────────────────────────────────────────────┘
198+
```
199+
200+
Transport from a monitor to `platform-connector` is gRPC over a **node-local Unix
201+
domain socket** (`grpc.insecure_channel("unix://<socket>")`), not a network
202+
connection, so no TLS is required on this hop. (The gRPC-TLS decision in
203+
ADR-030 applies to the janitor-controller ↔ janitor-provider path, which is a
204+
different, potentially cross-namespace connection, and is out of scope here.)
205+
206+
## Event Model
207+
208+
`system-services-monitor` reuses the existing `HealthEvent` envelope — no new
209+
transport model is required. Every event sets a coarse `checkName` for the
210+
category and carries the specific condition in `errorCode`.
211+
212+
| Condition | `checkName` | `errorCode` | `isFatal` | `recommendedAction` |
213+
|-----------|-------------|-------------|-----------|---------------------|
214+
| FM not running | `FabricManagerServiceDown` | `FABRIC_MANAGER_NOT_RUNNING` | true | `RESTART_BM` |
215+
| FM flapping | `FabricManagerServiceDown` | `FABRIC_MANAGER_FLAPPING` | true | `RESTART_BM` |
216+
| FM journal errors | `FabricManagerServiceDown` | `JOURNAL_NVSWITCH_ERROR` / `JOURNAL_INITIALIZATION_FAILED` / `JOURNAL_TIMEOUT` / `JOURNAL_GENERAL_ERROR` | true | `RESTART_BM` |
217+
| Fabric registration not started | `FabricStateUnhealthy` | `FM_NOT_STARTED` | true | `RESTART_BM` |
218+
| Fabric registration stuck | `FabricStateUnhealthy` | `FM_REGISTRATION_STUCK` | true | `RESTART_BM` |
219+
| Fabric error | `FabricStateUnhealthy` | `FM_FABRIC_ERROR` | true | `RESTART_BM` |
220+
| GPU support service down | `GpuServiceDown` | `GPU_SERVICE_NOT_RUNNING` | false | `CONTACT_SUPPORT` |
221+
222+
Multiple `errorCode` values combine on a single event (e.g. an FM-down event that
223+
is also flapping carries `[FABRIC_MANAGER_NOT_RUNNING, FABRIC_MANAGER_FLAPPING]`).
224+
225+
### HealthEvent schema
226+
227+
Events populate the shared `HealthEvent` protobuf. Fields set by this monitor:
228+
229+
| Field | Type | Value |
230+
|-------|------|-------|
231+
| `version` | int | `1` |
232+
| `agent` | string | `"system-services-monitor"` |
233+
| `componentClass` | string | `"INFRASTRUCTURE"` |
234+
| `checkName` | string | category (table above) |
235+
| `isFatal` | bool | per condition (table above) |
236+
| `isHealthy` | bool | `false` on failure, `true` on recovery |
237+
| `message` | string | human-readable detail |
238+
| `recommendedAction` | enum `RecommendedAction` | `RESTART_BM` (fatal) / `CONTACT_SUPPORT` (non-fatal) / `NONE` (healthy) |
239+
| `errorCode` | repeated string | condition codes (table above) |
240+
| `entitiesImpacted` | repeated `Entity` | `{entityType, entityValue}` — `NODE:<node>` for service checks, `GPU:<index>` for fabric-state checks |
241+
| `metadata` | map<string,string> | check-specific context |
242+
| `generatedTimestamp` | Timestamp | event time |
243+
| `nodeName` | string | node the monitor runs on |
244+
| `processingStrategy` | enum `ProcessingStrategy` | `EXECUTE_REMEDIATION` (default) or `STORE_ONLY` |
245+
246+
Example — Fabric Manager down (fatal, node-scoped):
247+
248+
```json
249+
{
250+
"version": 1,
251+
"agent": "system-services-monitor",
252+
"componentClass": "INFRASTRUCTURE",
253+
"checkName": "FabricManagerServiceDown",
254+
"isFatal": true,
255+
"isHealthy": false,
256+
"message": "Fabric Manager is failed on gpu-node-07",
257+
"recommendedAction": "RESTART_BM",
258+
"errorCode": ["FABRIC_MANAGER_NOT_RUNNING"],
259+
"entitiesImpacted": [{"entityType": "NODE", "entityValue": "gpu-node-07"}],
260+
"metadata": {"sub_state": "failed", "n_restarts": "4", "flapping": "true"},
261+
"nodeName": "gpu-node-07",
262+
"processingStrategy": "EXECUTE_REMEDIATION"
263+
}
264+
```
265+
266+
Example — fabric registration stuck (fatal, GPU-scoped):
267+
268+
```json
269+
{
270+
"version": 1,
271+
"agent": "system-services-monitor",
272+
"componentClass": "INFRASTRUCTURE",
273+
"checkName": "FabricStateUnhealthy",
274+
"isFatal": true,
275+
"isHealthy": false,
276+
"message": "FM_REGISTRATION_STUCK on gpu-node-07 GPU 7: state=In Progress, status=In Progress",
277+
"recommendedAction": "RESTART_BM",
278+
"errorCode": ["FM_REGISTRATION_STUCK"],
279+
"entitiesImpacted": [{"entityType": "GPU", "entityValue": "7"}],
280+
"metadata": {"gpu_index": "7", "fabric_state": "In Progress", "fabric_status": "In Progress", "failure_class": "FM_REGISTRATION_STUCK"},
281+
"nodeName": "gpu-node-07",
282+
"processingStrategy": "EXECUTE_REMEDIATION"
283+
}
284+
```
285+
286+
Example — GPU support service down (non-fatal, node-scoped):
287+
288+
```json
289+
{
290+
"version": 1,
291+
"agent": "system-services-monitor",
292+
"componentClass": "INFRASTRUCTURE",
293+
"checkName": "GpuServiceDown",
294+
"isFatal": false,
295+
"isHealthy": false,
296+
"message": "Service nvidia-persistenced is dead on gpu-node-07",
297+
"recommendedAction": "CONTACT_SUPPORT",
298+
"errorCode": ["GPU_SERVICE_NOT_RUNNING"],
299+
"entitiesImpacted": [{"entityType": "NODE", "entityValue": "gpu-node-07"}],
300+
"metadata": {"service_name": "nvidia-persistenced", "sub_state": "dead"},
301+
"nodeName": "gpu-node-07",
302+
"processingStrategy": "EXECUTE_REMEDIATION"
303+
}
304+
```
305+
306+
## Cached State
307+
308+
`system-services-monitor` emits **state transitions only**, not per-cycle events.
309+
The event processor keeps an in-memory `entity_cache` so an unchanged condition is
310+
not re-sent every poll interval.
311+
312+
- **Key:** `"<checkName>|<entityType>:<entityValue>"`, built from the impacted
313+
entities sorted by `(entityType, entityValue)` so key construction is
314+
order-independent (e.g. `FabricManagerServiceDown|NODE:gpu-node-07`).
315+
- **Value:** `CachedEntityState(is_fatal, is_healthy)` — the last emitted state for
316+
that key. An event is sent only when the key is unseen or either `is_fatal` or
317+
`is_healthy` differs from the cached value.
318+
- **Concurrency:** the read-decide-write sequence is guarded by a
319+
`threading.Lock`. Callbacks fire on a `ThreadPoolExecutor`, so without the lock
320+
two overlapping callbacks could observe the same stale entry and both emit while
321+
one blocks in the gRPC send. Under the lock, the new state is *reserved* into the
322+
cache immediately (before the send) so a concurrent callback skips the duplicate.
323+
- **Rollback:** the gRPC send retries with backoff (up to 5 attempts, ~26 s worst
324+
case). If it ultimately fails or raises, the reserved cache entries are rolled
325+
back (popped) so the next poll cycle re-attempts those events. The cache is only
326+
committed as the authoritative last-sent state after a successful send.
327+
- **Lifetime:** in-memory for the life of the process; there is no persistence, so
328+
the cache is empty on restart and the first post-restart cycle re-establishes
329+
current state.
330+
331+
### False-positive mitigations
332+
333+
- **Boot grace period** (default 300s): suppress unhealthy alerts during node
334+
startup.
335+
- **Flap detection**: only flag Fabric Manager as flapping when restarts within the
336+
configurable window exceed the threshold, rather than on any single restart.
337+
338+
## Scope
339+
340+
In scope for `system-services-monitor`:
341+
342+
- Fabric Manager systemd health (`ActiveState`/`SubState`)
343+
- Fabric Manager flap detection and journal-error classification
344+
- Per-GPU NVSwitch fabric-registration state
345+
- GPU support-service lifecycle (`nvidia-persistenced`)
346+
- gRPC client (Unix domain socket), state caching, structured logging, CLI
347+
348+
Out of scope (owned elsewhere):
349+
350+
- PCIe, NVLink, thermal/clock telemetry — `gpu-health-monitor` (DCGM watches)
351+
- DCGM host-engine connectivity — `gpu-health-monitor` (`GpuDcgmConnectivityFailure`)
352+
- XID / SXID kernel events — `syslog-health-monitor`
353+
354+
### Integration testing
355+
356+
- Device-level faults handled only by `gpu-health-monitor`
357+
- Kernel XID/SXID handled only by `syslog-health-monitor`
358+
- Service-level faults handled only by `system-services-monitor`
359+
- No overlap in event generation for the same underlying condition
360+
- State-cache deduplication verified (transition-only emission)
361+
- HealthEvent schema compatibility over gRPC
362+
363+
## Consequences
364+
365+
### Positive
366+
- Adds service-level monitoring that no existing monitor provides
367+
- No overlap or source-of-truth conflict with DCGM device health
368+
- Keeps `system-services-monitor` focused and small
369+
- Preserves consistent event semantics across monitors
370+
- Explicit, testable ownership boundary against `gpu-health-monitor` and
371+
`syslog-health-monitor`
372+
373+
### Negative
374+
- Adds a third node-local monitor DaemonSet to operate
375+
- Requires the event taxonomy above to stay in sync with remediation policy in
376+
`platform-connector`

0 commit comments

Comments
 (0)