Skip to content

Commit 2083bfa

Browse files
dmveventsdmvevents
committed
chart: system-services-monitor subchart + parent registration
Lands the Helm chart for system-services-monitor (#891 split, 4 of 5). Subchart under distros/kubernetes/nvsentinel/charts/ + values.yaml + Chart.yaml dependency registration in the parent chart. The chart is aligned to the actual runtime contract of the app on the implementation branch (cli.py / metrics.py), not an assumed one: DaemonSet - The app's entrypoint is a Click CLI whose --platform-connector-socket option is required=True with no env fallback. The container now passes it (plus --port/--poll-interval/--boot-grace-period/--flap-window/ --flap-threshold/--enable-fabric-check/--processing-strategy) via args:, and mounts the platform-connector Unix socket (hostPath /var/run/nvsentinel at /var/run, mirroring the slurm/nic/csp siblings). The app prepends unix:// itself, so the flag value is a bare path. - Dropped the envFrom configMapRef and the unused /var/run/dbus mount -- the app reads host systemd state via nsenter into PID 1, not dbus. - Keeps NODE_NAME (fieldRef) and LOG_LEVEL, the only env the app reads. Metrics / alerts - metricsPort now binds global.metricsPort (2112), matching siblings and the --port flag the app actually honors. - PrometheusRule alerts only on metrics the monitor exports: fabric_manager_up, fabric_state_healthy, nvidia_service_up, and fabric_manager_restarts_total (added in #1382). The flapping alert fires on increase(fabric_manager_restarts_total[10m]) > 3. Removed the CUDAValidationFailed alert -- cuda validation is an exit-code-only init container (#1384), no cuda_validation_passed metric. - Alert names follow the ADR-049 check taxonomy: FabricManagerServiceDown, FabricStateUnhealthy, GpuServiceDown. Config - Deleted the ConfigMap: its keys were either dead or are real CLI flags, now templated into args: from values.yaml. LOG_LEVEL is a plain env var. - ServiceMonitor + PrometheusRule default enabled: false (no health-monitor sibling ships them enabled) and ServiceMonitor's release label is now driven by .Values.serviceMonitor.labels (empty default) instead of a hardcoded release: prometheus. Mirrors the sibling pattern (nic-health-monitor) for .Values.global references; the parent chart supplies globals, so validate by rendering the parent chart (helm template distros/kubernetes/nvsentinel --set global.systemServicesMonitor.enabled=true), not standalone lint. Signed-off-by: Anton Alexander <dmvevents@users.noreply.github.com>
1 parent 983089f commit 2083bfa

13 files changed

Lines changed: 511 additions & 0 deletions

File tree

‎distros/kubernetes/nvsentinel/Chart.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,3 +79,6 @@ dependencies:
7979
- name: nic-health-monitor
8080
version: "0.1.0"
8181
condition: global.nicHealthMonitor.enabled
82+
- name: system-services-monitor
83+
version: "0.1.0"
84+
condition: global.systemServicesMonitor.enabled

‎distros/kubernetes/nvsentinel/charts/system-services-monitor/.helmignore‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Patterns to ignore when building packages.
2+
# This supports shell glob matching, relative path matching, and
3+
# negation (prefixed with !). Only one pattern per line.
4+
.DS_Store
5+
# Common VCS dirs
6+
.git/
7+
.gitignore
8+
.bzr/
9+
.bzrignore
10+
.hg/
11+
.hgignore
12+
.svn/
13+
# Common backup files
14+
*.swp
15+
*.bak
16+
*.tmp
17+
*.orig
18+
*~
19+
# Various IDEs
20+
.project
21+
.idea/
22+
*.tmproj
23+
.vscode/

‎distros/kubernetes/nvsentinel/charts/system-services-monitor/Chart.yaml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
apiVersion: v2
16+
name: system-services-monitor
17+
description: A Helm chart for the System Services Monitor (Fabric Manager + GPU systemd services)
18+
type: application
19+
version: 0.1.0
20+
appVersion: "1.16.0"

‎distros/kubernetes/nvsentinel/charts/system-services-monitor/templates/_helpers.tpl‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
{{/*
2+
Expand the name of the chart.
3+
*/}}
4+
{{- define "system-services-monitor.name" -}}
5+
{{- .Chart.Name | trunc 63 | trimSuffix "-" }}
6+
{{- end }}
7+
8+
{{/*
9+
Create a default fully qualified app name.
10+
*/}}
11+
{{- define "system-services-monitor.fullname" -}}
12+
{{- "system-services-monitor" | trunc 63 | trimSuffix "-" }}
13+
{{- end }}
14+
15+
{{/*
16+
Create chart name and version as used by the chart label.
17+
*/}}
18+
{{- define "system-services-monitor.chart" -}}
19+
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
20+
{{- end }}
21+
22+
{{/*
23+
Common labels
24+
*/}}
25+
{{- define "system-services-monitor.labels" -}}
26+
helm.sh/chart: {{ include "system-services-monitor.chart" . }}
27+
{{ include "system-services-monitor.selectorLabels" . }}
28+
{{- if .Chart.AppVersion }}
29+
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
30+
{{- end }}
31+
app.kubernetes.io/managed-by: {{ .Release.Service }}
32+
app.kubernetes.io/component: metrics
33+
{{- end }}
34+
35+
{{/*
36+
Selector labels
37+
*/}}
38+
{{- define "system-services-monitor.selectorLabels" -}}
39+
app.kubernetes.io/name: {{ include "system-services-monitor.name" . }}
40+
app.kubernetes.io/instance: {{ .Release.Name }}
41+
{{- end }}

‎distros/kubernetes/nvsentinel/charts/system-services-monitor/templates/clusterrole.yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
apiVersion: rbac.authorization.k8s.io/v1
15+
kind: ClusterRole
16+
metadata:
17+
name: {{ include "system-services-monitor.fullname" . }}
18+
labels:
19+
{{- include "system-services-monitor.labels" . | nindent 4 }}
20+
rules:
21+
- apiGroups: [""]
22+
resources: ["nodes"]
23+
verbs: ["get", "list"]

‎distros/kubernetes/nvsentinel/charts/system-services-monitor/templates/clusterrolebinding.yaml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
apiVersion: rbac.authorization.k8s.io/v1
15+
kind: ClusterRoleBinding
16+
metadata:
17+
name: {{ include "system-services-monitor.fullname" . }}
18+
labels:
19+
{{- include "system-services-monitor.labels" . | nindent 4 }}
20+
roleRef:
21+
apiGroup: rbac.authorization.k8s.io
22+
kind: ClusterRole
23+
name: {{ include "system-services-monitor.fullname" . }}
24+
subjects:
25+
- kind: ServiceAccount
26+
name: {{ include "system-services-monitor.fullname" . }}
27+
namespace: {{ .Release.Namespace }}

‎distros/kubernetes/nvsentinel/charts/system-services-monitor/templates/daemonset.yaml‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
apiVersion: apps/v1
15+
kind: DaemonSet
16+
metadata:
17+
name: {{ include "system-services-monitor.fullname" . }}
18+
labels:
19+
{{- include "system-services-monitor.labels" . | nindent 4 }}
20+
spec:
21+
updateStrategy:
22+
type: RollingUpdate
23+
rollingUpdate:
24+
maxUnavailable: 5%
25+
selector:
26+
matchLabels:
27+
{{- include "system-services-monitor.selectorLabels" . | nindent 6 }}
28+
template:
29+
metadata:
30+
{{- with .Values.podAnnotations }}
31+
annotations:
32+
{{- toYaml . | nindent 8 }}
33+
{{- end }}
34+
labels:
35+
{{- include "system-services-monitor.selectorLabels" . | nindent 8 }}
36+
app.kubernetes.io/component: metrics
37+
spec:
38+
serviceAccountName: {{ include "system-services-monitor.fullname" . }}
39+
# hostPID is required so the monitor can use nsenter against the host
40+
# PID 1 mount namespace to read systemd state and run nvidia-smi.
41+
hostPID: true
42+
{{- with .Values.global.imagePullSecrets }}
43+
imagePullSecrets:
44+
{{- toYaml . | nindent 8 }}
45+
{{- end }}
46+
containers:
47+
- name: monitor
48+
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default ((.Values.global).image).tag | default .Chart.AppVersion }}"
49+
imagePullPolicy: {{ .Values.image.pullPolicy }}
50+
# ENTRYPOINT is `system_services_monitor` (Click CLI). The socket flag
51+
# is required=True with no env fallback; the app prepends `unix://` to
52+
# the value, so pass a bare filesystem path here.
53+
args:
54+
- "--platform-connector-socket"
55+
- "{{ ((.Values.global).socketPath) | default "/var/run/nvsentinel.sock" }}"
56+
- "--port"
57+
- "{{ .Values.global.metricsPort }}"
58+
- "--poll-interval"
59+
- "{{ .Values.pollInterval }}"
60+
- "--boot-grace-period"
61+
- "{{ .Values.bootGracePeriod }}"
62+
- "--flap-window"
63+
- "{{ .Values.flapWindow }}"
64+
- "--flap-threshold"
65+
- "{{ .Values.flapThreshold }}"
66+
{{- if .Values.enableFabricCheck }}
67+
- "--enable-fabric-check"
68+
{{- else }}
69+
- "--disable-fabric-check"
70+
{{- end }}
71+
- "--processing-strategy"
72+
- "{{ .Values.processingStrategy }}"
73+
env:
74+
- name: NODE_NAME
75+
valueFrom:
76+
fieldRef:
77+
fieldPath: spec.nodeName
78+
- name: LOG_LEVEL
79+
value: "{{ .Values.logLevel }}"
80+
ports:
81+
- name: metrics
82+
containerPort: {{ .Values.global.metricsPort }}
83+
protocol: TCP
84+
livenessProbe:
85+
httpGet:
86+
path: /metrics
87+
port: metrics
88+
initialDelaySeconds: 10
89+
periodSeconds: 30
90+
resources:
91+
{{- toYaml .Values.resources | nindent 12 }}
92+
securityContext:
93+
# privileged is required for nsenter into the host PID 1 mount
94+
# namespace to inspect host systemd services and run nvidia-smi.
95+
privileged: true
96+
volumeMounts:
97+
# platform-connector Unix domain socket (gRPC target). The host
98+
# directory /var/run/nvsentinel is mounted at /var/run so the socket
99+
# resolves in-container at /var/run/nvsentinel.sock.
100+
- name: var-run-vol
101+
mountPath: /var/run/
102+
volumes:
103+
- name: var-run-vol
104+
hostPath:
105+
path: /var/run/nvsentinel
106+
type: DirectoryOrCreate
107+
nodeSelector:
108+
{{- with (.Values.global.nodeSelector | default .Values.nodeSelector) }}
109+
{{- toYaml . | nindent 8 }}
110+
{{- end }}
111+
{{- with (.Values.global.affinity | default .Values.affinity) }}
112+
affinity:
113+
{{- toYaml . | nindent 8 }}
114+
{{- end }}
115+
{{- with (.Values.global.tolerations | default .Values.tolerations) }}
116+
tolerations:
117+
{{- toYaml . | nindent 8 }}
118+
{{- end }}

‎distros/kubernetes/nvsentinel/charts/system-services-monitor/templates/prometheusrule.yaml‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
{{- if .Values.prometheusRule.enabled }}
15+
apiVersion: monitoring.coreos.com/v1
16+
kind: PrometheusRule
17+
metadata:
18+
name: {{ include "system-services-monitor.fullname" . }}
19+
labels:
20+
{{- include "system-services-monitor.labels" . | nindent 4 }}
21+
{{- with .Values.prometheusRule.labels }}
22+
{{- toYaml . | nindent 4 }}
23+
{{- end }}
24+
spec:
25+
groups:
26+
- name: gpu-node-health
27+
rules:
28+
# Critical: Fabric Manager down for 5 minutes.
29+
- alert: FabricManagerServiceDown
30+
expr: fabric_manager_up == 0
31+
for: 5m
32+
labels:
33+
severity: critical
34+
component: system-services-monitor
35+
annotations:
36+
summary: "Fabric Manager down on {{`{{ $labels.node }}`}}"
37+
description: >-
38+
Fabric Manager has been down for more than 5 minutes on node
39+
{{`{{ $labels.node }}`}}. NVLink/NVSwitch fabric is non-functional.
40+
Multi-GPU workloads will fail silently or hang.
41+
runbook: "ssh to node and run: sudo systemctl restart nvidia-fabricmanager"
42+
43+
# Warning: Fabric Manager flapping (restarting repeatedly).
44+
# Backed by the fabric_manager_restarts_total counter the app
45+
# increments from the systemd NRestarts delta observed between poll
46+
# cycles: more than 3 restarts within 10 minutes flags flapping.
47+
- alert: FabricManagerFlapping
48+
expr: increase(fabric_manager_restarts_total[10m]) > 3
49+
for: 5m
50+
labels:
51+
severity: warning
52+
component: system-services-monitor
53+
annotations:
54+
summary: "Fabric Manager flapping on {{`{{ $labels.node }}`}}"
55+
description: >-
56+
Fabric Manager is restarting repeatedly on {{`{{ $labels.node }}`}}.
57+
Check journalctl -u nvidia-fabricmanager for root cause.
58+
59+
# Critical: Per-GPU fabric state unhealthy.
60+
- alert: FabricStateUnhealthy
61+
expr: fabric_state_healthy == 0
62+
for: 5m
63+
labels:
64+
severity: critical
65+
component: system-services-monitor
66+
annotations:
67+
summary: "Fabric state unhealthy on {{`{{ $labels.node }}`}} GPU {{`{{ $labels.gpu_index }}`}}"
68+
description: >-
69+
Fabric Manager orchestration is in a bad state for GPU
70+
{{`{{ $labels.gpu_index }}`}} on {{`{{ $labels.node }}`}}.
71+
Check nvidia-smi -q for fabric.state and fabric.status.
72+
73+
# Warning: GPU systemd service down (non-fatal per PR classification).
74+
- alert: GpuServiceDown
75+
expr: nvidia_service_up == 0
76+
for: 3m
77+
labels:
78+
severity: warning
79+
component: system-services-monitor
80+
annotations:
81+
summary: "{{`{{ $labels.service_name }}`}} down on {{`{{ $labels.node }}`}}"
82+
description: >-
83+
NVIDIA service {{`{{ $labels.service_name }}`}} has been down for more
84+
than 3 minutes on {{`{{ $labels.node }}`}}.
85+
{{- end }}

‎distros/kubernetes/nvsentinel/charts/system-services-monitor/templates/service.yaml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
apiVersion: v1
15+
kind: Service
16+
metadata:
17+
name: {{ include "system-services-monitor.fullname" . }}
18+
labels:
19+
{{- include "system-services-monitor.labels" . | nindent 4 }}
20+
spec:
21+
# Headless service — Prometheus scrapes each pod individually.
22+
clusterIP: None
23+
selector:
24+
{{- include "system-services-monitor.selectorLabels" . | nindent 4 }}
25+
ports:
26+
- name: metrics
27+
port: {{ .Values.global.metricsPort }}
28+
targetPort: {{ .Values.global.metricsPort }}
29+
protocol: TCP

0 commit comments

Comments
 (0)