feat(chart): add priorityClassName support to all NVSentinel workloads

NVSentinel pods run at priority 0 by default, below any workload that sets a
priority class. On a CPU-saturated cluster the scheduler therefore leaves them
Pending instead of preempting a lower-value pod, and a node ends up with no health
monitor while the DaemonSet still reports healthy, because a DaemonSet whose pods
cannot be placed is not unhealthy. The chart had no way to set a priority class, so
the only workarounds were patching rendered manifests or a mutating webhook.

Adds two globals, mirroring the existing nodeSelector/tolerations split for the same
reason: the node agents and the control-plane components run on different node
classes and want different priorities.

  global.priorityClassName        node-level agents (DaemonSets, platform-connectors)
  global.systemPriorityClassName  control-plane components (Deployments)

Every component also takes a local priorityClassName, and each template uses the
established (global | default local) idiom already used for tolerations, so the
global wins when both are set.

Both globals default to empty, which omits the field from the pod spec entirely, so
a default render is unchanged and no existing deployment is affected.

Covers 21 pod specs: the platform-connectors DaemonSet, both gpu-health-monitor DCGM
variants, both syslog-health-monitor variants, metadata-collector, nic-health-monitor,
the preflight image cache, and the labeler, health-events-analyzer, fault-quarantine,
node-drainer, fault-remediation, janitor, janitor-provider, csp-health-monitor,
kubernetes-object-monitor, event-exporter, incluster-file-server, slurm-drain-monitor,
lifecycle-manager and preflight Deployments.

Short-lived Jobs and lifecycle hooks are left alone: they have their own local
scheduling conventions rather than reading the globals, and pod priority matters far
less for work that is not long-running. The vendored postgresql and mongodb subcharts
already support priorityClassName upstream.

Tests cover both halves of the split, including negative cases asserting that the
node-agent global does not reach the control-plane components and vice versa, so the
two cannot silently collapse into one.

Signed-off-by: Lonnie Friedman <lfriedman@together.ai>

Author: lfriedman-netllama

distros/kubernetes/nvsentinel/charts/csp-health-monitor/templates/deployment.yaml

@@ -268,3 +268,6 @@ spec:
       tolerations:
         {{- toYaml . | nindent 8 }}
       {{- end }}
+      {{- with (((.Values.global).systemPriorityClassName) | default .Values.priorityClassName) }}
+      priorityClassName: {{ . | quote }}
+      {{- end }}

distros/kubernetes/nvsentinel/charts/csp-health-monitor/values.yaml

@@ -45,6 +45,9 @@ quarantineTriggerEngine:
 # Scheduling configuration
 nodeSelector: {}
 affinity: {}
+# Pod priority for this component. Overridden by the matching global when that is set.
+priorityClassName: ""
+
 tolerations: []
 
 podAnnotations: {}

distros/kubernetes/nvsentinel/charts/event-exporter/templates/deployment.yaml

@@ -190,3 +190,6 @@ spec:
       tolerations:
         {{- toYaml . | nindent 8 }}
       {{- end }}
+      {{- with (.Values.global.systemPriorityClassName | default .Values.priorityClassName) }}
+      priorityClassName: {{ . | quote }}
+      {{- end }}

distros/kubernetes/nvsentinel/charts/event-exporter/values.yaml

@@ -92,3 +92,5 @@ exporter:
     maxBackoff: "5m"
     backoffMultiplier: 2.0
 
+# Pod priority for this component. Overridden by the matching global when that is set.
+priorityClassName: ""

distros/kubernetes/nvsentinel/charts/fault-quarantine/templates/deployment.yaml

@@ -210,3 +210,6 @@ spec:
       tolerations:
         {{- toYaml . | nindent 8 }}
       {{- end }}
+      {{- with (((.Values.global).systemPriorityClassName) | default .Values.priorityClassName) }}
+      priorityClassName: {{ . | quote }}
+      {{- end }}

distros/kubernetes/nvsentinel/charts/fault-quarantine/values.yaml

@@ -30,6 +30,9 @@ resources:
 # Scheduling configuration
 nodeSelector: {}
 affinity: {}
+# Pod priority for this component. Overridden by the matching global when that is set.
+priorityClassName: ""
+
 tolerations: []
 
 podAnnotations: {}

distros/kubernetes/nvsentinel/charts/fault-remediation/templates/deployment.yaml

@@ -203,3 +203,6 @@ spec:
       tolerations:
         {{- toYaml . | nindent 8 }}
       {{- end }}
+      {{- with (((.Values.global).systemPriorityClassName) | default .Values.priorityClassName) }}
+      priorityClassName: {{ . | quote }}
+      {{- end }}

distros/kubernetes/nvsentinel/charts/fault-remediation/values.yaml

@@ -40,6 +40,9 @@ affinity: {}
 podAnnotations: {}
 
 # Special tolerations for fault remediation - allow running on nodes with any taints for log collection
+# Pod priority for this component. Overridden by the matching global when that is set.
+priorityClassName: ""
+
 tolerations:
   - operator: "Exists"
 

distros/kubernetes/nvsentinel/charts/gpu-health-monitor/templates/daemonset-dcgm-3.x.yaml

@@ -156,3 +156,6 @@ spec:
       tolerations:
         {{- toYaml . | nindent 8 }}
       {{- end }}
+      {{- with (.Values.global.priorityClassName | default .Values.priorityClassName) }}
+      priorityClassName: {{ . | quote }}
+      {{- end }}

distros/kubernetes/nvsentinel/charts/gpu-health-monitor/templates/daemonset-dcgm-4.x.yaml

@@ -156,3 +156,6 @@ spec:
       tolerations:
         {{- toYaml . | nindent 8 }}
       {{- end }}
+      {{- with (.Values.global.priorityClassName | default .Values.priorityClassName) }}
+      priorityClassName: {{ . | quote }}
+      {{- end }}