demos: standalone system-services-monitor demo

demos/system-services-monitor/Dockerfile

@@ -0,0 +1,21 @@
+FROM python:3.13-slim
+
+# nsenter is in util-linux (already in slim), but ensure it's available
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    util-linux \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY config.py metrics.py monitor.py ./
+COPY checks/ ./checks/
+
+# nsenter requires root to enter host namespaces.
+# The DaemonSet securityContext controls the actual privilege level.
+
+EXPOSE 9101
+
+ENTRYPOINT ["python", "monitor.py"]

demos/system-services-monitor/README.md

@@ -0,0 +1,76 @@
+# System Service Monitor — Standalone Demo
+
+A standalone DaemonSet companion to NVSentinel that catches GPU infrastructure failures invisible to DCGM-based monitoring.
+
+**Related issue:** [#883 - NVSentinel not detecting fabric health on H100s](https://github.com/NVIDIA/NVSentinel/issues/883)
+
+## Problem
+
+NVIDIA Fabric Manager can fail and stay broken for weeks undetected. NVSentinel's existing monitors (DCGM-based, syslog-based) miss it because individual GPUs appear healthy to DCGM even when Fabric Manager is down. This tool fills the gap with service-level health checks.
+
+**Requirements:** Kubernetes cluster with GPU nodes, Prometheus Operator
+
+## What It Monitors
+
+| # | Check | What It Catches | Method |
+|---|-------|-----------------|--------|
+| 1 | **Fabric Manager Service** | FM not running, flapping, error state | `nsenter` + `systemctl` |
+| 2 | **Critical GPU Services** | persistenced dead | `nsenter` + `systemctl` |
+| 3 | **Per-GPU Fabric State** | FM_NOT_STARTED, FM_REGISTRATION_STUCK, FM_FABRIC_ERROR | `nsenter` + `nvidia-smi` |
+
+> **CUDA validation is not part of this monitor.** Polling CUDA context/memory tests from a long-running daemon contends for GPU memory with active workloads (see the [#891 review](https://github.com/NVIDIA/NVSentinel/pull/891)). The supported form is a preflight init-container that runs once before workloads schedule — see [`preflight-checks/cuda-validation/`](../../preflight-checks/cuda-validation/) (#1384).
+
+## Quick Start
+
+```bash
+# Build
+docker build -t system-services-monitor:latest .
+
+# Deploy (assumes nvsentinel namespace exists)
+kubectl apply -f k8s/rbac.yaml
+kubectl apply -f k8s/configmap.yaml
+kubectl apply -f k8s/daemonset.yaml
+kubectl apply -f k8s/servicemonitor.yaml
+
+# Verify
+kubectl get ds -n nvsentinel system-services-monitor
+
+# Port-forward to a specific node's pod
+NODE=<node-name>
+POD=$(kubectl get pod -n nvsentinel -o wide --field-selector spec.nodeName=${NODE} \
+  -l app=system-services-monitor -o jsonpath='{.items[0].metadata.name}')
+kubectl port-forward -n nvsentinel pod/${POD} 9101:9101
+curl -s localhost:9101/metrics | grep fabric_manager_up
+```
+
+## Metrics
+
+Exposed on port 9101. Key metrics:
+
+| Metric | Description |
+|--------|-------------|
+| `fabric_manager_up` | Fabric Manager running (1/0) |
+| `gpu_node_health_up` | Overall node health (1/0) |
+| `nvidia_service_up` | Per-service status |
+| `fabric_state_healthy` | Per-GPU fabric state (1/0) |
+
+## Alert Rules
+
+The ServiceMonitor includes PrometheusRule with alerts:
+- `FabricManagerDown` (critical, 5m)
+- `FabricManagerFlapping` (warning, 5m)
+- `FabricStateUnhealthy` (critical, 5m) -- per-GPU fabric orchestration failure
+- `GPUServiceDown` (critical, 3m)
+
+## Validated On
+
+- 2x P4d.24xlarge (8x A100-SXM4-40GB each) -- Amazon Linux 2023, EKS 1.32
+- All check categories produce correct metrics
+
+## Configuration
+
+All settings via ConfigMap environment variables. See `k8s/configmap.yaml`.
+
+## Relationship to NVSentinel
+
+This is a **standalone companion tool** that exposes Prometheus metrics and alerts. It does not integrate with NVSentinel's gRPC event pipeline or remediation workflow. See the native `health-monitors/system-services-monitor/` for an integrated version that emits HealthEvents to platform-connector.

demos/system-services-monitor/checks/__init__.py

@@ -0,0 +1,15 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Health check modules for GPU Node Health Validator."""

demos/system-services-monitor/checks/service_check.py

@@ -0,0 +1,229 @@
+"""Checks 1 & 2: Systemd service health for Fabric Manager and GPU services.
+
+Uses nsenter to inspect host systemd services from within a container.
+Includes flap detection (rapid restart cycling) and journal error parsing.
+"""
+
+import logging
+import subprocess
+import time
+from collections import deque
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import Dict, List, Optional
+
+logger = logging.getLogger(__name__)
+
+
+class ErrorCategory(Enum):
+    NVSWITCH_ERROR = "nvswitch_error"
+    INITIALIZATION_FAILED = "initialization_failed"
+    TIMEOUT = "timeout"
+    GENERAL_ERROR = "general_error"
+
+
+# Journal patterns that indicate specific failure modes
+_ERROR_PATTERNS = {
+    ErrorCategory.NVSWITCH_ERROR: [
+        "nvswitch",
+        "NVSwitch",
+        "fabric error",
+    ],
+    ErrorCategory.INITIALIZATION_FAILED: [
+        "initialization failed",
+        "failed to initialize",
+        "Init Failed",
+        "unable to start",
+    ],
+    ErrorCategory.TIMEOUT: [
+        "timed out",
+        "timeout",
+        "deadline exceeded",
+    ],
+}
+
+
+@dataclass
+class ServiceStatus:
+    """Result of a single systemd service check."""
+    name: str
+    active: bool               # True if ActiveState == "active"
+    sub_state: str = ""        # e.g. "running", "dead", "failed"
+    main_pid: int = 0
+    n_restarts: int = 0
+    start_timestamp: str = ""
+    error: Optional[str] = None  # non-None if the check itself failed
+
+
+@dataclass
+class FabricManagerStatus(ServiceStatus):
+    """Extended status for Fabric Manager with journal analysis."""
+    journal_errors: List[ErrorCategory] = field(default_factory=list)
+    flapping: bool = False
+
+
+class ServiceChecker:
+    """Checks host systemd services via nsenter."""
+
+    def __init__(self, flap_window: int = 600, flap_threshold: int = 3):
+        self._flap_window = flap_window
+        self._flap_threshold = flap_threshold
+        # Track restart timestamps per service for flap detection
+        self._restart_history: Dict[str, deque] = {}
+        # Track last-seen restart count to detect new restarts
+        self._last_restart_count: Dict[str, int] = {}
+
+    def _run_host_cmd(self, cmd: List[str], timeout: int = 10) -> subprocess.CompletedProcess:
+        """Run a command on the host via nsenter into PID 1's mount namespace."""
+        full_cmd = ["nsenter", "-t", "1", "-m", "--"] + cmd
+        return subprocess.run(
+            full_cmd,
+            capture_output=True,
+            text=True,
+            timeout=timeout,
+        )
+
+    def check_service(self, service_name: str) -> ServiceStatus:
+        """Check a single systemd service via nsenter.
+
+        Parses systemctl show output for ActiveState, SubState, MainPID,
+        and ExecMainStartTimestamp. NRestarts is queried separately since
+        older systemd versions don't support it.
+        """
+        try:
+            result = self._run_host_cmd([
+                "systemctl", "show", service_name,
+                "--property=ActiveState,SubState,MainPID,ExecMainStartTimestamp",
+            ])
+
+            if result.returncode != 0 and not result.stdout.strip():
+                return ServiceStatus(
+                    name=service_name,
+                    active=False,
+                    error=f"systemctl show failed: {result.stderr.strip()}",
+                )
+
+            props = {}
+            for line in result.stdout.strip().splitlines():
+                if "=" in line:
+                    key, _, value = line.partition("=")
+                    props[key.strip()] = value.strip()
+
+            active_state = props.get("ActiveState", "unknown")
+
+            # NRestarts isn't available on older systemd; query separately
+            n_restarts = self._get_restart_count(service_name)
+
+            # Flap detection
+            self._update_flap_tracking(service_name, n_restarts)
+
+            return ServiceStatus(
+                name=service_name,
+                active=(active_state == "active"),
+                sub_state=props.get("SubState", ""),
+                main_pid=int(props.get("MainPID", "0")),
+                n_restarts=n_restarts,
+                start_timestamp=props.get("ExecMainStartTimestamp", ""),
+            )
+
+        except subprocess.TimeoutExpired:
+            return ServiceStatus(
+                name=service_name,
+                active=False,
+                error="systemctl show timed out",
+            )
+        except Exception as e:
+            return ServiceStatus(
+                name=service_name,
+                active=False,
+                error=str(e),
+            )
+
+    def _get_restart_count(self, service_name: str) -> int:
+        """Get NRestarts from systemd, returning 0 if unsupported."""
+        try:
+            result = self._run_host_cmd([
+                "systemctl", "show", service_name, "--property=NRestarts",
+            ])
+            if result.returncode == 0 and result.stdout.strip():
+                _, _, val = result.stdout.strip().partition("=")
+                return int(val)
+        except Exception:
+            pass
+        return 0
+
+    def _update_flap_tracking(self, service_name: str, current_restarts: int) -> None:
+        """Track restart events for flap detection."""
+        if service_name not in self._restart_history:
+            self._restart_history[service_name] = deque()
+            self._last_restart_count[service_name] = current_restarts
+            return
+
+        last_count = self._last_restart_count[service_name]
+        if current_restarts > last_count:
+            # New restarts detected — record timestamp for each
+            now = time.monotonic()
+            for _ in range(current_restarts - last_count):
+                self._restart_history[service_name].append(now)
+            self._last_restart_count[service_name] = current_restarts
+
+        # Prune entries outside the flap window
+        cutoff = time.monotonic() - self._flap_window
+        history = self._restart_history[service_name]
+        while history and history[0] < cutoff:
+            history.popleft()
+
+    def is_flapping(self, service_name: str) -> bool:
+        """Return True if the service has restarted too many times within the window."""
+        history = self._restart_history.get(service_name, deque())
+        return len(history) >= self._flap_threshold
+
+    def check_fabric_manager(self) -> FabricManagerStatus:
+        """Check Fabric Manager with journal error analysis."""
+        base = self.check_service("nvidia-fabricmanager")
+
+        journal_errors = self._parse_journal_errors("nvidia-fabricmanager")
+        flapping = self.is_flapping("nvidia-fabricmanager")
+
+        return FabricManagerStatus(
+            name=base.name,
+            active=base.active,
+            sub_state=base.sub_state,
+            main_pid=base.main_pid,
+            n_restarts=base.n_restarts,
+            start_timestamp=base.start_timestamp,
+            error=base.error,
+            journal_errors=journal_errors,
+            flapping=flapping,
+        )
+
+    def _parse_journal_errors(self, service_name: str) -> List[ErrorCategory]:
+        """Scan recent journal entries for known error patterns."""
+        try:
+            result = self._run_host_cmd([
+                "journalctl", "-u", service_name,
+                "--since", "5 minutes ago",
+                "--no-pager", "-q",
+            ], timeout=15)
+
+            if result.returncode != 0 or not result.stdout.strip():
+                return []
+
+            found: List[ErrorCategory] = []
+            text = result.stdout.lower()
+            for category, patterns in _ERROR_PATTERNS.items():
+                if any(p.lower() in text for p in patterns):
+                    found.append(category)
+
+            return found
+
+        except (subprocess.TimeoutExpired, Exception) as e:
+            logger.warning("Journal parsing failed for %s: %s", service_name, e)
+            return []
+
+    def check_all_gpu_services(self, service_names: List[str]) -> Dict[str, ServiceStatus]:
+        """Check all configured GPU services."""
+        results = {}
+        for name in service_names:
+            results[name] = self.check_service(name)
+        return results