Skip to content

feat(mcp-server): merge donation of k8s-gpu-mcp-server - #1333

Open
ArangoGutierrez wants to merge 27 commits into
mainfrom
feat/mcp-server-merge
Open

feat(mcp-server): merge donation of k8s-gpu-mcp-server#1333
ArangoGutierrez wants to merge 27 commits into
mainfrom
feat/mcp-server-merge

Conversation

@ArangoGutierrez

@ArangoGutierrez ArangoGutierrez commented May 25, 2026

Copy link
Copy Markdown
Contributor

feat(mcp-server): merge donation of k8s-gpu-mcp-server

Summary

Adds the mcp-server component to NVSentinel — a Model Context Protocol (MCP) server that exposes NVSentinel's read-only health surface (HealthEventStore + Kubernetes API) as MCP tools, ported from the upstream donation ArangoGutierrez/k8s-gpu-mcp-server at SHA 80ac33d89ede70aa3f967088f8716d94b8e692e4.

The new component is gated off by default (global.mcpServer.enabled=false); existing deployments are unaffected.

Tools (10 total)

9 Working tools backed by store-client/pkg/datastore.HealthEventStore and (where needed) the Kubernetes API:

Tool Source
gpu_inventory store
gpu_health store
describe_gpu_node store + K8s API
pod_gpu_allocation K8s API
pod_failure store + K8s API
explain_failure store + ported pattern matcher
get_incident_report store
analyze_xid store
get_gpu_timeline store

1 Stub toolget_nvlink_topology. NVSentinel does not yet persist per-node NVLink topology in its store. The stub returns the NVSENTINEL_DATA_GAP envelope with a needed_monitor_extension field explaining the gap inline (see mcp-server/AUDIT.md § 6.1).

Architectural decisions (carried forward from donation audit)

  • Store-only surface. pkg/monitors/ from the donor was dropped — NVSentinel has no monitor-side read gRPC; only the PlatformConnector.HealthEventOccurredV1 write ingress exists.
  • Module path: github.com/nvidia/nvsentinel/mcp-server (lowercase, sibling-aligned).
  • Mirror event-exporter structure. commons/pkg/{logger,server,metrics,tracing} everywhere; donor's klog dependency dropped.
  • Helm chart at distros/kubernetes/nvsentinel/charts/mcp-server/ (not mcp-server/deploy/helm/).
  • Bearer auth on /mcp, not /metrics. Renamed donor's SetMetricsAuthTokenSetAuthToken.
  • Config.AuthToken tagged json:"-" to silence gosec G117 and prevent accidental leakage via marshaling.
  • main.go reads MCP_AUTH_TOKEN env var as a fallback when --auth-token is empty. The Helm chart injects this env var from authToken.secretName; the fallback closes the chart→binary path so bearer auth on /mcp actually engages when the operator wires up a Secret. End-to-end chain verified by helm template rendering of the deployment plus unit tests covering the four flag/env combinations.
  • pkg/incidents pattern matcher ported as a tool helper; runs at query time (no pre-computed incident object storage).

See mcp-server/AUDIT.md for the full audit of NVSentinel's read surface vs the donor's expectations.

Verification

Run at HEAD f2806679 from worktree .worktrees/mcp-server-merge.

Gate Command Result
Build make -C mcp-server build PASS
Unit tests (race) make -C mcp-server test PASS — 90 tests, ~58% statement coverage
Lint make -C mcp-server lint (golangci-lint v2.12.2) PASS — 0 issues
lint-test (vet + lint + test + cover) make -C mcp-server lint-test PASS
govulncheck govulncheck ./mcp-server/... PASS — No vulnerabilities found
Image scan trivy image --severity CRITICAL,HIGH on ko build output (chainguard/static base) PASS — 0 findings
Cluster smoke — CPU (helm install) holodeck single-node t3.xlarge ubuntu-24.04 + helm install with global.mcpServer.enabled=true (mcp-server-only deployment, --use-fake-store) PASS — pod Ready, /healthz returns ok
MCP smoke — protocol (every tool) kubectl port-forward + JSON-RPC tools/list and tools/call against all 10 tools PASS — tools/list returns 10 tools; all 9 Working tools return structured success envelopes; Stub returns the documented NVSENTINEL_DATA_GAP envelope
Cluster smoke — GPU (real K8s API query) holodeck single-node g4dn.xlarge (Tesla T4) with NVIDIA driver 580.159.04 + GPU Operator (DCGM hostengine + device-plugin) + mcp-server via the chart PASS — describe_gpu_node returned real GPU metadata (Tesla-T4, 15360 MiB, g4dn.xlarge, CUDA driver 580.159.04); pod_gpu_allocation correctly listed a test pod requesting nvidia.com/gpu: 1 with {node, namespace, pod, requested:1}
Auth chain end-to-end unit TestResolveAuthToken + helm template showing chart-injected MCP_AUTH_TOKEN env var + unit TestRequireBearerAuth_* (5 cases: missing/non-Bearer/wrong token/empty Bearer/correct Bearer) PASS — chart Secret → env var → Config.AuthTokenrequireBearerAuth enforcement chain covered

Documented deviations from the plan's AC gate

  • AC-2 (Tilt + kind integration suite): SKIPPED. Task 17 (committed integration test scaffold) deferred to a follow-up PR per scope-keeping decision; the cluster validation is exercised via AC-4/AC-5 manually instead of via a committed Tilt suite.
  • AC-3 (every Stub row linked to a real GitHub issue): No tracking issue is filed for get_nvlink_topology in this PR, per donor direction (2026-05-14). The stub's needed_monitor_extension envelope explains the gap inline and references AUDIT.md § 6.1, which contains a ready-to-file issue body for maintainers to use if requested.
  • AC-6 (pre-commit run --all-files): N/A. The repository does not maintain a .pre-commit-config.yaml; make lint-test (vet + golangci-lint + gotestsum + coverage) is the equivalent gate and is run above.

Cluster smoke setup (reproducer)

Protocol smoke (CPU cluster): single-node Kubernetes v1.31.1 via holodeck on AWS (t3.xlarge, us-west-2, ubuntu-24.04, no GPU). Image built locally with ko --local, sideloaded via ctr -n k8s.io image import, helm installed with global.mcpServer.enabled=true plus all other components disabled, deployment patched to add --use-fake-store. All 10 MCP tools returned the expected envelopes via kubectl port-forward + curl JSON-RPC.

Real-GPU smoke: single-node Kubernetes v1.31.1 via holodeck on AWS (g4dn.xlarge, us-east-1, ubuntu-24.04) with NVIDIA driver 580.159.04 (Tesla T4) + NVIDIA Container Toolkit installed by holodeck. NVIDIA GPU Operator v25.3.0 deployed via helm (with driver.enabled=false toolkit.enabled=false dcgm.enabled=true) to provide the DCGM hostengine + device-plugin + node feature discovery. mcp-server deployed via the NVSentinel chart (global.mcpServer.enabled=true, fake store). Created a CUDA test pod requesting nvidia.com/gpu: 1. Hit describe_gpu_node and pod_gpu_allocation via curl — both returned real GPU metadata and real pod allocation data from the cluster's K8s API.

Full monitor-chain validation was attempted and is documented as a known gap: loading global.gpuHealthMonitor.enabled=true + mongodbStore.enabled=true + eventExporter.enabled=true in the umbrella values failed cleanly with two infra dependencies absent from a vanilla cluster: (1) ghcr.io/nvidia/nvsentinel/{labeler,event-exporter,gpu-health-monitor,platform-connectors}:main return 401 — those component images are not public; (2) the bitnami MongoDB subchart wants a default StorageClass which holodeck's single-node K8s does not provide. Neither is something a donation PR for mcp-server should fix; both belong in a follow-up PR that wires up a reproducible full-stack integration env. The protocol+real-GPU smoke above is sufficient to assert that mcp-server itself is fit for review.

CI

GitHub Actions matrices wired in commit 229c871f:

  • lint-test matrix includes mcp-server
  • container-build-test matrix includes mcp-server
  • cleanup-untagged-images matrix includes mcp-server

First push of this branch will exercise CI against the new component.

Follow-up issues to file post-merge

  • Committed Task 17 integration suite (Tilt-driven cluster smoke) — should also cover the full monitor → store → mcp chain on a GPU cluster.
  • get_nvlink_topology real implementation (needs monitor extension — body drafted in AUDIT.md § 6.1).
  • TLS termination on /mcppkg/mcp/Config.TLS exists but main.go does not yet expose --tls-cert/--tls-key flags and the chart does not mount a cert; documented in the README.
  • Reproducible full-stack integration env: publish (or otherwise make pullable in CI) the ghcr.io/nvidia/nvsentinel/{labeler,event-exporter,gpu-health-monitor,platform-connectors} images, and ship a values overlay that picks a workable StorageClass for the mongodb subchart on single-node K8s.

Co-authorship

All commits are signed (-s -S) and include Co-authored-by: Carlos Arango Gutierrez <eduardoa@nvidia.com> per the donation merge spec § 9.4.

Summary by CodeRabbit

  • New Features

    • Added MCP (Model Context Protocol) server for GPU health diagnostics and monitoring
    • Added diagnostic tools for GPU inventory, health status analysis, failure investigation, incident reporting, and error code analysis
    • Added bearer token authentication support
    • Added built-in diagnostic prompts for common GPU troubleshooting scenarios
    • Added Prometheus metrics tracking
  • Infrastructure

    • Added Kubernetes Helm deployment chart with RBAC configuration
    • Updated CI/CD workflows to support building and testing the new component
    • Added module-level build and test targets

Review Change Stack

Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
…rage

Findings drive Task 2-22. 9 Working tools, 1 Stub (get_nvlink_topology).

Key spec divergences captured in AUDIT.md sections 2 and 7:

- No monitor gRPC services. Store-only read surface via store-client/pkg/datastore.HealthEventStore.

- Module path is github.com/nvidia/nvsentinel/mcp-server (lowercase).

- Go 1.26.0 toolchain go1.26.2 to match siblings.

- Helm subchart goes at distros/kubernetes/nvsentinel/charts/mcp-server/, not mcp-server/deploy/helm/.

- commons/pkg/{logger,server,metrics,tracing} adopted (event-exporter pattern).

- pkg/monitors/ dropped from the plan; Config struct loses Monitors field.

Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Stub tool get_nvlink_topology (Task 15) leaves tracking_issue empty and

documents the gap inline in needed_monitor_extension. The issue draft

stays in AUDIT.md \xc2\xa7 6.1 as supporting material in case maintainers

request a tracking issue during PR review.

Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Stateless Deployment skeleton mirroring event-exporter's layout. Uses commons/pkg/{logger,server,tracing} for structured logging, HTTP server, and OpenTelemetry tracing. Main loop starts metrics/health server on :9090 and a placeholder goroutine that blocks until SIGTERM. MCP transport wiring is Task 4.

Files added:

- mcp-server/go.mod (lowercase module path github.com/nvidia/nvsentinel/mcp-server, Go 1.26.0)

- mcp-server/main.go and main_test.go (testing CreateMetricsServer port parsing)

- mcp-server/Makefile (includes ../make/{common,go,docker}.mk)

- mcp-server/Tiltfile (ko-tilt-build, ghcr.io/nvidia/nvsentinel/mcp-server)

- mcp-server/README.md (stub pointing to AUDIT.md)

- mcp-server/.gitignore (excludes compiled binary, coverage)

Root files modified:

- .ko.yaml: add mcp-server build entry after event-exporter

- Makefile: add mcp-server to GO_MODULES + lint-test-mcp-server target

Smoke test: ./mcp-server --metrics-port=19091 logs JSON, serves /healthz + /metrics on the port, exits 0 on SIGTERM (verified).

Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Narrow read-only view of NVSentinel's HealthEventStore. MCP tools depend on the Reader interface; production wires it to DataStoreReader (thin wrapper around store-client/pkg/datastore.HealthEventStore), tests wire it to FakeReader (in-memory, with QueryBuilder capture for assertions).

Scope reduction from the original plan: pkg/monitors/ is dropped entirely. Per AUDIT.md, NVSentinel has no monitor-side gRPC read services; all data flows through the store. This collapses Task 3 to just pkg/store/.

Tests (5 cases, all green):

- EventsByNode returns seeded events in insertion order

- EventsByNode returns empty slice for unknown node (not an error)

- LatestEventForNode returns ErrNotFound for unknown node

- LatestEventForNode sorts by CreatedAt regardless of seed order

- EventsByQuery returns primed result + records the builder for inspection

Compile-time assertions ensure both DataStoreReader and FakeReader satisfy Reader; drift breaks the build before tests run.

Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Donated from ArangoGutierrez/k8s-gpu-mcp-server@80ac33d8. Reshape for NVSentinel idiom: Config struct rewritten to take store.Reader + kubernetes.Interface (no NVML/gateway/xid/k8s legacy deps); klog -> slog; oneshot stdio transport dropped; Bearer auth on /mcp instead of /metrics (metrics auth handled by commons/pkg/server).

Files:

- pkg/mcp/server.go: Config + Server + New + Run + Shutdown + empty registerTools skeleton (tool tasks 6-16 populate)

- pkg/mcp/http.go: streamable HTTP transport via mark3labs/mcp-go, Bearer auth middleware on /mcp, health/readyz/version endpoints, cert-reloader integration

- pkg/mcp/tls.go: cert-reloader (5min interval), klog -> slog

- pkg/mcp/metrics.go: mcp_server_{requests_total, request_duration_seconds, active_requests} via prometheus/client_golang; RecordRequest helper

- pkg/mcp/server_test.go: 3 real tests for New input validation (empty HTTPAddr, nil Store, valid config)

- main.go: replace placeholder goroutine with mcp.New() + Run() in errgroup; --mcp-addr and --auth-token flags; store wired to FakeReader as TODO until tool tasks bring real datastore client

- go.mod/go.sum: add github.com/mark3labs/mcp-go v0.54.0 as direct dep

Smoke verified: /version returns nvsentinel-mcp-server, /mcp reaches mcp-go (HTTP 400 for empty JSON-RPC body), /healthz on metrics port returns ok, SIGTERM -> clean exit 0.

Co-authored-by: Carlos Arango Gutierrez <eduardoa@nvidia.com>
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Donated from ArangoGutierrez/k8s-gpu-mcp-server@80ac33d8 verbatim — three prompts in the library: gpu-health-check, diagnose-xid-errors, gpu-triage. The donor's code had no legacy NVSentinel-incompatible imports, so only the copyright header was rewritten (dual NVIDIA + k8s-gpu-mcp-server contributors, verbose Apache block matching sibling components and pkg/mcp/ from Task 4).

Wired into pkg/mcp/server.go: added prompts import, set WithPromptCapabilities(true) on the mcp-go server, and registerPrompts() now iterates prompts.Library and calls AddPrompt for each. registerTools stays empty (tool tasks 6-16 populate).

Tests (donor-authored, 9 subtests, all green):

- TestPromptDef_ToMCPPrompt — round-trips Name/Description/Arguments through mcp-go's mcp.Prompt

- TestLibraryPrompts — every prompt in Library has non-empty Name and ToMCPPrompt success

- TestGPUHealthCheckPrompt — default vs custom node argument

- TestDiagnoseXIDErrorsPrompt — default vs custom time range argument

- TestGPUTriagePrompt — defaults vs with-incident-id

Co-authored-by: Carlos Arango Gutierrez <eduardoa@nvidia.com>
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Returns the per-node list of GPUs derived from health events, collapsed to the latest event per UUID. Source data is store-client's HealthEventStore (no NVML required, no per-monitor gRPC). Tool wired into mcp.Server.registerTools via a per-tool helper so subsequent tool tasks add one helper each rather than growing registerTools' body.

Co-authored-by: Carlos Arango Gutierrez <eduardoa@nvidia.com>
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Per-GPU health summary derived from health events: latest event state plus event-count and unhealthy-event-count aggregates. Optional gpu_uuid narrows the response to a single GPU. Shares the GPU UUID extraction helper with gpu_inventory.

Co-authored-by: Carlos Arango Gutierrez <eduardoa@nvidia.com>
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Pairs the latest store event for a node with a flattened Kubernetes Node description (labels, annotations, taints, conditions, GPU capacity/allocatable). Each data source is independently nullable; missing-data conditions surface as structured warnings rather than errors. Nil k8sClient is honoured per Config docs by skipping the K8s portion with a warning.

Co-authored-by: Carlos Arango Gutierrez <eduardoa@nvidia.com>
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Pure K8s tool: lists pods requesting GPUs across an optional namespace/node scope, with per-pod request count and UUIDs resolved from NVIDIA_VISIBLE_DEVICES. Rejects nil k8sClient since there is no fallback data source for this tool. Sentinel env values 'all'/'none'/'void' are excluded.

Co-authored-by: Carlos Arango Gutierrez <eduardoa@nvidia.com>
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Stitches three data sources for pod diagnosis: K8s Pod phase/restart count, K8s Events scoped to the pod, and NVSentinel store health events that name the pod in entitiesImpacted (matching either bare name or namespace/name).

Co-authored-by: Carlos Arango Gutierrez <eduardoa@nvidia.com>
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Ported from donor pkg/incidents/patterns.go and adapted to NVSentinel's event-only signals: XID codes via ErrorCode and substring matches against Message. Snapshot-derived indicators (temperature, ECC count, throttle reasons, mem utilisation) are deferred until a monitor extension persists that telemetry. Recommendations are kept verbatim. The pure MatchIncidents([]HealthEventWithStatus) []Incident function will back explain_failure (Task 12) and get_incident_report (Task 13).

Co-authored-by: Carlos Arango Gutierrez <eduardoa@nvidia.com>
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Runs the donated pattern matcher (MatchIncidents) over recent store events for a node, optionally narrowed to a GPU UUID and time-bounded by SinceMinutes (default 60). Returns a one-paragraph narrative naming the top pattern plus the full sorted list of matched incidents.

Co-authored-by: Carlos Arango Gutierrez <eduardoa@nvidia.com>
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Looks up an analyzer-synthesized event (agent=health-events-analyzer, id=incident_id) via EventsByQuery, then enriches with same-node events in a +/-30 min window. Recommendations come from MatchIncidents over the combined event set. Severity is derived from isFatal/isHealthy (critical/warning/info).

Co-authored-by: Carlos Arango Gutierrez <eduardoa@nvidia.com>
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Working per AUDIT (plan said 'Working OR Stub'; AUDIT confirms XID events are persisted by syslog-health-monitor with the numeric code in ErrorCode). Queries the store via EventsByQuery filtered on errorcode, attributes to nodes/GPUs, and pairs with the matching donor pattern via MatchIncidents.

Co-authored-by: Carlos Arango Gutierrez <eduardoa@nvidia.com>
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Per AUDIT § 3, NVSentinel does not persist NVLink topology in its store (the data lives in per-node gpu_metadata.json read by syslog-health-monitor). The tool returns the NVSENTINEL_DATA_GAP envelope from design spec § 6.3 with an inline explanation. tracking_issue is intentionally empty per donor direction (AUDIT § 6); the proposed monitor extension is documented in AUDIT § 6.1.

Co-authored-by: Carlos Arango Gutierrez <eduardoa@nvidia.com>
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Working per AUDIT (plan said 'Working OR Stub'; AUDIT confirms generatedTimestamp has nanosecond precision via proto3 google.protobuf.Timestamp). Reads EventsByNode, filters by SinceMinutes window and optional GPUUUID, and returns the timeline in ascending order with severity derived from isFatal/isHealthy.

Co-authored-by: Carlos Arango Gutierrez <eduardoa@nvidia.com>
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Replaces the FakeReader placeholder with a real store.DataStoreReader built from store-client's provider registry (env-driven Mongo or Postgres). Adds an in-cluster Kubernetes client; non-cluster runs disable K8s-touching tools per their nil-client contract. --use-fake-store retained for local dev.

Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
…lse default

Subchart at distros/kubernetes/nvsentinel/charts/mcp-server/ per AUDIT § 7 (not mcp-server/deploy/helm/ as the original plan said). Mirrors event-exporter's MongoDB+Postgres dual-mode datastore wiring. ClusterRole grants read on nodes/pods/events (cluster-wide because describe_gpu_node, pod_gpu_allocation, and pod_failure all need it). Default is opt-in: global.mcpServer.enabled: false.

Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
… E2E steps

Replaces the Task 2 stub with a real README: tool catalog (cross-linked to AUDIT.md), helm install quick start, Bearer-token auth config, and explicit 'test against a real GPU cluster' procedure that is the pre-PR acceptance gate (build/push, helm upgrade, port-forward, tools/list, per-tool tools/call smoke). Troubleshooting covers the common k8s-not-configured / FakeReader-only / NVLink-stub paths.

Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
…r, noctx)

Mechanical lint cleanup so CI's modules-lint-test matrix entry for mcp-server passes. Behavior unchanged; tests still pass.

Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
…ainer-build-test, cleanup-untagged-images)

Adds mcp-server to the three matrix-driven workflows that enumerate per-module CI work: modules-lint-test (runs make -C mcp-server lint-test), container-build-test (ko build, amd64+arm64), and cleanup-untagged-images (registry hygiene for nvsentinel/mcp-server).

Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
…names

Live MCP smoke against the running server caught the donor prompts referencing four tool names that don't exist in this server (get_gpu_inventory -> gpu_inventory, get_gpu_health -> gpu_health, get_pod_gpu_allocation -> pod_gpu_allocation, analyze_xid_errors -> analyze_xid). AI assistants following the prompts would call non-existent tools. Updates the templates and the corresponding test assertions to the canonical NVSentinel tool names from AUDIT.md § 3.

Co-authored-by: Carlos Arango Gutierrez <eduardoa@nvidia.com>
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Tag Config.AuthToken with `json:"-"` so it cannot accidentally leak via
JSON marshaling. Config is currently a DI boundary and not serialized,
but gosec G117 flags the field as matching a secret pattern; the tag
makes the safe-by-default contract explicit and silences the linter.

Co-authored-by: Carlos Arango Gutierrez <eduardoa@nvidia.com>
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
… auth actually works

The Helm chart injects MCP_AUTH_TOKEN via secretKeyRef when
authToken.secretName is set, but main.go previously only read the
--auth-token flag. The two did not connect, so every chart-deployed
instance served /mcp unauthenticated regardless of the values overlay.

Add resolveAuthToken(): the --auth-token flag wins when set, otherwise
fall back to os.Getenv("MCP_AUTH_TOKEN"). Unit tests cover the four
flag/env combinations. Verified end-to-end with helm template — the
rendered Deployment now wires the secret-sourced env var through to
the running binary.

README updated: correct the /readyz claim (chart probes /healthz for
both liveness and readiness — there is no separate /readyz), explain
the env-var fallback in the Authentication section, and call out the
remaining TLS-flag gap (pkg/mcp/Config.TLS exists but is not exposed
via main.go flags or the chart) as a follow-up.

Found by pre-push principal-engineer review.

Co-authored-by: Carlos Arango Gutierrez <eduardoa@nvidia.com>
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
The bearer-auth enforcement path (pkg/mcp/http.go requireBearerAuth)
shipped without direct test coverage; gaps surfaced during the
pre-push review when verifying the env-var fallback fix end-to-end.

Add five focused tests, one per failure mode:
  - missing Authorization header
  - non-Bearer scheme (e.g. Basic)
  - wrong token
  - empty Bearer token
  - correct Bearer reaches next handler

Each test names a real auth-bypass class. Combined with the existing
TestResolveAuthToken (env var fallback) and the helm-template
verification that the chart injects MCP_AUTH_TOKEN from the named
Secret, the chart \xe2\x86\x92 binary \xe2\x86\x92 enforcement chain is fully covered.

Co-authored-by: Carlos Arango Gutierrez <eduardoa@nvidia.com>
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
@coderabbitai

coderabbitai Bot commented May 25, 2026

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

This PR adds the mcp-server component to NVSentinel, a read-only Model Context Protocol server that exposes GPU health events and diagnostic tools. The implementation includes CI/CD integration, Kubernetes Helm charts, an HTTP server with optional TLS and bearer-token auth, ten diagnostic tools (gpu_inventory, gpu_health, describe_gpu_node, pod_gpu_allocation, pod_failure, explain_failure, analyze_xid, get_incident_report, get_gpu_timeline, get_nvlink_topology), incident pattern matching, prompt templates, and comprehensive documentation.

Changes

mcp-server: Complete MCP Server Implementation

Layer / File(s) Summary
Build & Development Infrastructure
.agents/plans/donation-source.md, .github/workflows/cleanup-untagged-images.yml, .github/workflows/container-build-test.yml, .github/workflows/lint-test.yml, .ko.yaml, Makefile, mcp-server/Makefile, mcp-server/.gitignore, mcp-server/Tiltfile, mcp-server/go.mod
mcp-server is added to CI/CD pipelines (lint, build, image cleanup), root Makefile module list, ko build configuration, and defines its own Go module with direct dependencies (mcp-go, prometheus, kubernetes, local commons). Local development via Tiltfile with ko-based builds.
Kubernetes Deployment & Helm Charts
distros/kubernetes/nvsentinel/Chart.yaml, distros/kubernetes/nvsentinel/values.yaml, distros/kubernetes/nvsentinel/charts/mcp-server/*
New Helm subchart for mcp-server (v0.1.0) with Deployment, Service, ServiceAccount, ClusterRole, ClusterRoleBinding templates; helpers for consistent naming and labels; configurable auth, ports, datastore cert mounts, and optional tracing. Parent chart enables the subchart via global.mcpServer.enabled.
HTTP Server & TLS Transport
mcp-server/pkg/mcp/http.go, mcp-server/pkg/mcp/http_test.go, mcp-server/pkg/mcp/tls.go
HTTPServer wires /mcp (stateless MCP), /healthz, /readyz, /version endpoints; optional bearer-token auth middleware; TLS with file-based certificate reloading (5-minute check interval). Tests cover auth rejection cases and correct token acceptance.
Server Entry Point & Lifecycle
mcp-server/main.go, mcp-server/main_test.go
Entry point: parses flags (listen address, metrics port, auth token, fake-store mode), resolves auth from flag or MCP_AUTH_TOKEN, creates metrics server, initializes store reader (FakeReader for dev, real datastore for prod), builds optional K8s client (graceful nil when not in cluster), runs metrics and MCP concurrently with signal-driven shutdown, graceful error handling.
Store Reader & Data Access
mcp-server/pkg/store/store.go, mcp-server/pkg/store/store_test.go
Read-only Reader interface: EventsByNode, LatestEventForNode (with ErrNotFound), EventsByQuery. DataStoreReader wraps datastore.HealthEventStore with error context. FakeReader: in-memory, mutex-protected, seeded events, recorded query builders for test assertions.
Prompt Definitions & Templates
mcp-server/pkg/prompts/prompts.go, mcp-server/pkg/prompts/library.go, mcp-server/pkg/prompts/prompts_test.go
PromptDef/ArgumentDef structs with ToMCPPrompt conversion, RenderTemplate ({{key}} substitution), and BuildHandler (validation, rendering, return MCP result). Library: three workflows (gpu-health-check, diagnose-xid-errors, gpu-triage) with argument schema and full templates.
MCP Server Core & Tool Registration
mcp-server/pkg/mcp/server.go, mcp-server/pkg/mcp/server_test.go, mcp-server/pkg/mcp/metrics.go
Server: Config struct (version, listen address, auth, store, optional K8s client, optional TLS), New validates required fields, Run creates/configures HTTP server and starts serving, Shutdown gracefully stops. Centralized registerTools() wires ten tools plus prompts. Prometheus metrics: RequestsTotal, RequestDuration, ActiveRequests by tool.
GPU Diagnostic Tools
mcp-server/pkg/tools/gpu_inventory.go, mcp-server/pkg/tools/gpu_health.go, mcp-server/pkg/tools/describe_gpu_node.go, mcp-server/pkg/tools/gpu_*_test.go
gpu_inventory: lists GPUs per node from store. gpu_health: aggregates per-GPU event counts, health status, latest message/check by timestamp. describe_gpu_node: combines store latest event with optional K8s node details (labels, capacity, taints, conditions), warnings when either source is unavailable.
Pod & Kubernetes-Aware Tools
mcp-server/pkg/tools/pod_gpu_allocation.go, mcp-server/pkg/tools/pod_failure.go, mcp-server/pkg/tools/pod_*_test.go
pod_gpu_allocation: lists pods requesting GPUs, extracts per-pod requested count and UUID list from NVIDIA_VISIBLE_DEVICES. pod_failure: fetches Kubernetes Pod, lists Events, queries store for health events mentioning the pod (by name or namespace/name), returns phase, restart count, event lists, and warnings.
Failure Analysis & Timeline Tools
mcp-server/pkg/tools/explain_failure.go, mcp-server/pkg/tools/analyze_xid.go, mcp-server/pkg/tools/get_gpu_timeline.go, mcp-server/pkg/tools/*_test.go
explain_failure: filters node events by time window and optional GPU UUID, matches incidents, returns narrative plus incident list. analyze_xid: validates XID code, queries by errorcode, aggregates affected nodes/GPUs, selects top pattern. get_gpu_timeline: returns chronologically sorted events with optional GPU/time filtering.
Incident Detection & Reporting
mcp-server/pkg/tools/get_incident_report.go, mcp-server/pkg/tools/get_nvlink_topology.go, mcp-server/pkg/tools/*_test.go
get_incident_report: fetches analyzer-synthesized incident event, loads related same-node events (±30 min window), derives severity, applies MatchIncidents for root-cause and recommendations. get_nvlink_topology: stub returning NVSENTINEL_DATA_GAP data-gap envelope (placeholder pending monitor persistence).
Incident Pattern Matching & Scoring
mcp-server/pkg/tools/incidents.go, mcp-server/pkg/tools/incidents_test.go
KnownIncidentPatterns registry: XID 79 (bus error), NVLink, ECC, software OOM, thermal cascade with XID codes and message phrases. MatchIncidents: extracts signals, scores by matching indicator types (XID presence + message substrings), returns sorted incidents by descending confidence with evidence and recommendations.
Documentation & Audit Trail
mcp-server/AUDIT.md, mcp-server/README.md
AUDIT.md: detailed monitor-surface audit, spec divergences, per-tool data sources matrix (9 working + 1 stub), canonical HealthEvent schema, architectural simplifications. README: purpose, tool status, Helm quick-start, bearer-token auth docs, configuration details, end-to-end testing checklist, troubleshooting, and licensing.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Poem

🐰 A server springs forth from the GitHub release,
Ten tools for diagnostics, GPU's health at peace,
With incident patterns and prompts in place,
The MCP server now logs every trace! 🚀✨

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/mcp-server-merge

@github-actions

Copy link
Copy Markdown
Contributor

Merging this branch will increase overall coverage

Impacted Packages Coverage Δ 🤖
github.com/nvidia/nvsentinel/mcp-server/pkg/mcp 7.74% (+7.74%) 👍

Coverage by file

Changed unit test files

  • github.com/nvidia/nvsentinel/mcp-server/pkg/mcp/http_test.go

@ArangoGutierrez
ArangoGutierrez requested a review from mchmarny May 25, 2026 13:25
@ArangoGutierrez ArangoGutierrez self-assigned this May 25, 2026
@ArangoGutierrez
ArangoGutierrez marked this pull request as ready for review May 25, 2026 13:45

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 13

🧹 Nitpick comments (8)
mcp-server/main.go (2)

194-194: ⚡ Quick win

Document the exported CreateMetricsServer function.

Please add a Go doc comment for CreateMetricsServer to satisfy exported-function documentation requirements.

As per coding guidelines "Include function comments for exported Go functions".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@mcp-server/main.go` at line 194, Add a Go doc comment immediately above the
exported CreateMetricsServer function that begins with "CreateMetricsServer" and
briefly describes what the function does, its parameters and its return values
(e.g., creating and returning a metrics HTTP server on the provided port and an
error if creation fails); ensure the comment follows Go doc style (starts with
the function name) to satisfy exported-function documentation requirements for
CreateMetricsServer.

15-16: ⚡ Quick win

Add package-level godoc for package main.

This package is missing a package doc comment; add a short // Package main ... comment immediately above the package declaration.

As per coding guidelines "Include package-level godoc for all Go packages".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@mcp-server/main.go` around lines 15 - 16, Add a package-level godoc comment
immediately above the package declaration for package main: insert a short
comment starting with "// Package main ..." that briefly describes the purpose
of the executable (what the program does and its role in the project) so the
package has proper documentation for tools like godoc; place it directly above
the existing "package main" line.
mcp-server/pkg/mcp/http_test.go (1)

49-102: ⚡ Quick win

Align test assertions with the repository’s testify convention.

These checks currently use only stdlib testing; please switch to require/assert for consistency with the repo’s Go test guidelines.

As per coding guidelines "Use testify/assert and testify/require for assertions in Go tests".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@mcp-server/pkg/mcp/http_test.go` around lines 49 - 102, Tests in
TestRequireBearerAuth_* use stdlib t.Errorf/t.Error instead of the repo standard
testify helpers; update each test
(TestRequireBearerAuth_MissingHeaderReturns401, _NonBearerSchemeReturns401,
_WrongTokenReturns401, _EmptyBearerReturns401, _CorrectBearerReachesNext) to
import "github.com/stretchr/testify/require" and replace the status and reached
checks with require.Equal/require.True/require.False calls (e.g.
require.Equal(t, http.StatusUnauthorized, status) and require.False(t, reached)
for cases that must not reach the next handler, and require.Equal(t,
http.StatusOK, status) plus require.True(t, reached) for the success case) while
keeping runAuthCase and testAuthToken usage unchanged.
mcp-server/main_test.go (1)

22-67: ⚡ Quick win

Use require/assert for these test assertions per repo test style.

The scenarios are good; switching to testify/require + testify/assert will align this file with the project’s test convention.

As per coding guidelines "Use testify/assert and testify/require for assertions in Go tests".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@mcp-server/main_test.go` around lines 22 - 67, Replace plain t.* checks in
TestCreateMetricsServer_InvalidPortReturnsError,
TestCreateMetricsServer_ValidPortReturnsServer and TestResolveAuthToken with
testify assertions: use require.Error/require.NoError/require.NotNil for fatal
prerequisites (e.g., require.Error(t, err) in
TestCreateMetricsServer_InvalidPortReturnsError, require.NoError(t, err) and
require.NotNil(t, srv) in TestCreateMetricsServer_ValidPortReturnsServer) and
use assert.Equal/require.Equal for value comparisons in table-driven
TestResolveAuthToken; import "github.com/stretchr/testify/assert" and
"github.com/stretchr/testify/require" and remove direct
t.Fatal/t.Fatalf/t.Error/t.Errorf usages accordingly.
mcp-server/pkg/mcp/server_test.go (1)

32-81: ⚡ Quick win

Align constructor tests to testify assertion style.

Please replace stdlib assertion patterns here with require/assert to match the repository’s Go test convention.

As per coding guidelines "Use testify/assert and testify/require for assertions in Go tests".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@mcp-server/pkg/mcp/server_test.go` around lines 32 - 81, Replace the stdlib
test assertions in TestNew_RejectsEmptyHTTPAddr, TestNew_RejectsNilStore, and
TestNew_ReturnsServerForValidConfig with testify-style calls: import
"github.com/stretchr/testify/require" (and "assert" if needed) and use
require.Error/require.NoError/require.NotNil and assert.Contains where
appropriate (e.g., require.Error(t, err) and assert.Contains(t, err.Error(),
"HTTPAddr"/"Store"), require.NoError(t, err) and require.NotNil(t, srv)); remove
t.Fatal/t.Fatalf/t.Errorf usage and adjust imports.
mcp-server/pkg/prompts/prompts_test.go (1)

27-27: ⚡ Quick win

Align top-level test names to the required naming pattern.

Several test names are descriptive but don’t follow TestFunctionName_Scenario_ExpectedBehavior consistently.

Example renames
-func TestPromptDef_RenderTemplate(t *testing.T) {
+func TestPromptDef_RenderTemplate_AppliesSubstitutionsAndDefaults(t *testing.T) {

-func TestGetAllPromptNames(t *testing.T) {
+func TestGetAllPromptNames_LibraryLoaded_ReturnsAllRegisteredNames(t *testing.T) {

-func TestLibraryPrompts(t *testing.T) {
+func TestLibraryPrompts_EachPromptDefinition_ConvertsAndBuildsHandler(t *testing.T) {

As per coding guidelines **/*_test.go: Name tests descriptively following the pattern TestFunctionName_Scenario_ExpectedBehavior in Go.

Also applies to: 45-45, 108-108, 184-184, 207-207, 223-223, 242-242, 261-261, 278-278

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@mcp-server/pkg/prompts/prompts_test.go` at line 27, Rename top-level test
functions to follow the Go convention
TestFunctionName_Scenario_ExpectedBehavior; for example change
TestPromptDef_ToMCPPrompt to a descriptive form like
TestPromptDef_ToMCPPrompt_SuccessfulConversion (or another scenario/expected
outcome that matches the test), and do the same for the other test functions
referenced (the tests at the indicated locations), updating each function name
to the Test{Function}_{Scenario}_{ExpectedBehavior} pattern so they are
consistent and descriptive.
mcp-server/pkg/store/store_test.go (1)

45-133: ⚡ Quick win

Use assert/require helpers consistently in this _test.go file.

This file currently mixes direct t.Fatalf/t.Errorf checks where repo test style requires testify/assert and testify/require.

Suggested conversion pattern
+import (
+    ...
+    "github.com/stretchr/testify/assert"
+    "github.com/stretchr/testify/require"
+)

 got, err := r.EventsByNode(ctx, "gpu-node-1")
-if err != nil {
-    t.Fatalf("EventsByNode unexpected error: %v", err)
-}
+require.NoError(t, err)

-if len(got) != 2 {
-    t.Fatalf("want 2 events, got %d", len(got))
-}
+require.Len(t, got, 2)

-if got[0].RawEvent["checkName"] != "xid" {
-    t.Errorf("first event checkName = %v, want xid", got[0].RawEvent["checkName"])
-}
+assert.Equal(t, "xid", got[0].RawEvent["checkName"])

As per coding guidelines **/*_test.go: Use testify/assert and testify/require for assertions in Go tests.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@mcp-server/pkg/store/store_test.go` around lines 45 - 133, Replace direct
t.Fatalf/t.Errorf checks with testify helpers in the test functions
TestFakeReader_EventsByNode,
TestFakeReader_EventsByNode_UnknownNodeReturnsEmpty,
TestFakeReader_LatestEventForNode_UnknownNodeReturnsErrNotFound,
TestFakeReader_LatestEventForNode_ReturnsByCreatedAt, and
TestFakeReader_EventsByQuery_ReturnsSeededResultAndRecordsBuilder: use
require.NoError(t, err) for error checks, require.Len/require.Empty for slice
length expectations, require.Equal/require.True for value comparisons (e.g.,
CreatedAt equality), and assert.Type/require.Equal for verifying recorded
builder type and contents; also replace the err-is check with
require.ErrorIs/require.True(errors.Is(...)) or assert.True as appropriate so
all assertions follow the repo test style using testify's assert/require
helpers.
mcp-server/pkg/tools/pod_failure_test.go (1)

165-174: ⚡ Quick win

Add a regression test for nil store.Reader behavior.

You already test nil k8sClient; add the sibling case for nil reader so this path cannot regress into panic again when a pod is node-assigned.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@mcp-server/pkg/tools/pod_failure_test.go` around lines 165 - 174, Add a
sibling test that asserts the PodFailure handler returns an error when the
store.Reader is nil: create a test like TestPodFailure_NilReader_ReturnsError
which constructs the handler with NewPodFailureHandler(nil, fakeK8sClient) and
calls h.Handle(context.Background(), tools.PodFailureInput{Pod: "p", Namespace:
"ns"}) then require.Error and require.Contains on the error message (matching
the existing "store" or similar message used in PodFailureHandler.Handle). This
mirrors TestPodFailure_NilK8sClient_ReturnsError and prevents regressions that
previously caused a panic when a pod is node-assigned.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@distros/kubernetes/nvsentinel/charts/mcp-server/values.yaml`:
- Around line 15-30: Add short inline comments to document each top-level
default: explain replicaCount (number of mcp-server pods),
image.repository/image.pullPolicy/image.tag (container image source, pull policy
and tag/default behavior), podAnnotations (optional Pod annotations to be merged
into Pod spec), and resources.limits/resources.requests (CPU and memory limits
and requests) so the values.yaml is self-documenting and follows the chart
guideline; update the block containing replicaCount, image.*, podAnnotations,
and resources.* with one-line comments for each key describing purpose and
expected format/defaults.

In `@distros/kubernetes/nvsentinel/values.yaml`:
- Around line 183-184: Add an inline comment documenting the new Helm global
flag by describing what global.mcpServer.enabled controls, its default value
(false), and typical use-case (e.g., enabling the MCP server for multi-cluster
control plane or integration tests), placed immediately above or on the same
line as the global.mcpServer.enabled entry in values.yaml so it’s discoverable
in the main values file; reference the symbol global.mcpServer.enabled when
adding the short comment.

In `@mcp-server/AUDIT.md`:
- Line 8: Replace the hardcoded machine-local design spec path in the AUDIT.md
table entry ("| **Design spec** |
/Users/.../2026-05-13-merge-gpu-mcp-into-nvsentinel-design.md |") with a
repo-relative reference or a short note pointing to the tracked spec location
(for example a relative path inside docs/superpowers/specs or "See
docs/superpowers/specs/2026-05-13-merge-gpu-mcp-into-nvsentinel-design.md");
update the table cell so it no longer contains any /Users/... absolute path and
instead references the repo path or a pointer to where the spec is stored in the
repository.

In `@mcp-server/pkg/mcp/http.go`:
- Around line 259-276: The current handler uses http.Error which resets
Content-Type to text/plain, breaking the JSON API contract for the 401
responses; update the three places that call http.Error (the "authorization
required", "invalid authorization scheme", and "invalid token" branches) to
write JSON responses directly by setting
w.Header().Set("Content-Type","application/json"), calling
w.WriteHeader(http.StatusUnauthorized), and writing the JSON body (e.g.,
{"error":"..."}) to the response body, or add a small helper like
writeJSONError(w, status, message) and call it from those branches; locate these
changes around the auth/token logic that references auth, token, and
h.authToken.

In `@mcp-server/pkg/prompts/prompts.go`:
- Around line 115-118: The required-argument check in the prompt validation only
verifies presence in the args map and allows empty strings; update the check
around arg.Required so it fetches the value (e.g., v, ok := args[arg.Name]) and
treat missing OR empty-string values as missing (if !ok || v == "" { return nil,
fmt.Errorf("prompt %q: missing required argument %q", p.Name, arg.Name) }).
Ensure you reference arg.Required, args[arg.Name], p.Name and arg.Name in the
same validation block so empty required arguments fail validation.

In `@mcp-server/pkg/tools/analyze_xid.go`:
- Around line 158-169: The function recentEventSummaries builds the summaries
but truncates them before ensuring they are ordered by recency; modify
recentEventSummaries to sort the built slice (out) by the event timestamp in
descending order (newest first) using sort.Slice or sort.SliceStable on the
EventSummary.Timestamp (or the underlying events[*].Timestamp if you prefer) and
only then apply the limit slice operation, so that the returned slice contains
the true most-recent events; keep references to recentEventSummaries and
eventSummaryFromStored when locating the code to change.

In `@mcp-server/pkg/tools/get_incident_report.go`:
- Around line 191-193: The slice variable out is truncated with
getIncidentReportRelatedLimit before being deterministically ordered, causing
nondeterministic matching; sort out by event time (or other deterministic key)
prior to applying the 50-event cap so the truncated set is stable. In
get_incident_report.go, locate the code that builds the out slice (variable out)
and the truncation using getIncidentReportRelatedLimit, then insert a stable
sort (e.g., by timestamp or event_time) of out before the if len(out) >
getIncidentReportRelatedLimit check so downstream matching (used around the
incident matching logic) sees a deterministic, time-ordered subset.

In `@mcp-server/pkg/tools/get_nvlink_topology.go`:
- Around line 81-83: Trim and validate the node input before checking emptiness:
call strings.TrimSpace on in.Node and use the trimmed value for validation so
whitespace-only values are rejected and an error is returned (update the
existing if checking in.Node == "" and the other similar check that validates
in.Node later in the function). Ensure you use the trimmed variable for any
subsequent logic that references the node identifier.

In `@mcp-server/pkg/tools/gpu_health.go`:
- Around line 139-145: The current comparison uses only
ews.CreatedAt.After(entry.LastEventTime) which skips updating when both
timestamps are zero; change the condition in the update block for ews.CreatedAt
vs entry.LastEventTime to also handle zero-value timestamps (for example check
entry.LastEventTime.IsZero() or use a non-strict comparison like
!ews.CreatedAt.Before(entry.LastEventTime)) so that entry.LastEventTime,
entry.Healthy, entry.LastMessage, entry.LastCheck and entry.ErrorCodes are
populated when appropriate; update the block referencing ews.CreatedAt,
entry.LastEventTime, entry.Healthy, he.GetIsHealthy(), he.GetMessage(),
he.GetCheckName(), and he.GetErrorCode() accordingly.

In `@mcp-server/pkg/tools/gpu_inventory.go`:
- Around line 73-75: NewGPUInventoryHandler currently accepts a nil store.Reader
which causes a panic later in Handle when calling h.reader.EventsByNode; update
NewGPUInventoryHandler to check if the incoming reader (r) is nil and return nil
(or an explicit error variant if you prefer changing the signature) instead of
constructing a handler with a nil reader, and update callers of
NewGPUInventoryHandler to handle a nil return (or the new error) accordingly;
reference symbols: NewGPUInventoryHandler, GPUInventoryHandler, reader, and
Handle (which calls EventsByNode).

In `@mcp-server/pkg/tools/pod_failure.go`:
- Around line 78-79: NewPodFailureHandler currently allows a nil store.Reader
which causes a panic when relatedStoreEvents dereferences h.reader (e.g., in
relatedStoreEvents or Handle when node != ""), so add a nil-guard: either
validate and return an error or panic/fail-fast in NewPodFailureHandler if
reader is nil, or (preferable) keep constructor but have relatedStoreEvents and
Handle check h.reader != nil before calling reader methods and log a
warning/degrade behavior when nil. Update NewPodFailureHandler,
PodFailureHandler.relatedStoreEvents (and Handle where it uses reader) to
perform this nil check and handle the nil-reader path safely.
- Around line 151-162: The List call in listPodEvents currently fetches all
namespace events then filters client-side, which is inefficient and misses
enforcing ev.InvolvedObject.Kind == "Pod"; update the ListOptions in
pod_failure.go where evList is fetched (the call using
h.k8sClient.CoreV1().Events(in.Namespace).List) to include a FieldSelector
restricting involvedObject.name=in.Pod, involvedObject.namespace=in.Namespace
and involvedObject.kind=Pod so the server returns only pod events; keep or
simplify the subsequent loop checks (ev.InvolvedObject.Name/Namespace/Kind) as a
safety guard but rely on the server-side selector to reduce load.

In `@mcp-server/README.md`:
- Around line 56-60: The README mixes two Helm values key styles (`mcpServer` vs
`mcp-server`) which is confusing; pick one canonical form (e.g., `mcpServer`)
and update all examples so they use that same key consistently — update the YAML
overlay block that currently shows `mcpServer:` and any `--set` CLI examples
that use `mcp-server` to the chosen form, and do the same for the other
occurrence noted around the later example so all references to the mcp server
values use the identical key name (`mcpServer` or `mcp-server`) throughout the
file.

---

Nitpick comments:
In `@mcp-server/main_test.go`:
- Around line 22-67: Replace plain t.* checks in
TestCreateMetricsServer_InvalidPortReturnsError,
TestCreateMetricsServer_ValidPortReturnsServer and TestResolveAuthToken with
testify assertions: use require.Error/require.NoError/require.NotNil for fatal
prerequisites (e.g., require.Error(t, err) in
TestCreateMetricsServer_InvalidPortReturnsError, require.NoError(t, err) and
require.NotNil(t, srv) in TestCreateMetricsServer_ValidPortReturnsServer) and
use assert.Equal/require.Equal for value comparisons in table-driven
TestResolveAuthToken; import "github.com/stretchr/testify/assert" and
"github.com/stretchr/testify/require" and remove direct
t.Fatal/t.Fatalf/t.Error/t.Errorf usages accordingly.

In `@mcp-server/main.go`:
- Line 194: Add a Go doc comment immediately above the exported
CreateMetricsServer function that begins with "CreateMetricsServer" and briefly
describes what the function does, its parameters and its return values (e.g.,
creating and returning a metrics HTTP server on the provided port and an error
if creation fails); ensure the comment follows Go doc style (starts with the
function name) to satisfy exported-function documentation requirements for
CreateMetricsServer.
- Around line 15-16: Add a package-level godoc comment immediately above the
package declaration for package main: insert a short comment starting with "//
Package main ..." that briefly describes the purpose of the executable (what the
program does and its role in the project) so the package has proper
documentation for tools like godoc; place it directly above the existing
"package main" line.

In `@mcp-server/pkg/mcp/http_test.go`:
- Around line 49-102: Tests in TestRequireBearerAuth_* use stdlib
t.Errorf/t.Error instead of the repo standard testify helpers; update each test
(TestRequireBearerAuth_MissingHeaderReturns401, _NonBearerSchemeReturns401,
_WrongTokenReturns401, _EmptyBearerReturns401, _CorrectBearerReachesNext) to
import "github.com/stretchr/testify/require" and replace the status and reached
checks with require.Equal/require.True/require.False calls (e.g.
require.Equal(t, http.StatusUnauthorized, status) and require.False(t, reached)
for cases that must not reach the next handler, and require.Equal(t,
http.StatusOK, status) plus require.True(t, reached) for the success case) while
keeping runAuthCase and testAuthToken usage unchanged.

In `@mcp-server/pkg/mcp/server_test.go`:
- Around line 32-81: Replace the stdlib test assertions in
TestNew_RejectsEmptyHTTPAddr, TestNew_RejectsNilStore, and
TestNew_ReturnsServerForValidConfig with testify-style calls: import
"github.com/stretchr/testify/require" (and "assert" if needed) and use
require.Error/require.NoError/require.NotNil and assert.Contains where
appropriate (e.g., require.Error(t, err) and assert.Contains(t, err.Error(),
"HTTPAddr"/"Store"), require.NoError(t, err) and require.NotNil(t, srv)); remove
t.Fatal/t.Fatalf/t.Errorf usage and adjust imports.

In `@mcp-server/pkg/prompts/prompts_test.go`:
- Line 27: Rename top-level test functions to follow the Go convention
TestFunctionName_Scenario_ExpectedBehavior; for example change
TestPromptDef_ToMCPPrompt to a descriptive form like
TestPromptDef_ToMCPPrompt_SuccessfulConversion (or another scenario/expected
outcome that matches the test), and do the same for the other test functions
referenced (the tests at the indicated locations), updating each function name
to the Test{Function}_{Scenario}_{ExpectedBehavior} pattern so they are
consistent and descriptive.

In `@mcp-server/pkg/store/store_test.go`:
- Around line 45-133: Replace direct t.Fatalf/t.Errorf checks with testify
helpers in the test functions TestFakeReader_EventsByNode,
TestFakeReader_EventsByNode_UnknownNodeReturnsEmpty,
TestFakeReader_LatestEventForNode_UnknownNodeReturnsErrNotFound,
TestFakeReader_LatestEventForNode_ReturnsByCreatedAt, and
TestFakeReader_EventsByQuery_ReturnsSeededResultAndRecordsBuilder: use
require.NoError(t, err) for error checks, require.Len/require.Empty for slice
length expectations, require.Equal/require.True for value comparisons (e.g.,
CreatedAt equality), and assert.Type/require.Equal for verifying recorded
builder type and contents; also replace the err-is check with
require.ErrorIs/require.True(errors.Is(...)) or assert.True as appropriate so
all assertions follow the repo test style using testify's assert/require
helpers.

In `@mcp-server/pkg/tools/pod_failure_test.go`:
- Around line 165-174: Add a sibling test that asserts the PodFailure handler
returns an error when the store.Reader is nil: create a test like
TestPodFailure_NilReader_ReturnsError which constructs the handler with
NewPodFailureHandler(nil, fakeK8sClient) and calls
h.Handle(context.Background(), tools.PodFailureInput{Pod: "p", Namespace: "ns"})
then require.Error and require.Contains on the error message (matching the
existing "store" or similar message used in PodFailureHandler.Handle). This
mirrors TestPodFailure_NilK8sClient_ReturnsError and prevents regressions that
previously caused a panic when a pod is node-assigned.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 90ed20ec-d23f-40ca-a482-b0cd2e0ca6b3

📥 Commits

Reviewing files that changed from the base of the PR and between 451cb62 and 9584b68.

⛔ Files ignored due to path filters (1)
  • mcp-server/go.sum is excluded by !**/*.sum
📒 Files selected for processing (56)
  • .agents/plans/donation-source.md
  • .github/workflows/cleanup-untagged-images.yml
  • .github/workflows/container-build-test.yml
  • .github/workflows/lint-test.yml
  • .ko.yaml
  • Makefile
  • distros/kubernetes/nvsentinel/Chart.yaml
  • distros/kubernetes/nvsentinel/charts/mcp-server/Chart.yaml
  • distros/kubernetes/nvsentinel/charts/mcp-server/templates/_helpers.tpl
  • distros/kubernetes/nvsentinel/charts/mcp-server/templates/clusterrole.yaml
  • distros/kubernetes/nvsentinel/charts/mcp-server/templates/deployment.yaml
  • distros/kubernetes/nvsentinel/charts/mcp-server/templates/service.yaml
  • distros/kubernetes/nvsentinel/charts/mcp-server/templates/serviceaccount.yaml
  • distros/kubernetes/nvsentinel/charts/mcp-server/values.yaml
  • distros/kubernetes/nvsentinel/values.yaml
  • mcp-server/.gitignore
  • mcp-server/AUDIT.md
  • mcp-server/Makefile
  • mcp-server/README.md
  • mcp-server/Tiltfile
  • mcp-server/go.mod
  • mcp-server/main.go
  • mcp-server/main_test.go
  • mcp-server/pkg/mcp/http.go
  • mcp-server/pkg/mcp/http_test.go
  • mcp-server/pkg/mcp/metrics.go
  • mcp-server/pkg/mcp/server.go
  • mcp-server/pkg/mcp/server_test.go
  • mcp-server/pkg/mcp/tls.go
  • mcp-server/pkg/prompts/library.go
  • mcp-server/pkg/prompts/prompts.go
  • mcp-server/pkg/prompts/prompts_test.go
  • mcp-server/pkg/store/store.go
  • mcp-server/pkg/store/store_test.go
  • mcp-server/pkg/tools/analyze_xid.go
  • mcp-server/pkg/tools/analyze_xid_test.go
  • mcp-server/pkg/tools/describe_gpu_node.go
  • mcp-server/pkg/tools/describe_gpu_node_test.go
  • mcp-server/pkg/tools/explain_failure.go
  • mcp-server/pkg/tools/explain_failure_test.go
  • mcp-server/pkg/tools/get_gpu_timeline.go
  • mcp-server/pkg/tools/get_gpu_timeline_test.go
  • mcp-server/pkg/tools/get_incident_report.go
  • mcp-server/pkg/tools/get_incident_report_test.go
  • mcp-server/pkg/tools/get_nvlink_topology.go
  • mcp-server/pkg/tools/get_nvlink_topology_test.go
  • mcp-server/pkg/tools/gpu_health.go
  • mcp-server/pkg/tools/gpu_health_test.go
  • mcp-server/pkg/tools/gpu_inventory.go
  • mcp-server/pkg/tools/gpu_inventory_test.go
  • mcp-server/pkg/tools/incidents.go
  • mcp-server/pkg/tools/incidents_test.go
  • mcp-server/pkg/tools/pod_failure.go
  • mcp-server/pkg/tools/pod_failure_test.go
  • mcp-server/pkg/tools/pod_gpu_allocation.go
  • mcp-server/pkg/tools/pod_gpu_allocation_test.go

Comment on lines +15 to +30
replicaCount: 1

image:
repository: ghcr.io/nvidia/nvsentinel/mcp-server
pullPolicy: IfNotPresent
tag: ""

podAnnotations: {}

resources:
limits:
cpu: "500m"
memory: "512Mi"
requests:
cpu: "100m"
memory: "128Mi"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Add inline comments for the remaining top-level defaults.

Line 15-Line 30 introduces undocumented values (replicaCount, image.*, podAnnotations, resources.*). Please add brief inline comments for each key to keep this chart compliant and self-documenting.

As per coding guidelines "Document all values in Helm chart values.yaml with inline comments."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@distros/kubernetes/nvsentinel/charts/mcp-server/values.yaml` around lines 15
- 30, Add short inline comments to document each top-level default: explain
replicaCount (number of mcp-server pods),
image.repository/image.pullPolicy/image.tag (container image source, pull policy
and tag/default behavior), podAnnotations (optional Pod annotations to be merged
into Pod spec), and resources.limits/resources.requests (CPU and memory limits
and requests) so the values.yaml is self-documenting and follows the chart
guideline; update the block containing replicaCount, image.*, podAnnotations,
and resources.* with one-line comments for each key describing purpose and
expected format/defaults.

Comment on lines +183 to +184
mcpServer:
enabled: false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Document the new global flag inline.

Please add a short inline comment for global.mcpServer.enabled so this new switch is discoverable in the main values file.

As per coding guidelines "Document all values in Helm chart values.yaml with inline comments."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@distros/kubernetes/nvsentinel/values.yaml` around lines 183 - 184, Add an
inline comment documenting the new Helm global flag by describing what
global.mcpServer.enabled controls, its default value (false), and typical
use-case (e.g., enabling the MCP server for multi-cluster control plane or
integration tests), placed immediately above or on the same line as the
global.mcpServer.enabled entry in values.yaml so it’s discoverable in the main
values file; reference the symbol global.mcpServer.enabled when adding the short
comment.

Comment thread mcp-server/AUDIT.md
| **Date** | 2026-05-14 |
| **Auditor** | Claude Code session, working on `feat/mcp-server-merge` |
| **Donation source** | `ArangoGutierrez/k8s-gpu-mcp-server@80ac33d89ede70aa3f967088f8716d94b8e692e4` (pinned in `.agents/plans/donation-source.md`) |
| **Design spec** | `/Users/eduardoa/src/github/ArangoGutierrez/k8s-gpu-mcp-server/.worktrees/merge-into-nvsentinel/docs/superpowers/specs/2026-05-13-merge-gpu-mcp-into-nvsentinel-design.md` |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Replace the machine-local spec path with a repo-relative reference.

Line 8 hardcodes a local /Users/... path, which is not usable by other contributors and leaks workstation-specific info. Prefer a repo-relative path or a short note pointing to the tracked spec location.

🧰 Tools
🪛 LanguageTool

[uncategorized] ~8-~8: The official name of this software platform is spelled with a capital “H”.
Context: ...ation-source.md) | | **Design spec** | /Users/eduardoa/src/github/ArangoGutierrez/k8s-gpu-mcp-server/.wor...

(GITHUB)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@mcp-server/AUDIT.md` at line 8, Replace the hardcoded machine-local design
spec path in the AUDIT.md table entry ("| **Design spec** |
/Users/.../2026-05-13-merge-gpu-mcp-into-nvsentinel-design.md |") with a
repo-relative reference or a short note pointing to the tracked spec location
(for example a relative path inside docs/superpowers/specs or "See
docs/superpowers/specs/2026-05-13-merge-gpu-mcp-into-nvsentinel-design.md");
update the table cell so it no longer contains any /Users/... absolute path and
instead references the repo path or a pointer to where the spec is stored in the
repository.

Comment on lines +259 to +276
w.Header().Set("Content-Type", "application/json")
http.Error(w, `{"error":"authorization required"}`, http.StatusUnauthorized)

return
}

token := strings.TrimPrefix(auth, "Bearer ")
if token == auth {
w.Header().Set("Content-Type", "application/json")
http.Error(w, `{"error":"invalid authorization scheme"}`, http.StatusUnauthorized)

return
}

if subtle.ConstantTimeCompare([]byte(token), []byte(h.authToken)) != 1 {
w.Header().Set("Content-Type", "application/json")
http.Error(w, `{"error":"invalid token"}`, http.StatusUnauthorized)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Return real JSON errors instead of http.Error text responses.

http.Error overwrites Content-Type to text/plain, so these 401 responses are not actually JSON despite JSON-looking strings. Write status/body directly (or via a small helper) to keep the API contract consistent.

💡 Suggested fix
+func writeJSONError(w http.ResponseWriter, code int, body string) {
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(code)
+	_, _ = w.Write([]byte(body))
+}
+
 func (h *HTTPServer) requireBearerAuth(next http.Handler) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		auth := r.Header.Get("Authorization")
 		if auth == "" {
-			w.Header().Set("Content-Type", "application/json")
-			http.Error(w, `{"error":"authorization required"}`, http.StatusUnauthorized)
-
+			writeJSONError(w, http.StatusUnauthorized, `{"error":"authorization required"}`)
 			return
 		}
@@
 		token := strings.TrimPrefix(auth, "Bearer ")
 		if token == auth {
-			w.Header().Set("Content-Type", "application/json")
-			http.Error(w, `{"error":"invalid authorization scheme"}`, http.StatusUnauthorized)
-
+			writeJSONError(w, http.StatusUnauthorized, `{"error":"invalid authorization scheme"}`)
 			return
 		}
@@
 		if subtle.ConstantTimeCompare([]byte(token), []byte(h.authToken)) != 1 {
-			w.Header().Set("Content-Type", "application/json")
-			http.Error(w, `{"error":"invalid token"}`, http.StatusUnauthorized)
-
+			writeJSONError(w, http.StatusUnauthorized, `{"error":"invalid token"}`)
 			return
 		}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
w.Header().Set("Content-Type", "application/json")
http.Error(w, `{"error":"authorization required"}`, http.StatusUnauthorized)
return
}
token := strings.TrimPrefix(auth, "Bearer ")
if token == auth {
w.Header().Set("Content-Type", "application/json")
http.Error(w, `{"error":"invalid authorization scheme"}`, http.StatusUnauthorized)
return
}
if subtle.ConstantTimeCompare([]byte(token), []byte(h.authToken)) != 1 {
w.Header().Set("Content-Type", "application/json")
http.Error(w, `{"error":"invalid token"}`, http.StatusUnauthorized)
func writeJSONError(w http.ResponseWriter, code int, body string) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(code)
_, _ = w.Write([]byte(body))
}
func (h *HTTPServer) requireBearerAuth(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
auth := r.Header.Get("Authorization")
if auth == "" {
writeJSONError(w, http.StatusUnauthorized, `{"error":"authorization required"}`)
return
}
token := strings.TrimPrefix(auth, "Bearer ")
if token == auth {
writeJSONError(w, http.StatusUnauthorized, `{"error":"invalid authorization scheme"}`)
return
}
if subtle.ConstantTimeCompare([]byte(token), []byte(h.authToken)) != 1 {
writeJSONError(w, http.StatusUnauthorized, `{"error":"invalid token"}`)
return
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@mcp-server/pkg/mcp/http.go` around lines 259 - 276, The current handler uses
http.Error which resets Content-Type to text/plain, breaking the JSON API
contract for the 401 responses; update the three places that call http.Error
(the "authorization required", "invalid authorization scheme", and "invalid
token" branches) to write JSON responses directly by setting
w.Header().Set("Content-Type","application/json"), calling
w.WriteHeader(http.StatusUnauthorized), and writing the JSON body (e.g.,
{"error":"..."}) to the response body, or add a small helper like
writeJSONError(w, status, message) and call it from those branches; locate these
changes around the auth/token logic that references auth, token, and
h.authToken.

Comment on lines +115 to +118
if arg.Required {
if _, ok := args[arg.Name]; !ok {
return nil, fmt.Errorf("prompt %q: missing required argument %q", p.Name, arg.Name)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Treat empty required arguments as missing.

A required argument currently passes validation when present but empty (""), which can silently produce invalid prompt content.

Proposed fix
 for _, arg := range p.Arguments {
     if arg.Required {
-        if _, ok := args[arg.Name]; !ok {
+        value, ok := args[arg.Name]
+        if !ok || strings.TrimSpace(value) == "" {
             return nil, fmt.Errorf("prompt %q: missing required argument %q", p.Name, arg.Name)
         }
     }
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if arg.Required {
if _, ok := args[arg.Name]; !ok {
return nil, fmt.Errorf("prompt %q: missing required argument %q", p.Name, arg.Name)
}
for _, arg := range p.Arguments {
if arg.Required {
value, ok := args[arg.Name]
if !ok || strings.TrimSpace(value) == "" {
return nil, fmt.Errorf("prompt %q: missing required argument %q", p.Name, arg.Name)
}
}
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@mcp-server/pkg/prompts/prompts.go` around lines 115 - 118, The
required-argument check in the prompt validation only verifies presence in the
args map and allows empty strings; update the check around arg.Required so it
fetches the value (e.g., v, ok := args[arg.Name]) and treat missing OR
empty-string values as missing (if !ok || v == "" { return nil,
fmt.Errorf("prompt %q: missing required argument %q", p.Name, arg.Name) }).
Ensure you reference arg.Required, args[arg.Name], p.Name and arg.Name in the
same validation block so empty required arguments fail validation.

Comment on lines +139 to +145
if ews.CreatedAt.After(entry.LastEventTime) {
entry.LastEventTime = ews.CreatedAt
entry.Healthy = he.GetIsHealthy()
entry.LastMessage = he.GetMessage()
entry.LastCheck = he.GetCheckName()
entry.ErrorCodes = append([]string{}, he.GetErrorCode()...)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Handle zero-value timestamps when selecting latest event.

On Line 139, using only After can skip population when entry.LastEventTime is zero and incoming CreatedAt is also zero, leaving Healthy/LastMessage/LastCheck at defaults.

Suggested fix
-		if ews.CreatedAt.After(entry.LastEventTime) {
+		if entry.LastEventTime.IsZero() || ews.CreatedAt.After(entry.LastEventTime) {
 			entry.LastEventTime = ews.CreatedAt
 			entry.Healthy = he.GetIsHealthy()
 			entry.LastMessage = he.GetMessage()
 			entry.LastCheck = he.GetCheckName()
 			entry.ErrorCodes = append([]string{}, he.GetErrorCode()...)
 		}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if ews.CreatedAt.After(entry.LastEventTime) {
entry.LastEventTime = ews.CreatedAt
entry.Healthy = he.GetIsHealthy()
entry.LastMessage = he.GetMessage()
entry.LastCheck = he.GetCheckName()
entry.ErrorCodes = append([]string{}, he.GetErrorCode()...)
}
if entry.LastEventTime.IsZero() || ews.CreatedAt.After(entry.LastEventTime) {
entry.LastEventTime = ews.CreatedAt
entry.Healthy = he.GetIsHealthy()
entry.LastMessage = he.GetMessage()
entry.LastCheck = he.GetCheckName()
entry.ErrorCodes = append([]string{}, he.GetErrorCode()...)
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@mcp-server/pkg/tools/gpu_health.go` around lines 139 - 145, The current
comparison uses only ews.CreatedAt.After(entry.LastEventTime) which skips
updating when both timestamps are zero; change the condition in the update block
for ews.CreatedAt vs entry.LastEventTime to also handle zero-value timestamps
(for example check entry.LastEventTime.IsZero() or use a non-strict comparison
like !ews.CreatedAt.Before(entry.LastEventTime)) so that entry.LastEventTime,
entry.Healthy, entry.LastMessage, entry.LastCheck and entry.ErrorCodes are
populated when appropriate; update the block referencing ews.CreatedAt,
entry.LastEventTime, entry.Healthy, he.GetIsHealthy(), he.GetMessage(),
he.GetCheckName(), and he.GetErrorCode() accordingly.

Comment on lines +73 to +75
func NewGPUInventoryHandler(r store.Reader) *GPUInventoryHandler {
return &GPUInventoryHandler{reader: r}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Guard against nil store.Reader to avoid panic on Line 86.

NewGPUInventoryHandler currently allows a nil dependency, which can crash at runtime when Handle calls h.reader.EventsByNode(...).

Suggested fix
 func NewGPUInventoryHandler(r store.Reader) *GPUInventoryHandler {
-	return &GPUInventoryHandler{reader: r}
+	return &GPUInventoryHandler{reader: r}
 }
 
 func (h *GPUInventoryHandler) Handle(ctx context.Context, in GPUInventoryInput) (GPUInventoryOutput, error) {
+	if h == nil || h.reader == nil {
+		return GPUInventoryOutput{}, errors.New("gpu_inventory: reader is not configured")
+	}
 	if in.Node == "" {
 		return GPUInventoryOutput{}, errors.New("gpu_inventory: node is required")
 	}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@mcp-server/pkg/tools/gpu_inventory.go` around lines 73 - 75,
NewGPUInventoryHandler currently accepts a nil store.Reader which causes a panic
later in Handle when calling h.reader.EventsByNode; update
NewGPUInventoryHandler to check if the incoming reader (r) is nil and return nil
(or an explicit error variant if you prefer changing the signature) instead of
constructing a handler with a nil reader, and update callers of
NewGPUInventoryHandler to handle a nil return (or the new error) accordingly;
reference symbols: NewGPUInventoryHandler, GPUInventoryHandler, reader, and
Handle (which calls EventsByNode).

Comment on lines +78 to +79
func NewPodFailureHandler(r store.Reader, k kubernetes.Interface) *PodFailureHandler {
return &PodFailureHandler{reader: r, k8sClient: k}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Guard reader to prevent nil dereference in store-event path.

NewPodFailureHandler accepts a nil store.Reader, but relatedStoreEvents dereferences it when node != "", which can panic requests. Add a nil-reader guard and degrade to warning (or fail fast at constructor/Handle).

💡 Suggested fix
 func (h *PodFailureHandler) relatedStoreEvents(
 	ctx context.Context, in PodFailureInput, node string,
 ) ([]EventSummary, string, error) {
 	if node == "" {
 		return nil, "pod has no assigned node; skipping store events", nil
 	}
+	if h.reader == nil {
+		return nil, "store reader not configured; skipping store events", nil
+	}
 
 	events, err := h.reader.EventsByNode(ctx, node)
 	if err != nil {
 		return nil, "", fmt.Errorf("pod_failure: events by node: %w", err)
 	}

Also applies to: 184-187

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@mcp-server/pkg/tools/pod_failure.go` around lines 78 - 79,
NewPodFailureHandler currently allows a nil store.Reader which causes a panic
when relatedStoreEvents dereferences h.reader (e.g., in relatedStoreEvents or
Handle when node != ""), so add a nil-guard: either validate and return an error
or panic/fail-fast in NewPodFailureHandler if reader is nil, or (preferable)
keep constructor but have relatedStoreEvents and Handle check h.reader != nil
before calling reader methods and log a warning/degrade behavior when nil.
Update NewPodFailureHandler, PodFailureHandler.relatedStoreEvents (and Handle
where it uses reader) to perform this nil check and handle the nil-reader path
safely.

Comment on lines +151 to +162
evList, err := h.k8sClient.CoreV1().Events(in.Namespace).List(ctx, metav1.ListOptions{})
if err != nil {
return nil, fmt.Errorf("pod_failure: list events: %w", err)
}

out := make([]K8sEventSummary, 0)

for i := range evList.Items {
ev := &evList.Items[i]
if ev.InvolvedObject.Name != in.Pod || ev.InvolvedObject.Namespace != in.Namespace {
continue
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify current implementation lists namespace-wide events with empty ListOptions
# and does not filter by InvolvedObject.Kind.
rg -n -C3 'Events\(in\.Namespace\)\.List\(ctx,\s*metav1\.ListOptions\{\}\)' mcp-server/pkg/tools/pod_failure.go
rg -n -C3 'InvolvedObject\.Kind' mcp-server/pkg/tools/pod_failure.go

Repository: NVIDIA/NVSentinel

Length of output: 376


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Show relevant portion around listPodEvents
nl -ba mcp-server/pkg/tools/pod_failure.go | sed -n '120,220p'

# Confirm whether InvolvedObject.Kind is referenced anywhere in this file
rg -n 'InvolvedObject\.Kind' mcp-server/pkg/tools/pod_failure.go || true

# Confirm whether any FieldSelector is already used for Events List
rg -n 'FieldSelector' mcp-server/pkg/tools/pod_failure.go || true

Repository: NVIDIA/NVSentinel

Length of output: 104


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Show relevant portion around listPodEvents
nl -ba mcp-server/pkg/tools/pod_failure.go | sed -n '120,220p'

# Confirm whether InvolvedObject.Kind is referenced anywhere in this file
rg -n 'InvolvedObject\.Kind' mcp-server/pkg/tools/pod_failure.go || true

# Confirm whether any FieldSelector is already used for Events List
rg -n 'FieldSelector' mcp-server/pkg/tools/pod_failure.go || true

Repository: NVIDIA/NVSentinel

Length of output: 104


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Show relevant portion around listPodEvents (using cat -n since nl is unavailable)
cat -n mcp-server/pkg/tools/pod_failure.go | sed -n '120,220p'

# Confirm whether InvolvedObject.Kind is referenced anywhere in this file
rg -n 'InvolvedObject\.Kind' mcp-server/pkg/tools/pod_failure.go || true

# Confirm whether any FieldSelector is already used for Events List
rg -n 'FieldSelector' mcp-server/pkg/tools/pod_failure.go || true

Repository: NVIDIA/NVSentinel

Length of output: 3224


🏁 Script executed:

#!/bin/bash
set -euo pipefail

rg -n --hidden --no-ignore -S 'InvolvedObject\.Kind' mcp-server/pkg/tools/pod_failure.go || true
rg -n --hidden --no-ignore -S 'ListOptions\{[^}]*FieldSelector' mcp-server/pkg/tools/pod_failure.go || true

Repository: NVIDIA/NVSentinel

Length of output: 43


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Find imports block
sed -n '1,120p' mcp-server/pkg/tools/pod_failure.go

# Check for existing sort usage in this file
rg -n '\bsort\.' mcp-server/pkg/tools/pod_failure.go || true

# Find K8sEventSummary definition and check its fields for timestamp naming
rg -n 'type K8sEventSummary' -n mcp-server/pkg/tools/pod_failure.go
# show around definition
line=$(rg -n 'type K8sEventSummary' mcp-server/pkg/tools/pod_failure.go | head -n1 | cut -d: -f1)
start=$((line-20))
end=$((line+80))
sed -n "${start},${end}p" mcp-server/pkg/tools/pod_failure.go

Repository: NVIDIA/NVSentinel

Length of output: 8158


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Find imports block
sed -n '1,120p' mcp-server/pkg/tools/pod_failure.go

# Check for existing sort usage in this file
rg -n '\bsort\.' mcp-server/pkg/tools/pod_failure.go || true

# Find K8sEventSummary definition and check its fields for timestamp naming
rg -n 'type K8sEventSummary' mcp-server/pkg/tools/pod_failure.go
line=$(rg -n 'type K8sEventSummary' mcp-server/pkg/tools/pod_failure.go | head -n1 | cut -d: -f1)
start=$((line-20))
end=$((line+80))
sed -n "${start},${end}p" mcp-server/pkg/tools/pod_failure.go

Repository: NVIDIA/NVSentinel

Length of output: 8158


Filter Events server-side and constrain to Pod-kind events.

listPodEvents lists all namespace events via Events(in.Namespace).List(ctx, metav1.ListOptions{}) and then filters only by ev.InvolvedObject.Name/ev.InvolvedObject.Namespace; it does not enforce ev.InvolvedObject.Kind == "Pod" anywhere, so non-Pod events can slip through and the list call is needlessly expensive.

💡 Suggested fix
 import (
 	"context"
 	"errors"
 	"fmt"
+	"sort"
 	"strings"
 	"time"
@@
 func (h *PodFailureHandler) listPodEvents(ctx context.Context, in PodFailureInput) ([]K8sEventSummary, error) {
-	evList, err := h.k8sClient.CoreV1().Events(in.Namespace).List(ctx, metav1.ListOptions{})
+	evList, err := h.k8sClient.CoreV1().Events(in.Namespace).List(ctx, metav1.ListOptions{
+		FieldSelector: "involvedObject.kind=Pod,involvedObject.name=" + in.Pod,
+	})
 	if err != nil {
 		return nil, fmt.Errorf("pod_failure: list events: %w", err)
 	}
@@
 	for i := range evList.Items {
 		ev := &evList.Items[i]
-		if ev.InvolvedObject.Name != in.Pod || ev.InvolvedObject.Namespace != in.Namespace {
+		if ev.InvolvedObject.Kind != "Pod" ||
+			ev.InvolvedObject.Name != in.Pod ||
+			ev.InvolvedObject.Namespace != in.Namespace {
 			continue
 		}
@@
 	}
+	sort.Slice(out, func(i, j int) bool {
+		return out[i].LastTimestamp.After(out[j].LastTimestamp)
+	})
 
 	return out, nil
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
evList, err := h.k8sClient.CoreV1().Events(in.Namespace).List(ctx, metav1.ListOptions{})
if err != nil {
return nil, fmt.Errorf("pod_failure: list events: %w", err)
}
out := make([]K8sEventSummary, 0)
for i := range evList.Items {
ev := &evList.Items[i]
if ev.InvolvedObject.Name != in.Pod || ev.InvolvedObject.Namespace != in.Namespace {
continue
}
evList, err := h.k8sClient.CoreV1().Events(in.Namespace).List(ctx, metav1.ListOptions{
FieldSelector: "involvedObject.kind=Pod,involvedObject.name=" + in.Pod,
})
if err != nil {
return nil, fmt.Errorf("pod_failure: list events: %w", err)
}
out := make([]K8sEventSummary, 0)
for i := range evList.Items {
ev := &evList.Items[i]
if ev.InvolvedObject.Kind != "Pod" ||
ev.InvolvedObject.Name != in.Pod ||
ev.InvolvedObject.Namespace != in.Namespace {
continue
}
// ... rest of loop body
}
sort.Slice(out, func(i, j int) bool {
return out[i].LastTimestamp.After(out[j].LastTimestamp)
})
return out, nil
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@mcp-server/pkg/tools/pod_failure.go` around lines 151 - 162, The List call in
listPodEvents currently fetches all namespace events then filters client-side,
which is inefficient and misses enforcing ev.InvolvedObject.Kind == "Pod";
update the ListOptions in pod_failure.go where evList is fetched (the call using
h.k8sClient.CoreV1().Events(in.Namespace).List) to include a FieldSelector
restricting involvedObject.name=in.Pod, involvedObject.namespace=in.Namespace
and involvedObject.kind=Pod so the server returns only pod events; keep or
simplify the subsequent loop checks (ev.InvolvedObject.Name/Namespace/Kind) as a
safety guard but rely on the server-side selector to reduce load.

Comment thread mcp-server/README.md
Comment on lines +56 to +60
mcpServer:
authToken:
secretName: mcp-server-auth
secretKey: token
```

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Unify Helm values key notation across examples.

This README mixes mcpServer and mcp-server for value paths, which can mislead users during install/override. Please standardize to one canonical key style and use it consistently in both YAML overlays and --set examples.

Also applies to: 95-100

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@mcp-server/README.md` around lines 56 - 60, The README mixes two Helm values
key styles (`mcpServer` vs `mcp-server`) which is confusing; pick one canonical
form (e.g., `mcpServer`) and update all examples so they use that same key
consistently — update the YAML overlay block that currently shows `mcpServer:`
and any `--set` CLI examples that use `mcp-server` to the chosen form, and do
the same for the other occurrence noted around the later example so all
references to the mcp server values use the identical key name (`mcpServer` or
`mcp-server`) throughout the file.

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

@ArangoGutierrez this PR has been inactive for 14 days. Do you need help finishing it, or should we close it for now? Feel free to reopen anytime.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant