kube-investigator (binary: kinv) is a CLI tool that collects diagnostic data from a Kubernetes cluster and uses an LLM (OpenAI-compatible API) to analyze pod failures and produce structured incident reports with root cause analysis, timelines, and remediation commands.
The tool is organized into five internal layers that process data in a sequential pipeline:
CLI (Cobra) --> Collectors --> Context Builder --> LLM Analyzer --> Report Renderer
|
History Store
-
CLI Layer (
cmd/): Cobra-based command tree with subcommands for investigation, history management, and a real-time watcher. Handles flag parsing, kubeconfig setup, and API key resolution from environment variables. -
Collectors (
internal/collector/): Five independent data collectors that each implement aCollectorinterface:- Pod Collector -- Fetches the pod spec, status, and container logs (configurable tail lines)
- Event Collector -- Fetches Kubernetes events for the pod, filtered by a time window and sorted by timestamp
- Node Collector -- Fetches the hosting node's conditions, resources, and container images
- Dependency Collector -- Fetches related Services, EndpointSlices, and PVCs
- Metrics Collector -- Optionally queries a Prometheus API for memory, CPU throttling, restart rate, and CPU usage metrics
-
Context Builder (
internal/context/): Aggregates data from all collectors into a singleIncidentContextstruct. Pod collection is fatal; all other collectors append non-fatal errors to an error list. -
Analyzer (
internal/analyzer/): Formats the incident context into a prompt using Go templates and sends it to an OpenAI-compatible chat completions API. The LLM returns a JSON response matching theIncidentReportschema, which includes root cause, confidence score, incident type, timeline, immediate actions (with kubectl commands), and prevention steps. -
Report Renderer (
internal/report/): Renders the structured report as formatted markdown to stdout. JSON rendering is also available but currently unused in command handlers (all paths render markdown). -
History Store (
internal/storage/): Persists investigation records to a JSON file at~/.kubeinvestigator/history.jsonwith atomic writes. Supports save, list, and get-by-ID operations. -
Watcher (
internal/watcher/): Uses Kubernetes informers (watch-based, no polling) to monitor pods in a namespace. When a pod enters a trigger state (defaults: CrashLoopBackOff, OOMKilled), it automatically initiates an investigation. Includes a 5-minute debounce per pod to prevent duplicate investigations.
When you run kinv investigate pod --name <pod>:
- The tool connects to your Kubernetes cluster using your kubeconfig (defaults to
~/.kube/config). - It fetches the pod object and its container logs.
- It collects related events, node data, services, endpoint slices, and PVCs.
- If a Prometheus URL is provided, it gathers memory and CPU metrics.
- All collected data is assembled into a structured context.
- The context is sent to the configured LLM API with a system prompt instructing it to act as an SRE and return a JSON report.
- The structured report is saved to local history and printed as a human-readable markdown report covering root cause, timeline, and remediation actions.
The kinv watch command runs continuously, using Kubernetes informers to detect failures as they happen and trigger investigations automatically.
- Go 1.26 or later
- Access to a Kubernetes cluster (kubeconfig)
- An OpenAI-compatible API key
git clone https://github.com/debjotimallick/kube-investigator.git
cd kube-investigator
make buildThis produces the kinv binary in the project root.
The tool accepts configuration through CLI flags and environment variables:
| Variable | Flag | Description | Default |
|---|---|---|---|
OPENAI_API_KEY |
-- | API key for the LLM provider | required |
OPENAI_MODEL |
-- | Model name to use | gpt-4o-mini |
OPENAI_API_ENDPOINT |
-- | API endpoint URL | https://api.openai.com/v1/chat/completions |
| -- | --kubeconfig |
Path to kubeconfig | ~/.kube/config |
| -- | -n, --namespace |
Target namespace | default |
| -- | --prometheus |
Prometheus HTTP API URL | (optional) |
| -- | --tail-lines |
Log lines per container | 100 |
OPENAI_API_KEY is required. The endpoint can point to any OpenAI-compatible API (Azure OpenAI, local LLMs with compatible APIs, etc.).
Investigate a specific pod:
export OPENAI_API_KEY="sk-..."
kinv investigate pod --name my-pod -n productionScan and investigate all failing pods in a namespace:
kinv investigate all -n production --since 1hList past investigations:
kinv history listView a past investigation:
kinv history show <id>Watch for pod failures and investigate automatically:
kinv watch -n production --trigger CrashLoopBackOff,OOMKilledThe watch mode runs continuously and initiates investigations when pods enter the specified failure states.
├── main.go Entry point
├── cmd/ CLI commands (root, investigate, history, watch, kube helpers)
├── internal/
│ ├── collector/ Kubernetes data collectors (pod, node, events, metrics, dependencies)
│ ├── context/ Context builder that assembles collector output
│ ├── analyzer/ LLM client, prompt templates, and report schema
│ ├── report/ Markdown and JSON report renderers
│ ├── storage/ History persistence (JSON file store)
│ └── watcher/ Real-time pod failure watcher using informers
├── Makefile Build automation
└── go.mod / go.sum Go module definitions and dependencies
- spf13/cobra -- CLI framework
- spf13/viper -- Configuration management
- k8s.io/client-go -- Kubernetes API client
- prometheus/client_golang -- Prometheus HTTP API client
- google/uuid -- UUID generation for history records
- No test files are present in the project
- No CI/CD or Docker build configuration is included
- Prometheus metrics collection is optional and silently skipped when no URL is configured