grafana-llmops-forge

Point it at any Grafana. Get a complete AI/LLM observability suite: discovered, generated, deployed, and visually verified.

English · Français

---

Your teams ship LLM features. Your CFO asks what they cost. Your board asks about the EU AI Act. Your SREs get paged about latency on a system nobody instrumented. And your Grafana, the tool you already trust, shows none of it.

grafana-llmops-forge fixes that with one prerequisite: a reachable Grafana (URL + service-account token). Everything else is discovered, not assumed.

Runs anywhere your platform already is. Prometheus, Thanos, Mimir, VictoriaMetrics, AWS AMP, Grafana Cloud or Kubernetes under the Prometheus Operator: the generated rules ship in both the portable format and a PrometheusRule manifest. Dashboards render in English by default, French with --locale fr. Governance evidence reads against the EU AI Act, ISO/IEC 42001 and NIST AI RMF; --framework picks which.

Not the same thing as Grafana Cloud AI Observability. Grafana shipped its own AI/agent observability (public preview, April 2026). It is excellent, but Cloud-only, and it asks you to adopt their SDK. This one works on self-hosted OSS, on whatever telemetry you already emit, and adds cost by provider origin and EU AI Act evidence. Provider origin does not establish processing or storage locations. See the FAQ.

export GRAFANA_URL=https://grafana.your-company.com GRAFANA_TOKEN=glsa_...

python3 scripts/discover.py --out capability_map.json          # ① what do you actually have?
python3 scripts/forge_dashboards.py \
        --capability capability_map.json \
        --blueprints auto --deploy --with-alerts               # ② forge + deploy + SLO alerts
python3 scripts/visual_audit.py --dashboards generated_dashboards  # ③ prove it renders right

Multi-org deployments can pin --org-id <id> on discovery, forge and visual audit. Use --uid-scope <tenant-or-folder> when deploying the same blueprints to multiple folders; omitting it preserves all historical UIDs. Discovery is fail-closed on datasource errors; --tolerate-datasource-errors only tolerates an unhealthy datasource when another one is healthy and records the error. Deployment writes a v2 manifest with success|partial|failed; --best-effort changes only the exit code. Visual audit likewise remains failed when an expected capture is absent, even when --allow-empty explicitly permits exit 0.

_{Illustration (hand-drawn SVG, not a screenshot) of the FinOps blueprint: cost composed from a 33-model price registry. Current panels label the regional split as provider origin, which does not establish processing or storage locations. For the real thing on real data, run make demo: it boots Grafana + Prometheus + a synthetic LLM workload and deploys these dashboards for real in about a minute.}

Why this is different

Most "LLM dashboards" are static JSON that assume your metric names. This is a forge:

1. Discovery-first, never assume. discover.py probes your datasources and captures the actual metric names present (OTel exporters disagree on suffixes: _seconds, _token, _total). Panels are only generated for queries that will return data. Missing signals become an instrumentation gap report with exact configs, not empty panels.
2. Four telemetry dialects, one mental model. OpenTelemetry GenAI (gen_ai_*), LiteLLM gateway (litellm_*, native USD spend), inference engines (vllm:*, tgi_*), GPU (DCGM_*). Each blueprint is translated into whatever you actually emit.
3. Cost is computed, not hoped for. Native gateway spend when available; otherwise PromQL composed by joining your token counters with a bundled 33-model price registry (US/EU/Asia, input/output/cached, tiered pricing), refreshable from official pricing pages when stale.
4. Governance is observable. The EU AI Act dashboard maps articles (12, 26§6, 50, 73) to live signals: logging evidence, retention posture, incident watch, an auto-built model inventory with provider-origin and GPAI flags, and the post-Digital-Omnibus timeline. A separate optional deployment inventory shows declared processing/storage locations; absent declarations remain unknown.
5. Verified by eye, not just by API. HTTP 200 proves the JSON was accepted, not that the render is right. After deploy, visual_audit.py captures every panel (native Grafana renderer, Playwright fallback) and an AI vision pass checks scale plausibility, "No data" panels, p50>p95 impossibilities, cross-panel coherence, then loops remediation (max 2 iterations, then a report of what remains).

_{Diagram, not a screenshot. Every box is a script in scripts/.}

The seven blueprints

Dashboard	Answers	Key panels
💰 Executive FinOps	What does AI cost, where, is it drifting?	spend/day, cost/request, provider-origin split 🇪🇺🇺🇸🌏, per-team spend, top models, unpriced-models watchlist
🛡 Gateway Operations	Are we meeting SLOs right now?	availability, p50/p95/p99, TTFT, errors by type, provider rate-limit headroom, $model variable
🤖 Agents & RAG	What do our agents do, where do they fail?	invoke/tool rates, per-tool errors, tokens per agent, embeddings latency, TraceQL panel (Tempo)
📈 Adoption	Who actually adopted what?	active teams, new adopters (7d), model mix over time, top token consumers (shadow AI shows up here)
⚡ Inference (self-hosted)	Do our GPUs hold, at what cost vs API?	vLLM TTFT/TPOT, queue, KV-cache saturation, preemptions, GPU util/VRAM, API-price benchmark table
✅ Quality & Evals	Is the output any good, and is it drifting?	eval score p50/p10, score per model, guardrail blocks, eval volume (a system can be green on latency and wrong on content)
⚖ Governance Evidence	What do we show an auditor?	the same telemetry read against EU AI Act, ISO/IEC 42001 and NIST AI RMF (--framework): regulatory timeline, logging evidence, auto model inventory, provider-origin split, declared deployment locations, incident watch

Blueprints only materialize when the underlying signals exist. No empty panels.

_{Diagram, not a screenshot. The measured panels are identical whichever framework you select; only the reading changes.}

Plus provisioned SLO alerts built the way SREs expect them: multi-window burn-rate on the error budget (5m/1h page, 30m/6h ticket, --slo-target), telemetry-signal-lost (with noDataState: Alerting, because the alert that catches a dead pipeline must not go quiet when the pipeline dies), daily budget breach, TTFT p95, vLLM KV-cache saturation, and eval-score drop.

Cost that scales

Financial selection is independent of the operational dashboards: --cost-mode auto prefers the exact total metric llm:cost_usd_per_second, then native LiteLLM spend, then OTel tokens × registry prices. Candidates are resolved across all Prometheus datasources. If several sources have the same priority, financial generation/deployment stops before remote writes; pass --datasource <UID|unique name> to forge or repeat discovery with that selector. --cost-mode inline ignores recorded totals but retains native-spend priority. --cost-mode recorded requires the discovered total; :input/:output components alone are insufficient.

FinOps and its budget alert use the selected datasource. A cost/request ratio is omitted when that datasource lacks a request signal. Native spend is not an audited invoice, and recorded costs retain their upstream provenance; neither is attributed to the local price registry or Artificial Analysis. Missing native/recorded monetary series stay absent, and the budget alert uses Grafana's NoData state to report unknown cost. The dashboard description and deployment manifest expose the selected UID, mode and status; discovery does not establish current data availability.

Discovery preserves every metric name, model and provider value returned by the backend, with deterministic ordering and counts in discovery_coverage. This establishes no local truncation, not backend exhaustiveness: backend warnings, time scope and pagination are not certified. Inline estimates expose fully priced, partially priced and unpriced model counts in FinOps, CLI diagnostics and financial_source.coverage. Missing prices make the amount an explicit subtotal and omit the budget alert. Older maps without consistent coverage metadata still produce estimates limited to listed models; re-run discover.py before enabling their budget. Even an eligible inline budget only covers returned models. A valid zero price counts as priced.

Inline composition is capped at 40 models with usable prices. Above that limit, no monetary subtotal or budget is generated: select the financial datasource with --datasource, generate and load its recording rules, then repeat discovery and forge. All priced models supplied to the rule generator are retained, including beyond 60 models. Recorded queries are shorter, but execution cost still depends on series count and time window. Native and recorded sources retain unverified upstream coverage and do not depend on registry coverage.

With --deploy --with-alerts, an ineligible budget also pauses an existing Forge budget in the resolved organization and folder. Forge checks its deterministic UID, rule group and ownership labels before updating only isPaused; an absent budget is not created. The manifest records budget_alert.action, result (paused, already_paused, absent or failed) and the coverage reason. Alert resource counts include this maintenance operation, including confirmed absence or an already paused rule. A collision or unconfirmed HTTP response makes the deployment partial/failed. Generation, dry-run and deployments without --with-alerts do not perform this check. When coverage becomes eligible again, Forge updates the budget while preserving its existing pause (upserted_paused): review the coverage and resume it explicitly in Grafana. Other SLO pause behavior is unchanged.

Provider origin and declared deployment locations

The registry field and Prometheus label region remain compatible and describe provider origin. They establish neither processing nor storage location. Governance always includes a separate deployment panel: locations default to unknown. Optionally pass --deployment-inventory ../instance/deployment_inventory.json, keeping real inventory outside the packaged skill. A US-origin model can have declared processing in France; both facts are shown separately.

The local JSON format is strict version 1, limited to 1 MiB / 500 records. Each deployment has its own unique ID; several deployments may use the same model. Exact model matches only mean observed_in_capability_map. Locations, endpoints, evidence references and dates remain declarations without independent checks. References are plain text, never links or fetched documents. Validation occurs before price fallback, cache writes or Grafana access, including dry runs. An inventory can cover several datasources: validate against the full capability map, then render only the --datasource selection.

See the field contract and local-path restrictions and synthetic example. The deployment manifest records scope and bounded declaration counts, including records_with_observed_model; it does not copy endpoint hosts or evidence references.

OTel pricing and generated rules remain independent of that financial choice. When discovery finds OTel GenAI signals, token counters, a model label, and usable prices, the forge writes prometheus_rules_llmops.yml: prices become series (llm:price_input_usd_per_token{model=...}) and cost becomes one recorded metric joined by vector matching. An explicitly requested Artificial Analysis fallback can still price those OTel artifacts while FinOps displays native spend. The manifest's recording_rules.datasource_uid identifies their target separately from financial_source.datasource_uid. Load the rules into that Prometheus and repeat discovery to make the recorded total a candidate. Without rules or native spend, on-the-fly composition is useful to bootstrap and becomes expensive past ~15 models.

_{Conceptual cost flow for native gateway spend and OTel GenAI token pricing. Not a Grafana screenshot.}

Try it in 60 seconds (no Grafana needed)

make demo    # Grafana + Prometheus + a synthetic LLM metrics emitter, then the full pipeline

Spins up the stack, discovers it, forges and deploys all applicable dashboards with alerts, loads the generated cost recording rules, and prints the URLs (admin/admin). make shots captures every panel; make demo-down removes everything. This is also the integration test: it runs the exact code path a production instance would.

Quick start

As an Agent Skill (Claude, or any agentskills.io-compatible agent)

Drop the folder into your skills directory, or download the packaged .skill from Releases; it is built by CI from these sources and published with its checksum, never committed as a binary. Then just talk:

"Audit my Grafana at https://grafana.internal and deploy whatever makes sense, then prove it visually."

The skill handles discovery → registry refresh → blueprint selection → deploy → vision-verified loop, and reports gaps with exact instrumentation configs.

As a standalone CLI (no AI required)

Pure Python 3.9+ stdlib. No pip install. The three commands at the top of this README are the whole workflow. --dry-run writes JSON without touching your instance; --selftest renders all seven blueprints offline from a simulated capability map.

Unknown or officially unpriced detected models can use an explicit third-party fallback:

export ARTIFICIAL_ANALYSIS_API_KEY="..."
python3 scripts/forge_dashboards.py --capability capability_map.json \
  --pricing-fallback artificial-analysis --dry-run

It is off by default. Only the fixed Artificial Analysis Free endpoint is called, only for detected models that need a price, and only the ARTIFICIAL_ANALYSIS_API_KEY environment variable is accepted. The key is not logged or written. Results are median multi-provider estimates, marked and attributed in dashboards and recording-rule labels, with a 24-hour atomic overlay in model_registry.artificial-analysis.cache.json next to the capability map. It never contains the merged registry. Official pricing always wins. See pricing provenance.

No LLM telemetry yet?

Run discovery anyway. You'll get a prioritized gap report, and references/instrumentation_guide.md contains copy-paste configs ordered by value/effort: LiteLLM gateway (~30 min → native USD spend) → OTel GenAI SDK setup → vLLM/TGI scrape → dcgm-exporter → Loki retention for AI-Act evidence.

How this is verified

Every defect found in this project so far has been silent: a model billed at another model's price, a regex that parsed everywhere except against the real engine, an or that dropped half the cost, an alert that stayed quiet exactly when its subject failed, a datasource setting that skewed every rate(). None crashed anything. All produced a plausible, wrong dashboard. That is the failure mode that matters in observability, and the reason for these test layers:

Layer	What it can catch	Where
Offline harness, four simulated instance topologies	structure, regressions, escaping, cardinality, packaging, supply chain	tests/audit_harness.py
Live query check against a real Prometheus	queries that are valid but return nothing, or error at query time	tests/live_query_check.py
Value invariants	numbers that exist but disagree with each other: ordered quantiles, bounded ratios, cost paths converging	tests/value_invariants.py
End-to-end deploy on the demo stack	the full pipeline against a real Grafana	CI e2e job
Vision pass on rendered panels	what only an eye sees: empty panels, implausible scales	scripts/visual_audit.py

All of it runs in CI on every push. The forge runtime makes no third-party calls by default; CI downloads pinned test tools and container images.

What it touches, and how to back it out

The question a change advisory board asks first. The answer is deliberately narrow, and the narrowness is enforced rather than promised: tests/audit_harness.py fails the build if a write endpoint outside this list ever appears.

Operation	Where	Reversible by
Create a folder	AI Observability, one folder, nothing else	deleting that folder
Upsert dashboards	inside that folder, deterministic UIDs, overwrite: true	deleting that folder
Provision alert rules	inside that folder, UIDs prefixed alr-	deleting that folder
Read	datasource metadata, metric and label names via the datasource proxy	nothing to reverse

The tool has no delete path. It never removes a dashboard, a datasource, a rule or a folder, by design, so a misfire cannot destroy anything you already had. Reversal is therefore an action you take, and it is a single one:

# Everything the forge created lives in one folder. Remove it and the deployment
# is gone: dashboards, alert rules and all.
curl -X DELETE -H "Authorization: Bearer $GRAFANA_TOKEN" \
     "$GRAFANA_URL/api/folders/$(curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
     "$GRAFANA_URL/api/folders" | python3 -c \
     "import sys,json;print(next(f['uid'] for f in json.load(sys.stdin) if f['title']=='AI Observability'))")"

Or delete it from the UI: Dashboards → AI Observability → Folder settings → Delete. Generated recording rules are separate: they are a file you copied into Prometheus, so removing them is removing that file and reloading.

What leaves your network: nothing by default. The scripts talk only to your Grafana, which talks to your own Prometheus, Loki and Tempo. The explicit --pricing-fallback artificial-analysis exception sends authenticated GET requests to https://artificialanalysis.ai/api/v2/language/models/free; it can be skipped entirely. No Grafana data, including detected model names, is sent in the request: model matching happens locally against the unfiltered paginated catalog. The bundled registry works offline.

What an agent sees. Used as a skill, Claude reads the capability map, which contains metric names, model names, and team or service label values from your instance. That is organisational metadata entering an LLM context; if your policy forbids it, run the scripts as a plain CLI; they are self-contained and need no model at all.

🔒 Security model

Skills execute code, and a 2026 Snyk audit found 36% of published skills had at least one flaw. This repo is designed to be auditable in one sitting:

• Zero dependencies. Python 3.9+ stdlib only (urllib, json, hashlib). Playwright is optional, only for the visual-audit fallback.
• Least privilege. Works with an Editor service-account token. Alert provisioning degrades gracefully on 403 (exports JSON for manual import).
• No secret leakage. Grafana credentials and the optional Artificial Analysis key are never logged, embedded in dashboards, persisted, or placed in URLs.
• No prompt-content capture. gen_ai.input/output.messages stay off by default; the docs treat enabling them as a GDPR decision, not a flag.
• Idempotent & reversible. Deterministic UIDs, one folder, overwrite semantics: delete the folder and it's gone.
• Offline-testable. --selftest + tests/audit_harness.py (4 simulated instance topologies plus regression tests) runs with zero network, tests/live_query_check.py executes every generated query against a real Prometheus, and the demo stack gives a full end-to-end deploy. That's the CI; the badge above is the live workflow status.

Language

Dashboards, alert rules and recording rules are generated in English. --locale fr renders them in French; other languages are a JSON file in references/locale.<code>.json away, with no change to the blueprint code.

Repo layout

SKILL.md                      # agent playbook (7-phase pipeline, doctrine, pitfalls)
scripts/
  grafana_client.py           # universal client: OSS/Cloud/Enterprise, legacy + K8s-style APIs
  discover.py                 # capability map: real metric names, dialects, gaps
  forge_dashboards.py         # 7 blueprints × detected dialect, cost engine, alerts
  visual_audit.py             # render/Playwright capture + DOM pre-scan for vision review
references/
  model_registry.json         # 30+ models: $/1M in·out·cached, context, provider origin, GPAI
  locale.fr.json              # label translations (--locale fr)
  ai_governance_frameworks.md # crosswalk: EU AI Act × ISO/IEC 42001 × NIST AI RMF
  query_library.md            # PromQL/LogQL/TraceQL per dialect, anti-patterns
  dashboard_blueprints.md     # panel-by-panel specs + optional extensions
  instrumentation_guide.md    # exact configs to close each gap
  eu_ai_act_observability.md  # article → signal → panel mapping, deployer checklist
  visual_verification.md      # vision checklist, failure signatures → fixes
  grafana_api_compat.md       # 3 API generations, editions matrix, schema v2 notes
demo/                         # docker-compose stack: Grafana + Prometheus + synthetic LLM emitter
tools/package.py              # builds the .skill reproducibly and verifies it matches sources
tests/live_query_check.py     # runs every generated query against a real Prometheus
tests/value_invariants.py     # checks the numbers agree with each other, not just exist
tests/audit_harness.py        # offline checks across 4 instance topologies + regressions

FAQ

Why not Grafana Cloud's own AI Observability? Use it if you're on Cloud and happy to instrument with Grafana's SDK, which does evaluations and conversation replay well. This project targets the other case: self-hosted OSS/Enterprise, telemetry you already emit (OTel, LiteLLM, vLLM, no SDK migration), plus cost attribution by provider origin and an EU AI Act evidence layer, which no vendor ships. Declared deployment locations are shown independently of provider origin. They compose fine: nothing here conflicts with the Grafana plugins.

Does it overwrite my existing dashboards? No. Everything lives in its own folder with llmops-forge-tagged, deterministically-UID'd dashboards. Re-running updates in place.

Grafana Cloud? Yes. Cloud is auto-detected; the image renderer is built in, so visual audit works out of the box.

My models aren't in the registry. They appear in an "unpriced models" panel instead of being billed wrong. Add a price or alias to model_registry.json, or explicitly opt in to --pricing-fallback artificial-analysis. The fallback accepts only a unique exact normalized ID, slug, name, or alias match. Ambiguous, absent, null, or failed responses stay unpriced.

Can I publish the generated dashboards? Yes: --export-portable emits JSON with __inputs/${DS_PROMETHEUS} placeholders, the format grafana.com/dashboards requires.

Multiple Prometheus (prod + staging)? Financial sources at the same priority are ambiguous: the forge refuses a silent choice. Pin discovery or forge with --datasource <UID|unique name>; costs from separate datasources are never added together.

Multiple orgs or folders? Pin the Grafana organization with --org-id and give each independently managed folder a stable --uid-scope. The forge refuses to overwrite an existing dashboard UID found in another folder.

Is the governance dashboard legal advice? No, and it says so on the dashboard. It is the evidence layer your counsel or auditor will ask for: the logging continuity, model inventory and incident trail that ISO 42001 Stage 2, NIST MANAGE 4.1 and AI Act Art. 12 all want to see. It proves monitoring happened; it does not prove a management system exists, and references/ai_governance_frameworks.md is explicit about both halves.

Roadmap

Make every AI change a decision you can defend. The proposed direction is reproducible operational decisions for European enterprises on self-hosted Grafana, using their existing telemetry and evaluation tools.

These are planned horizons, with evidence gates rather than release dates:

• H0 · Trusted foundations: interpretable measurements, explicit gaps and read-only configuration diffs.
• H1 · Compare a change: cost per successful task at explicit quality and latency criteria, including a reasoned refusal when evidence is insufficient.
• H2 · Ship with evidence: portable promotion dossiers and checks consumed by the customer's existing CI and provisioning tools.
• H3 · Rehearse the next migration: executed migration tests or clearly labeled sensitivity scenarios, after the earlier workflow proves useful.

As of 2026-09-14, v2.0.2 is the released baseline; the five foundational corrections and budget-alert lifecycle fix in PR #5 are integrated into main but not yet included in a release. The proposed comparison and dossier workflows are future work.

Read the canonical roadmap or its French translation for the future demonstration, scope and pass/fail criteria.

Support

Issues and discussions are the place for bugs and questions; I read everything. If your organisation needs this wired into a real platform (multi-tenant Grafana, AI Act evidence pack for an audit, FinOps governance across business units), I do that professionally: ia-b2b.fr.

Contributing

Model prices drift quarterly, so registry PRs are the most valuable contribution and take 2 minutes (guide). Dialect additions (new gateway/engine signatures) are second. python3 tests/audit_harness.py must stay green.

_{Built with the Agent Skills open standard · works in Claude Code, Claude.ai, Cowork, and as a plain CLI.
If this saved your platform team some time, a ⭐ helps others find it.}