Skip to content

Commit 58ee18d

Browse files
Alexandre Beguelclaude
Alexandre Beguel
and
claude
committed
Initial public release — Grafana LLMOps Forge v0.1.0
Discovery-first, zero-dependency toolkit + Claude Code Skill that turns any Grafana into an AI/LLM observability command center: auto-discovers telemetry (OTel gen_ai, LiteLLM, vLLM/TGI/Ollama, DCGM), forges 6 dashboards (FinOps, gateway, agents & RAG, adoption, self-hosted inference, EU AI Act governance) with SLO alerts, and vision-verifies the render. Python stdlib only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
0 parents  commit 58ee18d

35 files changed

Lines changed: 3758 additions & 0 deletions

‎.gitattributes‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Normalize line endings: LF in the repo for all text, regardless of OS.
2+
* text=auto eol=lf
3+
4+
*.py text eol=lf
5+
*.md text eol=lf
6+
*.json text eol=lf
7+
*.yml text eol=lf
8+
*.yaml text eol=lf
9+
*.cff text eol=lf
10+
*.txt text eol=lf
11+
12+
# Binary assets — never touch line endings, exclude from diffs.
13+
*.png binary
14+
*.jpg binary
15+
*.gif binary
16+
*.webp binary

‎.github/ISSUE_TEMPLATE/bug_report.yml‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
name: "🐛 Bug report"
2+
description: A panel didn't render, a query failed, or the forge misbehaved.
3+
labels: ["bug"]
4+
body:
5+
- type: markdown
6+
attributes:
7+
value: |
8+
Thanks for the report. Please **never paste your Grafana token, prompt
9+
contents, or a screenshot that reveals real costs/teams.** Redact first.
10+
- type: textarea
11+
id: what-happened
12+
attributes:
13+
label: What happened?
14+
description: What did you expect, and what did you get instead?
15+
placeholder: "The finops dashboard shows 'No data' on the cost panels, but discovery detected litellm."
16+
validations:
17+
required: true
18+
- type: textarea
19+
id: repro
20+
attributes:
21+
label: Steps to reproduce
22+
description: The exact commands you ran (redact secrets).
23+
render: shell
24+
placeholder: |
25+
python3 scripts/discover.py --out capability_map.json
26+
python3 scripts/forge_dashboards.py --capability capability_map.json --blueprints finops --dry-run
27+
validations:
28+
required: true
29+
- type: input
30+
id: grafana
31+
attributes:
32+
label: Grafana version & edition
33+
placeholder: "11.2.0 · OSS (self-hosted)"
34+
validations:
35+
required: true
36+
- type: input
37+
id: python
38+
attributes:
39+
label: Python version & OS
40+
placeholder: "3.11 · Ubuntu 24.04"
41+
validations:
42+
required: true
43+
- type: dropdown
44+
id: dialect
45+
attributes:
46+
label: Telemetry dialect(s) involved
47+
multiple: true
48+
options:
49+
- OpenTelemetry GenAI (gen_ai.*)
50+
- LiteLLM
51+
- vLLM / TGI / Ollama
52+
- NVIDIA DCGM (GPU)
53+
- None / not sure
54+
- type: checkboxes
55+
id: checks
56+
attributes:
57+
label: Sanity checks
58+
options:
59+
- label: "`python3 scripts/forge_dashboards.py --selftest` passes on my machine."
60+
- label: I have redacted all secrets and sensitive values from this report.

‎.github/ISSUE_TEMPLATE/config.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
blank_issues_enabled: true
2+
contact_links:
3+
- name: 💬 Questions & ideas (Discussions)
4+
url: https://github.com/alebgl77/grafana-llmops-forge/discussions
5+
about: Ask how to instrument a stack, share a dashboard, or propose a direction.
6+
- name: 🔒 Report a security issue (private)
7+
url: https://github.com/alebgl77/grafana-llmops-forge/security/advisories/new
8+
about: Please disclose vulnerabilities privately, not in a public issue.

‎.github/ISSUE_TEMPLATE/feature_request.yml‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
name: "✨ Feature request"
2+
description: A new blueprint, dialect, query, provider price, or capability.
3+
labels: ["enhancement"]
4+
body:
5+
- type: textarea
6+
id: problem
7+
attributes:
8+
label: What problem are you trying to solve?
9+
description: The observability / FinOps / governance question you can't answer today.
10+
validations:
11+
required: true
12+
- type: textarea
13+
id: proposal
14+
attributes:
15+
label: Proposed solution
16+
description: A new panel, a dialect to detect, a provider to price, an alert rule…
17+
validations:
18+
required: true
19+
- type: dropdown
20+
id: area
21+
attributes:
22+
label: Area
23+
options:
24+
- New / extended blueprint
25+
- New telemetry dialect
26+
- Model pricing registry
27+
- Query library (PromQL / LogQL / TraceQL)
28+
- EU AI Act governance
29+
- Grafana API / compatibility
30+
- Docs / translation
31+
- Other
32+
validations:
33+
required: true
34+
- type: checkboxes
35+
id: constraints
36+
attributes:
37+
label: Design constraints (this project is opinionated)
38+
options:
39+
- label: My proposal keeps the zero-dependency (stdlib-only) rule.
40+
- label: My proposal is discovery-first (keys off real metric names, no hard-coded assumptions).

‎.github/PULL_REQUEST_TEMPLATE.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
<!-- Thanks for contributing! Keep PRs focused; describe the *why*. -->
2+
3+
## What & why
4+
5+
<!-- What does this change, and what problem does it solve? -->
6+
7+
## Type
8+
9+
- [ ] Bug fix
10+
- [ ] New / extended blueprint or dialect
11+
- [ ] Model pricing registry update
12+
- [ ] Query library / docs
13+
- [ ] Other
14+
15+
## Checklist
16+
17+
- [ ] `python3 scripts/forge_dashboards.py --selftest` passes.
18+
- [ ] `python3 audit_harness.py` passes.
19+
- [ ] **No new third-party dependency** (stdlib only).
20+
- [ ] New metric usage flows from the capability map — no hard-coded metric names.
21+
- [ ] Any regulatory / pricing claim is date-stamped and sourced.

‎.github/workflows/ci.yml‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
name: CI
2+
3+
# Zero-dependency by design: no `pip install`, no external services.
4+
# The self-test and the audit harness generate all six dashboards from a
5+
# simulated capability map and assert the invariants — offline, on every OS.
6+
7+
on:
8+
push:
9+
branches: [main]
10+
pull_request:
11+
branches: [main]
12+
workflow_dispatch:
13+
14+
permissions:
15+
contents: read
16+
17+
jobs:
18+
selftest:
19+
name: selftest & audit (${{ matrix.os }} · py${{ matrix.python-version }})
20+
runs-on: ${{ matrix.os }}
21+
strategy:
22+
fail-fast: false
23+
matrix:
24+
os: [ubuntu-latest, windows-latest, macos-latest]
25+
python-version: ['3.8', '3.11', '3.13']
26+
exclude:
27+
# 3.8 is not available on the macos-latest (arm64) runners.
28+
- os: macos-latest
29+
python-version: '3.8'
30+
steps:
31+
- uses: actions/checkout@v4
32+
33+
- name: Set up Python ${{ matrix.python-version }}
34+
uses: actions/setup-python@v5
35+
with:
36+
python-version: ${{ matrix.python-version }}
37+
38+
- name: Confirm zero third-party dependencies
39+
shell: bash
40+
run: |
41+
test ! -f requirements.txt || (echo "Unexpected requirements.txt" && exit 1)
42+
echo "stdlib-only: no dependencies to install."
43+
44+
- name: Offline self-test (generate the 6 blueprints)
45+
run: python scripts/forge_dashboards.py --selftest
46+
47+
- name: Full audit harness (multi-topology invariants)
48+
run: python audit_harness.py
49+
50+
- name: Byte-compile all sources
51+
run: python -m compileall -q scripts audit_harness.py

‎.gitignore‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# --- grafana-llmops-forge : artefacts d'exécution (never commit) ---
2+
# Runtime outputs — may contain real costs, teams, model names. Keep them local.
3+
capability_map.json
4+
model_registry.local.json
5+
deploy_manifest.json
6+
selftest_output/
7+
generated_dashboards/
8+
visual_audit/
9+
*.audit.png
10+
11+
# --- Python ---
12+
__pycache__/
13+
*.py[cod]
14+
*$py.class
15+
*.egg-info/
16+
.eggs/
17+
build/
18+
dist/
19+
.pytest_cache/
20+
.mypy_cache/
21+
.ruff_cache/
22+
.venv/
23+
venv/
24+
env/
25+
26+
# --- Secrets / env (the token must NEVER be committed) ---
27+
.env
28+
.env.*
29+
*.local
30+
31+
# --- Local working docs (private growth/strategy, not for publication) ---
32+
LAUNCH_KIT.md
33+
launch_brief.md
34+
35+
# --- OS / editor ---
36+
.DS_Store
37+
Thumbs.db
38+
desktop.ini
39+
.idea/
40+
.vscode/
41+
*.swp
42+
*~

‎CHANGELOG.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Changelog
2+
3+
All notable changes to this project are documented here. The format is based on
4+
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project aims
5+
to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
6+
7+
## [Unreleased]
8+
9+
### Added
10+
- Cross-platform CI (Linux/macOS/Windows · Python 3.8 → 3.13) running the
11+
offline self-test and the multi-topology audit harness.
12+
13+
## [0.1.0] — 2026-07-12
14+
15+
First public release.
16+
17+
### Added
18+
- **Auto-discovery** (`discover.py`): probes a Grafana instance and produces a
19+
capability map — version/edition, datasources, and the LLM telemetry dialects
20+
actually present, with the **real metric names** (OpenTelemetry GenAI
21+
`gen_ai.*`, LiteLLM, vLLM/TGI/Ollama, NVIDIA DCGM).
22+
- **Six dashboard blueprints** (`forge_dashboards.py`): FinOps & cost, Gateway
23+
Operations, Agents & RAG, internal Adoption, self-hosted Inference, and EU AI
24+
Act Governance. Classic schema v41 via the legacy API — compatible with
25+
Grafana 9 → 13+, OSS/Cloud/Enterprise.
26+
- **Cost computed, not hoped**: an embedded, date-stamped pricing registry of
27+
~30 models (input/output/cache, US/EU/Asia sovereignty) composed into PromQL,
28+
with a 30-day refresh protocol.
29+
- **SLO alerting** (`--with-alerts`): error burn-rate, TTFT p95, daily budget,
30+
KV-cache saturation, and signal-absence rules.
31+
- **Graceful degradation**: zero LLM signal produces an instrumentation-gap
32+
report plus a working governance dashboard, not a failure.
33+
- **Idempotence**: deterministic UIDs, upsert with overwrite, and a `--dry-run`
34+
mode for everything that writes.
35+
- **Vision-verified rendering** (`visual_audit.py`): native Grafana renderer or
36+
Playwright fallback captures the real render for inspection.
37+
- **Offline self-test** (`--selftest`) and a portable multi-topology audit
38+
harness (`audit_harness.py`).
39+
- Reference library: query recipes, blueprint specs, Grafana API compatibility
40+
matrix, instrumentation guide, and an EU AI Act × observability mapping
41+
(verified July 2026, post-Digital Omnibus).
42+
- Packaged as a Claude Code / Agent **Skill** (`SKILL.md`).
43+
44+
[Unreleased]: https://github.com/alebgl77/grafana-llmops-forge/compare/v0.1.0...HEAD
45+
[0.1.0]: https://github.com/alebgl77/grafana-llmops-forge/releases/tag/v0.1.0

‎CITATION.cff‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
cff-version: 1.2.0
2+
message: "If you use Grafana LLMOps Forge in your work, please cite it as below."
3+
title: "Grafana LLMOps Forge"
4+
abstract: >-
5+
A zero-dependency Python toolkit and Claude Code Skill that turns any Grafana
6+
instance into an AI/LLM observability command center: discovery-first
7+
auto-generation of FinOps, gateway, agent/RAG, adoption, self-hosted inference
8+
and EU AI Act governance dashboards, with SLO alerting and vision-verified
9+
rendering.
10+
type: software
11+
authors:
12+
- family-names: Beguel
13+
given-names: Alexandre
14+
license: MIT
15+
repository-code: "https://github.com/alebgl77/grafana-llmops-forge"
16+
version: 0.1.0
17+
date-released: "2026-07-12"
18+
keywords:
19+
- llmops
20+
- observability
21+
- grafana
22+
- opentelemetry
23+
- finops
24+
- eu-ai-act
25+
- prometheus
26+
- genai

‎CODE_OF_CONDUCT.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Code of Conduct
2+
3+
## Our pledge
4+
5+
We — contributors and maintainers — pledge to make participation in this project
6+
a harassment-free experience for everyone, regardless of age, body size,
7+
visible or invisible disability, ethnicity, sex characteristics, gender identity
8+
and expression, level of experience, education, socio-economic status,
9+
nationality, personal appearance, race, religion, or sexual identity and
10+
orientation.
11+
12+
## Our standards
13+
14+
Examples of behavior that contributes to a positive environment:
15+
16+
- Being respectful of differing viewpoints and experiences.
17+
- Giving and gracefully accepting constructive feedback.
18+
- Focusing on what is best for the community and the project.
19+
- Showing empathy toward other community members.
20+
21+
Examples of unacceptable behavior:
22+
23+
- Harassment, insulting or derogatory comments, and personal or political attacks.
24+
- Publishing others' private information without explicit permission.
25+
- Trolling, deliberate intimidation, or other conduct which could reasonably be
26+
considered inappropriate in a professional setting.
27+
28+
## Enforcement
29+
30+
Instances of abusive, harassing, or otherwise unacceptable behavior may be
31+
reported to the maintainer via the email on the GitHub profile, or through a
32+
private GitHub report. All complaints will be reviewed and investigated
33+
promptly and fairly. Maintainers are obligated to respect the privacy and
34+
security of the reporter of any incident.
35+
36+
## Attribution
37+
38+
This Code of Conduct is adapted from the
39+
[Contributor Covenant](https://www.contributor-covenant.org), version 2.1.

0 commit comments

Comments
 (0)