Skip to content

Commit 69846d2

Browse files
committed
feat: release v2.0.0
1 parent fde7391 commit 69846d2

13 files changed

Lines changed: 1636 additions & 62 deletions

CHANGELOG.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,12 @@
11
# Changelog
22

3+
## [2.0.0] - 2026-09-01
4+
### Changed : chaîne de déploiement et de preuve v2
5+
- **Rupture de compatibilité : manifests v2 et portée organisationnelle stricte.** Les artefacts de déploiement portent désormais leur contrat de version, l'organisation cible est vérifiée sans repli implicite, et un garde-fou bloque les collisions avant toute écriture.
6+
- **Audit visuel fail-closed et Playwright durci.** Une preuve visuelle manquante ou invalide fait échouer l'audit ; l'exécution du navigateur applique les limites et contrôles de sécurité attendus.
7+
- **Release reproductible avec SBOM.** La chaîne de publication produit et vérifie ses artefacts de supply chain, tandis que la CI couvre Grafana 9 à 13.
8+
- **Registre de prix rafraîchi depuis les sources officielles.** Chaque entrée expose sa provenance et sa date de vérification ; les tarifs absents restent non chiffrés. Artificial Analysis n'est disponible qu'en fallback tiers opt-in, avec attribution et restrictions de redistribution explicites.
9+
310
## [1.6.0] - 2026-08-30
411
### Fixed : exploitabilité en production
512
- **Un 403 de permissions sortait une trace Python.** Un exploitant ne pouvait pas savoir qu'il s'agissait d'un rôle de jeton insuffisant, ni dans quel état il avait laissé l'instance. Message actionnable nommant le rôle requis, codes de sortie distincts (3 = rien écrit, 4 = déploiement partiel), et l'emplacement des JSON générés pour un import manuel.

README.md

Lines changed: 27 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,24 @@ The skill handles discovery → registry refresh → blueprint selection → dep
110110
<summary><b>As a standalone CLI (no AI required)</b></summary>
111111

112112
Pure Python 3.8+ stdlib. No pip install. The three commands at the top of this README are the whole workflow. `--dry-run` writes JSON without touching your instance; `--selftest` renders all seven blueprints offline from a simulated capability map.
113+
114+
Unknown or officially unpriced detected models can use an explicit third-party
115+
fallback:
116+
117+
```bash
118+
export ARTIFICIAL_ANALYSIS_API_KEY="..."
119+
python3 scripts/forge_dashboards.py --capability capability_map.json \
120+
--pricing-fallback artificial-analysis --dry-run
121+
```
122+
123+
It is off by default. Only the fixed Artificial Analysis Free endpoint is
124+
called, only for detected models that need a price, and only the
125+
`ARTIFICIAL_ANALYSIS_API_KEY` environment variable is accepted. The key is not
126+
logged or written. Results are median multi-provider estimates, marked and
127+
attributed in dashboards and recording-rule labels, with a 24-hour atomic
128+
overlay in `model_registry.artificial-analysis.cache.json` next to the
129+
capability map. It never contains the merged registry. Official pricing always
130+
wins. See [pricing provenance](references/pricing_provenance.md).
113131
</details>
114132

115133
<details>
@@ -169,10 +187,13 @@ Or delete it from the UI: Dashboards → AI Observability → Folder settings
169187
Delete. Generated recording rules are separate: they are a file you copied into
170188
Prometheus, so removing them is removing that file and reloading.
171189

172-
**What leaves your network: nothing.** The scripts talk only to your Grafana,
173-
which talks to your own Prometheus, Loki and Tempo. The one optional exception is
174-
the model-price refresh, a web search you can skip entirely; the bundled
175-
registry works offline and the dashboards display its verification date.
190+
**What leaves your network: nothing by default.** The scripts talk only to your
191+
Grafana, which talks to your own Prometheus, Loki and Tempo. The explicit
192+
`--pricing-fallback artificial-analysis` exception sends authenticated GET
193+
requests to `https://artificialanalysis.ai/api/v2/language/models/free`; it can
194+
be skipped entirely. No Grafana data, including detected model names, is sent
195+
in the request: model matching happens locally against the unfiltered paginated
196+
catalog. The bundled registry works offline.
176197

177198
**What an agent sees.** Used as a skill, Claude reads the capability map, which
178199
contains metric names, model names, and team or service label values from your
@@ -186,7 +207,7 @@ Skills execute code, and [a 2026 Snyk audit found 36% of published skills had at
186207

187208
- **Zero dependencies.** Python stdlib only (`urllib`, `json`, `hashlib`). ~2,000 lines total. Playwright is *optional*, only for the visual-audit fallback.
188209
- **Least privilege.** Works with an Editor service-account token. Alert provisioning degrades gracefully on 403 (exports JSON for manual import).
189-
- **No secret leakage.** The token is never logged, never embedded in dashboards, never placed in URLs.
210+
- **No secret leakage.** Grafana credentials and the optional Artificial Analysis key are never logged, embedded in dashboards, persisted, or placed in URLs.
190211
- **No prompt-content capture.** `gen_ai.input/output.messages` stay off by default; the docs treat enabling them as a GDPR decision, not a flag.
191212
- **Idempotent & reversible.** Deterministic UIDs, one folder, `overwrite` semantics: delete the folder and it's gone.
192213
- **Offline-testable.** `--selftest` + `tests/audit_harness.py` (4 simulated instance topologies plus regression tests) runs with zero network, `tests/live_query_check.py` executes every generated query against a real Prometheus, and the demo stack gives a full end-to-end deploy. That's the CI; the badge above is the live workflow status.
@@ -231,7 +252,7 @@ tests/audit_harness.py # offline checks across 4 instance topologies + re
231252

232253
**Grafana Cloud?** Yes. Cloud is auto-detected; the image renderer is built in, so visual audit works out of the box.
233254

234-
**My models aren't in the registry.** They appear in an "unpriced models" panel instead of being billed wrong. Add a price or alias to `model_registry.json`, re-forge. (The matcher scores by specificity: `gpt-5.4-mini` will never be billed at `gpt-5.4` rates. There is a test for that.)
255+
**My models aren't in the registry.** They appear in an "unpriced models" panel instead of being billed wrong. Add a price or alias to `model_registry.json`, or explicitly opt in to `--pricing-fallback artificial-analysis`. The fallback accepts only a unique exact normalized ID, slug, name, or alias match. Ambiguous, absent, null, or failed responses stay unpriced.
235256

236257
**Can I publish the generated dashboards?** Yes: `--export-portable` emits JSON with `__inputs`/`${DS_PROMETHEUS}` placeholders, the format grafana.com/dashboards requires.
237258

SKILL.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,20 @@ Produces the capability map: version/edition/namespace, API availability (legacy
4747

4848
Read `references/model_registry.json`. If `_meta.verified_at` is more than 30 days old AND web search is available: refresh the prices of the models actually present in the capability map (not the whole registry) from the URLs in `_meta.sources`, then write `model_registry.local.json` next to the capability map. The generator loads the local file first. Without web access, use the seed as-is; cost dashboards display the registry date in their description.
4949

50+
Artificial Analysis is a third-party fallback, never an automatic refresh.
51+
Use `--pricing-fallback artificial-analysis` only after the user opts in and
52+
only with `ARTIFICIAL_ANALYSIS_API_KEY` in the environment. The forge calls the
53+
fixed Free endpoint, caches results next to the capability map for 24 hours,
54+
and accepts only a unique exact normalized ID, slug, name, or alias match.
55+
The cache is the separate overlay
56+
`model_registry.artificial-analysis.cache.json`; it never replaces the official
57+
local registry `model_registry.local.json` or the seed.
58+
Official prices always win. Ambiguous, absent, null, invalid, unauthorized,
59+
rate-limited, or server-error responses leave the model unpriced and do not
60+
stop the forge. Relay that these values are Artificial Analysis median
61+
multi-provider estimates. Never print or persist the key. Full protocol:
62+
`references/pricing_provenance.md`.
63+
5064
### Phase 2b: Cost recording rules (strongly recommended)
5165

5266
Every run writes `prometheus_rules_llmops.yml`: prices become series (`llm:price_*_usd_per_token{<model_label>=…}`) and cost becomes an aggregate metric (`llm:cost_usd_per_second`) joined by vector matching. Two files are written: the portable rule file and a `PrometheusRule` manifest for Kubernetes clusters running the Prometheus Operator, which is where most enterprise deployments live. Same rules, different packaging; the loading recipe per backend (Mimir, Thanos, VictoriaMetrics, AMP, Google Managed Prometheus) is in `references/grafana_api_compat.md` §6. Copy the portable file into Prometheus `rule_files` and reload: on the next run `discover.py` detects the `recorded` dialect and cost panels drop from a 2N-term sum to an O(1) query: unlimited models, and prices updatable without regenerating dashboards. Without it, on-the-fly composition stays active (40-model ceiling).
@@ -76,6 +90,8 @@ python3 scripts/forge_dashboards.py --capability capability_map.json --blueprint
7690
# Useful options:
7791
# --slo-target 0.995 burn-rate SLO target (default 0.99)
7892
# --cost-mode recorded force recording rules (default: auto-detected)
93+
# --pricing-fallback artificial-analysis opt-in third-party price estimates
94+
# --pricing-cache-max-age-hours 24 local cache freshness
7995
# --export-portable ${DS_*} JSON, publishable on grafana.com/dashboards
8096
# --datasource <uid|name> pin one datasource
8197
# --locale fr translate panel labels (default: English)

docs/assets/architecture.svg

Lines changed: 4 additions & 3 deletions
Loading

references/grafana_api_compat.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,14 @@ Depuis avril 2026, l'UI migre les dashboards ouverts vers le schéma v2
8686
erreurs structurées. Un échec demandé produit un code non nul; `--best-effort`
8787
autorise uniquement le code 0, sans transformer `partial|failed` en succès.
8888

89+
Les prix tiers ne transitent jamais par Grafana. Le fallback opt-in Artificial
90+
Analysis est résolu avant la génération, puis les recording rules portent les
91+
labels `pricing_source_kind` et `price_estimate`. L'overlay
92+
`model_registry.artificial-analysis.cache.json` et les règles générées sont des
93+
objets distincts. Cet overlay ne contient jamais le registre fusionné et
94+
n'entre jamais dans le paquet. Le protocole de réseau, de matching et
95+
d'attribution est décrit dans `pricing_provenance.md`.
96+
8997
## 6. Où déposer les recording rules générées
9098

9199
La forge écrit deux fichiers équivalents ; le contenu des règles est identique,

references/locale.fr.json

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,8 @@
8989
"The low tail is the real signal; the mean hides the failures.": "La queue basse est le vrai signal : la moyenne masque les échecs.",
9090
"The shape of the mix tells you the real strategy: a cheaper model gaining share means a FinOps trade-off that is working.": "La forme du mix raconte la stratégie réelle : montée d'un modèle économique = arbitrage FinOps qui fonctionne.",
9191
"These models are observed but **excluded from cost** (unknown price):": "Ces modèles sont observés mais **exclus du calcul de coût** (prix inconnu) :",
92+
"Third-party pricing estimates": "Estimations tarifaires tierces",
93+
"Some cost estimates use **median multi-provider pricing** from [Artificial Analysis](https://artificialanalysis.ai/). They are marked `estimate=true`; official provider pricing remains the priority. Attribution: Artificial Analysis.": "Certaines estimations utilisent la **médiane tarifaire multi-provider** d'[Artificial Analysis](https://artificialanalysis.ai/). Elles portent `estimate=true` ; les tarifs officiels restent prioritaires. Attribution : Artificial Analysis.",
9294
"Time-to-first-token: the first sign of saturation.": "Time-to-first-token : premier signal de saturation.",
9395
"Token throughput (prompt vs generation)": "Débit tokens/s (prompt vs génération)",
9496
"Tokens per agent/s": "Tokens par agent / s",
@@ -121,4 +123,4 @@
121123
"🇪🇺 EU providers": "🇪🇺 Providers UE",
122124
"🇺🇸 US providers": "🇺🇸 Providers US",
123125
"🌏 Asia providers": "🌏 Providers Asie"
124-
}
126+
}

0 commit comments

Comments
 (0)