headroomlabs-ai
diff --git a/‎docs/metrics-technical-guide.md‎
Lines changed: 223 additions & 0 deletions b/‎docs/metrics-technical-guide.md‎
Lines changed: 223 additions & 0 deletions
diff --git a/‎headroom/agent_savings.py‎
Lines changed: 48 additions & 10 deletions b/‎headroom/agent_savings.py‎
Lines changed: 48 additions & 10 deletions
@@ -0,0 +1,223 @@
+# Headroom Metrics — Dashboard Guide
+
+What each metric shows, so you can build panels against it.
+
+**Two endpoints.** Both are on the proxy (default `:8787`).
+
+| Surface | How to get it | Use it for |
+|---|---|---|
+| **Prometheus** — `GET /metrics` | Always on, no config | Everything below. Start here. |
+| **OpenTelemetry** — OTLP/HTTP | `HEADROOM_OTEL_METRICS_ENABLED=1` + `pip install "headroom-ai[proxy,otel]"` | Same data, dotted names, plus per-tenant labels |
+
+Names differ between them: Prometheus uses `headroom_tokens_saved_total` (**milliseconds** for timings), OTel uses `headroom.proxy.tokens.saved` (**seconds**). Both are listed below.
+
+---
+
+## The savings panel — start here
+
+**`headroom.proxy.tokens.saved`** is the headline number. It already combines compression + tool-schema deferral — no need to add anything to it.
+
+| Metric | What it shows |
+|---|---|
+| **`headroom.proxy.tokens.saved`** *(OTel)* | **Total input tokens Headroom kept out of the request.** Compression + tool savings, combined. This is your hero number. |
+| `headroom.proxy.savings.usd{source}` *(OTel)* | **Dollars saved**, split by layer: `compression`, `tool_schema`, `output_shaping`, `provider_cache`. Sum for the total. |
+| `headroom_persistent_savings_tokens_saved_total` | Same tokens-saved number, but **survives proxy restarts**. Use for "lifetime saved" tiles. |
+| `headroom_persistent_savings_compression_savings_usd_total` | **Lifetime dollars saved**, durable across restarts. |
+| `headroom_tokens_input_total` | Input tokens actually sent upstream (post-compression). The denominator for a reduction %. |
+| `headroom_tokens_output_total` | Output tokens returned by the provider. |
+
+```promql
+# Hero tile: tokens saved per second
+rate(headroom_tokens_saved_total[5m])
+  + sum(rate(headroom_savings_attributed_tokens_total{source="tool_search",realized="true"}[5m]))
+
+# Context reduction %
+100 * rate(headroom_tokens_saved_total[5m])
+    / clamp_min(rate(headroom_tokens_input_total[5m]) + rate(headroom_tokens_saved_total[5m]), 1)
+
+# Lifetime tiles (survive restart)
+headroom_persistent_savings_tokens_saved_total
+headroom_persistent_savings_compression_savings_usd_total
+```
+
+> **One catch on the Prometheus side.** `headroom_tokens_saved_total` is compression **only** — it leaves out tool-schema deferral. The OTel `headroom.proxy.tokens.saved` includes both. That's why the query above adds the `tool_search` term back in. On tool-heavy workloads the gap is large.
+
+---
+
+## Latency panel
+
+All Prometheus timings are in **milliseconds**, exposed as `_sum` / `_count` / `_min` / `_max`. Build means with `rate(sum)/rate(count)`.
+
+| Metric | What it shows |
+|---|---|
+| **`headroom_overhead_ms_*`** | **Latency Headroom itself adds.** Handler entry → end of compression. Excludes the LLM call. This is the "what does this cost us" number. |
+| `headroom_latency_ms_*` | Total request duration, including the provider. |
+| `headroom_ttfb_ms_*` | Time to first byte from upstream. Streaming requests only. |
+| `headroom_stage_timing_ms_*{path,stage}` | Where time went inside the handler — `compression_first_stage`, `upstream_connect`, `memory_context`, etc. |
+| `headroom_transform_timing_ms_*{transform}` | Time per compression transform. Use to find a slow transform. |
+
+```promql
+# Headroom's added overhead, mean ms
+rate(headroom_overhead_ms_sum[5m]) / rate(headroom_overhead_ms_count[5m])
+
+# End-to-end, mean ms
+rate(headroom_latency_ms_sum[5m]) / rate(headroom_latency_ms_count[5m])
+
+# Slowest stages
+topk(5, rate(headroom_stage_timing_ms_sum[5m]) / rate(headroom_stage_timing_ms_count[5m]))
+```
+
+> **No percentiles are available.** There are no histogram buckets on `/metrics`, and the OTel histograms ship with default buckets that put every request into one bucket, so `histogram_quantile()` returns nonsense. **Means work fine.** For real p95/p99 today, use the `headroom perf` CLI.
+>
+> Also: divide each `_sum` by **its own** `_count`. Overhead and TTFB are only sampled when > 0, so their counts are smaller than the latency count.
+
+---
+
+## Cache panel
+
+| Metric | What it shows |
+|---|---|
+| `headroom_provider_cache_hit_requests_total{provider}` | Requests that read from the provider's prompt cache. |
+| `headroom_provider_cache_requests_total{provider}` | Requests with any cache activity. **The correct denominator for hit rate.** |
+| `headroom_cache_read_tokens_total{provider}` | Tokens served from cache (the discounted ones). |
+| `headroom_cache_write_tokens_total{provider}` | Tokens written into cache (these carry a premium). |
+| `headroom_cache_write_ttl_tokens_total{provider,ttl}` | Cache writes split by TTL — `5m` vs `1h`. |
+| `headroom_uncached_input_tokens_total{provider}` | Input tokens that missed cache entirely. |
+| `headroom_cache_bust_total` | Requests where compression broke a cached prefix. **Should stay near zero.** |
+| `headroom_cache_miss_attribution_total{provider,reason}` | Why a cached prefix missed — `ttl_expiry`, `prefix_change`, `unknown`. |
+
+```promql
+# Cache hit rate by provider
+sum by (provider) (rate(headroom_provider_cache_hit_requests_total[5m]))
+  / sum by (provider) (rate(headroom_provider_cache_requests_total[5m]))
+
+# Compression breaking cache — alert if this rises
+rate(headroom_cache_bust_total[5m])
+```
+
+> **Don't use `headroom_requests_cached_total` as a hit rate.** It mixes the provider's prompt cache with Headroom's own response cache into one boolean, so it measures neither.
+
+---
+
+## Traffic & health panel
+
+| Metric | What it shows |
+|---|---|
+| `headroom_requests_total` | Requests handled. Unlabelled. |
+| `headroom_requests_by_provider{provider}` | Traffic split by provider — `anthropic`, `openai`, `gemini`, `bedrock`… |
+| `headroom_requests_by_model{model}` | Traffic split by model. Capped at 1024 distinct; overflow lands in `model="other"`. |
+| `headroom_requests_failed_total` | Upstream 5xx errors. |
+| `headroom_requests_rate_limited_total` | Requests **Headroom** rejected via its own rate limiter (not upstream 429s). |
+| `headroom_compression_failed_total{reason}` | Compression failures — `timeout` or `error`. Fails open, so traffic keeps flowing but savings quietly stop. **Worth an alert.** |
+| `headroom_compression_quarantine_total{event}` | Compression disabled after repeated timeouts — `activated`, `skipped`, `released`. |
+| `headroom_inbound_requests_active` | In-flight requests, gauge. Counts all HTTP including `/metrics`. |
+| `headroom_active_ws_sessions` | Live Codex WebSocket sessions, gauge. |
+
+```promql
+# Failure rate
+rate(headroom_requests_failed_total[5m])
+  / clamp_min(rate(headroom_requests_total[5m]) + rate(headroom_requests_failed_total[5m]), 1)
+
+# Savings silently stopped
+sum by (reason) (rate(headroom_compression_failed_total[5m]))
+
+# Traffic mix
+sum by (provider) (rate(headroom_requests_by_provider[5m]))
+```
+
+---
+
+## Anthropic subscription panel
+
+Only if you're on an Anthropic OAuth/subscription plan. OTel only, gauges, no labels.
+
+| Metric | What it shows |
+|---|---|
+| `headroom.subscription.5h_utilization_pct` | How much of the 5-hour rate-limit window is used (0–100). |
+| `headroom.subscription.7d_utilization_pct` | Same for the 7-day window. |
+| `headroom.subscription.5h_seconds_to_reset` | Seconds until the 5-hour window resets. |
+| `headroom.subscription.7d_seconds_to_reset` | Seconds until the 7-day window resets. |
+| `headroom.subscription.overage_usd` | Extra-usage credits consumed, in dollars. |
+
+---
+
+## Attribution — where savings came from
+
+| Metric | What it shows |
+|---|---|
+| `headroom_savings_attributed_tokens_total{source,realized}` | Tokens saved, broken out by named source. `source="tool_search"` is tool-schema deferral. |
+| `headroom_savings_attributed_usd_total{source,realized}` | Dollars saved by source. **Gauge, can go negative** — don't `rate()` it. |
+| `headroom_savings_attribution_events_total{source,realized}` | How often each source contributed. |
+| `headroom_waste_signal_tokens_total{signal}` | Wasteful patterns *detected* in the input — `json_bloat`, `base64`, `repetition`, `reread`… This is diagnosis, **not savings**. |
+
+These rows *explain* the headline total — they are never added to it.
+
+---
+
+## Compression internals
+
+| Metric | What it shows |
+|---|---|
+| `headroom.compression.tokens.input` *(OTel)* | Tokens going into the compression pipeline. |
+| `headroom.compression.tokens.output` *(OTel)* | Tokens coming out. |
+| `headroom.compression.tokens.saved` *(OTel)* | The difference. Pipeline-level view of compression only. |
+| `headroom.compression.runs` *(OTel)* | Pipeline executions. Note: **per pipeline run, not per request.** |
+| `headroom.compression.pipeline.duration` *(OTel, seconds)* | How long the pipeline took. |
+| `headroom.compression.transforms{transform}` *(OTel)* | Which transforms fired. **High cardinality — drop or aggregate at the collector.** |
+
+---
+
+## Five things that will break a dashboard
+
+1. **Only savings counters survive a restart.** 55 of 60 Prometheus families reset to zero when the proxy restarts. Only `headroom_persistent_savings_*` is durable, and it needs `HEADROOM_WORKSPACE_DIR` on a persistent volume — otherwise it resets on every deploy.
+
+2. **No percentiles anywhere.** Use means. See the latency section.
+
+3. **`headroom_latency_ms` measures differently for streaming.** On streaming requests the timer starts *after* compression, so end-to-end is `latency + overhead`. On non-streaming it's just `latency`. Don't mix both in one panel.
+
+4. **A 5xx erases its own savings.** Requests that fail upstream are dropped from every savings and token counter. During a provider incident, savings rates look artificially clean while throughput falls.
+
+5. **`/metrics` needs auth if you set a proxy token.** With `HEADROOM_PROXY_TOKEN` set, any non-loopback scraper must send `Authorization: Bearer <token>`. Loopback is always exempt.
+
+---
+
+## Metrics the docs mention that don't exist
+
+If panels came back empty, this is probably why. These names appear in the published docs but not in the code:
+
+`headroom_compression_ratio` · `headroom_latency_seconds` (and `_bucket`) · `headroom_cache_hits_total` · `headroom_cache_misses_total` · `headroom_cost_usd_total` · the `mode="optimize"` label on `headroom_requests_total`
+
+The shipped `examples/grafana/headroom-dashboard.json` also filters every panel on `pool` and `hook` labels that no metric emits — the dropdowns will be permanently empty. Its metric names are otherwise correct.
+
+---
+
+## Setup reference
+
+```bash
+# Prometheus — nothing to do, GET /metrics is always on
+
+# OpenTelemetry
+pip install "headroom-ai[proxy,otel]"
+export HEADROOM_OTEL_METRICS_ENABLED=1
+export HEADROOM_OTEL_METRICS_ENDPOINT=https://otel.corp.example/v1/metrics
+export HEADROOM_OTEL_METRICS_HEADERS="authorization=Bearer XXX"
+export HEADROOM_OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$HOSTNAME"
+```
+
+| Variable | Default | Notes |
+|---|---|---|
+| `HEADROOM_OTEL_METRICS_ENABLED` | `0` | Master switch |
+| `HEADROOM_OTEL_METRICS_EXPORTER` | `otlp_http` | Or `console`. No gRPC exporter exists. |
+| `HEADROOM_OTEL_METRICS_ENDPOINT` | unset | Passed verbatim — `/v1/metrics` is **not** appended |
+| `HEADROOM_OTEL_METRICS_HEADERS` | unset | `k=v,k2=v2` |
+| `HEADROOM_OTEL_METRICS_EXPORT_INTERVAL_MS` | `10000` | |
+| `HEADROOM_OTEL_SERVICE_NAME` | `headroom-proxy` | |
+| `HEADROOM_OTEL_RESOURCE_ATTRIBUTES` | unset | **Set `service.instance.id` here** — Headroom doesn't, and replicas will collide |
+
+Verify with `curl -s localhost:8787/stats | jq .otel`.
+
+**Multi-tenant labels:** `register_otel_metric_attribute_provider()` adds request-scoped attributes (tenant, team, cost centre) to every OTel datapoint. Max 16 attributes, 256 chars each.
+
+**Air-gapped deployments:** `HEADROOM_OFFLINE=1` disables all outbound traffic — the anonymous usage beacon (which is **on by default**), the update check, and model downloads.
+
+---
@@ -207,27 +207,43 @@ def apply_proxy_env_defaults(self, env: MutableMapping[str, str]) -> MutableMapp
 def get_agent_savings_profile(name: str | None = None) -> AgentSavingsProfile:
     """Return a named agent savings profile.
 
-    An unrecognized name falls back to the ``balanced`` profile with a warning
-    instead of raising. The savings profile is a soft config knob, but it is
-    resolved during proxy startup (``proxy_pipeline_kwargs`` -> ``create_app``),
-    so raising here takes the whole proxy down before it can open its port. That
-    happens on desktop/runtime version skew: a newer client requests a profile
-    (e.g. ``coding``) that an older pinned or fallback runtime predates. Degrade
-    to ``balanced`` rather than leaving the user with no proxy at all.
+    An unrecognized name degrades with a warning instead of raising. The savings
+    profile is a soft config knob, but it is resolved during proxy startup
+    (``proxy_pipeline_kwargs`` -> ``create_app``), so raising here takes the
+    whole proxy down before it can open its port. That happens on desktop/runtime
+    version skew: a newer client requests a profile (e.g. ``coding``) that an
+    older pinned or fallback runtime predates. Degrade rather than leaving the
+    user with no proxy at all.
+
+    **Where it degrades to matters.** This used to land on ``balanced``
+    unconditionally, which is a drastically different posture from the
+    out-of-box default: cache->token mode, cross-turn dedup off, tool-search
+    off, user messages uncompressed, the message floor 25x higher (250 vs 10)
+    and the block floor 20x higher (500 vs 25). A single typo in
+    ``HEADROOM_SAVINGS_PROFILE`` therefore silently reconfigured the whole
+    proxy, and the only trace was one WARNING at startup that operators read
+    past. Prefer :data:`DEFAULT_PROFILE` — the documented out-of-box posture and
+    the same thing an unset variable resolves to, so a typo now costs nothing.
+    ``balanced`` remains the last resort for the genuine version-skew case,
+    where an older runtime has no ``DEFAULT_PROFILE`` entry to fall back to.
     """
 
     key = (name or DEFAULT_PROFILE).strip().lower()
     profile = _PROFILES.get(key)
     if profile is not None:
         return profile
+    fallback_name = DEFAULT_PROFILE if DEFAULT_PROFILE in _PROFILES else FALLBACK_PROFILE
     valid = ", ".join(sorted(_PROFILES))
     logger.warning(
-        "unknown savings profile %r; falling back to %r (known: %s)",
+        "unknown savings profile %r; falling back to %r (known: %s). "
+        "Set HEADROOM_SAVINGS_PROFILE to one of the known names, or unset it to "
+        "get %r explicitly.",
         name,
-        FALLBACK_PROFILE,
+        fallback_name,
         valid,
+        DEFAULT_PROFILE,
     )
-    return _PROFILES[FALLBACK_PROFILE]
+    return _PROFILES[fallback_name]
 
 
 def apply_agent_savings_env_defaults(
@@ -300,6 +316,28 @@ def proxy_pipeline_kwargs(config: object) -> dict[str, object]:
         # unset → Kompress decides / ambient default applies).
         if profile.target_ratio is not None:
             kwargs["target_ratio"] = profile.target_ratio
+        # Block-compression char floor. Every OTHER router pipeline kwarg in this
+        # function travels on the config object; this one alone was populated
+        # only from ``HEADROOM_MIN_CHARS_FOR_BLOCK`` (read below), so a proxy
+        # whose config carries ``savings_profile="coding"`` but whose process env
+        # was never seeded applied every sibling coding knob while this floor
+        # silently stayed at ``ContentRouterConfig.min_chars_for_block_compression``
+        # (500) instead of the profile's 25 — a 20x gap on the gate that governs
+        # tool_result blocks, the dominant content type in agent traffic.
+        #
+        # NOTE: this does not make the profile fully config-deliverable. The
+        # profile's ``cross_turn_dedup`` / ``tool_search`` / ``lossless_then_lossy``
+        # / ``protect_reads`` / ``code_aware`` / ``effort_router`` / ``lossless``
+        # fields are still env-only, but by a different mechanism: their consumers
+        # read ``os.environ`` directly (ContentRouter.__init__ for HEADROOM_DEDUPE,
+        # the Anthropic handler for HEADROOM_TOOL_SEARCH) and never pass through
+        # this function at all. Those remain seed-dependent and are the reason a
+        # profile can still be half-applied; fixing them means threading each
+        # consumer, which is a larger change than this one.
+        #
+        # The env read below still wins, since it is an explicit operator override.
+        if profile.min_chars_for_block is not None:
+            kwargs["min_chars_for_block_compression"] = profile.min_chars_for_block
 
     if getattr(config, "compress_user_messages", False):
         kwargs["compress_user_messages"] = True