RFC: a single model-attribute layer (one catalog, read everywhere)

[romone22]

Problem

Model facts — context window, tokenizer, pricing, cache economics — live in five places that drift:

1. Each Python provider hardcodes its own table + prefix logic + inconsistent fallback: providers/google.py raises ValueError for unknown models (get_token_counter, get_context_limit), while providers/openai.py / providers/anthropic.py degrade gracefully (_load_custom_model_config + family inference + default). Same job, three behaviours.
2. crates/headroom-proxy/src/compression/model_limits.rs reads the vendored LiteLLM catalog — but only extracts max_input_tokens.
3. crates/headroom-proxy/data/model_prices_and_context_window.json — the rich catalog (~2,700 models incl. gpt-5.2, gemini-3-pro-preview, with input_cost_per_token, cache_read_input_token_cost, cache_creation_input_token_cost) — is under-exploited.
4. The installed litellm package: Python providers use litellm.get_model_info as a soft lookup — a second copy of the same catalog, version-dependent, that can drift from the vendored snapshot.
5. compression_policy.{py,rs} hardcodes cache economics (CACHE_WRITE_MULTIPLIER=1.25, CACHE_READ_MULTIPLIER=0.10) for every provider, ignoring the catalog's per-model cache costs.

Cost of the fragmentation:

• New models break unevenly. gemini-3-* raises in the Google provider, so the in-process LangChain wrapper (integrations/langchain/chat_model.py, which calls get_context_limit) falls back to no optimization — while the same model degrades cleanly on OpenAI. Silent and provider-specific.
• Every model/vendor is a manual PR. fix: add DeepSeek V4/V3.2/R1 tokenizer mappings and context limits #912 (DeepSeek) and [FEATURE] Bedrock-native compression parity for non-Anthropic vendors (Amazon Nova, ZAI/GLM, MiniMax, Kimi) + cross-region inference profile fix #953 (Nova/GLM/Kimi) are recurring symptoms of editing per-provider dicts by hand.
• Cache decisions are provider-blind. The net-cost gate (feat(policy): consume net-cost mutation gate in ContentRouter (#856 P2) #905) uses Claude's economics for GPT-5 / Gemini, so it under-compresses them.

Proposal

One catalog, read the same way everywhere; providers and policy become thin consumers.

1. Source of truth: the vendored catalog (model_prices_and_context_window.json), already in-repo, already refreshed by scripts/refresh_model_limits.sh. Keep vendoring (no runtime/build network — the existing model_limits.rs rationale stands).
2. One accessor per language. Extend model_limits.rs to expose all relevant fields (context, pricing, cache read/creation), not just max_input_tokens. Add the Python equivalent reading the same data.
3. Providers become thin and uniform. get_context_limit / get_token_counter / estimate_cost resolve via one chain for all three: operator override (models.json / HEADROOM_MODEL_LIMITS) → catalog → family inference → graceful default. Never raise. The per-provider hardcoded dicts shrink to a small set of known-stable anchors for offline robustness.
4. Policy consumes the catalog. CompressionPolicy derives (w, r) per-model from the catalog's cache costs, defaulting to today's constants when absent (so existing anchors are unchanged).
5. Operator override + freshness, unchanged in spirit. models.json (generalized to every provider) for local additions; a scheduled CI job re-vendors the snapshot and opens a PR.

Concrete shape

Fields the layer reads from the catalog (already present, per model): max_input_tokens, input_cost_per_token, output_cost_per_token, cache_read_input_token_cost, cache_creation_input_token_cost, supports_prompt_caching. Everything the providers and the policy need is already there — nothing new to author per model.

Operator override (~/.headroom/config/models.json), generalized to every provider — the same shape the OpenAI/Anthropic loaders already accept, now honored by Google too:

{
  "google":   { "context_limits": { "gemini-9-experimental": 1048576 },
                "pricing":        { "gemini-9-experimental": [2.5, 15.0] } },
  "openai":   { "context_limits": { "gpt-6": 400000 } },
  "fallback": { "on_unknown_model": "infer_then_default" }
}

Resolution order, identical for every provider: operator override → vendored catalog → family inference → graceful default (never raise).

Migration — incremental, each piece shippable on its own

1. fix(providers): Google adopts the graceful-fallback + models.json loader already in OpenAI / Anthropic. Pure bug fix — fixes the gemini-3 raise. Independent.
2. refactor(providers): extract the shared catalog accessor; route all three Python providers through it; collapse the duplicated dicts to anchors. Behaviour-preserving.
3. feat(policy): policy derives per-model (w, r) from the catalog (Rust + Python), default = today's constants. Builds on feat(policy): consume net-cost mutation gate in ContentRouter (#856 P2) #905; provable via HEADROOM_NET_COST_POLICY=1.
4. ci: scheduled re-vendor workflow (freshness).
5. Optional: retire the Python CompressionPolicy hand-mirror via PyO3 — the module docstring already flags this as desirable.

Each step is small, reversible, and leaves the tree green; the end state is one coherent layer.

Non-goals

• No change to the compression algorithms.
• No runtime/build-time catalog fetch (keep the vendored snapshot).
• No removal of the operator override.
• Not a model router (which model to call) — that's [FEATURE] Dynamic model registry for headroom wrap — auto-discover available models #77's /v1/models discovery; this is the attribute substrate underneath it.

Relationship to current work

• [FEATURE] Dynamic model registry for headroom wrap — auto-discover available models #77 (dynamic model registry): complementary. /v1/models answers "which models exist"; this answers "what are this model's compression-relevant attributes". This is the static fallback layer [FEATURE] Dynamic model registry for headroom wrap — auto-discover available models #77 lists as component 3.
• fix: add DeepSeek V4/V3.2/R1 tokenizer mappings and context limits #912 (DeepSeek tokenizer/context PR): the per-model treadmill the layer removes — new models stop needing per-provider edits.
• [FEATURE] Bedrock-native compression parity for non-Anthropic vendors (Amazon Nova, ZAI/GLM, MiniMax, Kimi) + cross-region inference profile fix #953 (Bedrock vendor parity: Nova/GLM/Kimi): those vendors are already in the catalog, so they're served by the same layer.
• feat(policy): consume net-cost mutation gate in ContentRouter (#856 P2) #905 (net-cost mutation gate, merged): the layer makes its (w, r) per-model instead of Claude-calibrated.
• feat: compression extraction — Rust knob exposure, CCR hardening, traffic audits #818 (Rust knob exposure / CCR hardening / read maturation): orthogonal — it tunes how/when to compress and hardens CCR, not the model-attribute layer. No file collision expected. The two compose: feat: compression extraction — Rust knob exposure, CCR hardening, traffic audits #818 decides timing, this decides per-model economics.

Design decisions

1. Catalog stays in headroom-proxy; it injects (w, r) into the core policy. Core must not carry a 1.4MB JSON or depend on proxy, so the proxy (which already owns the snapshot via include_str!) computes the per-model economics and passes them through CompressionPolicy::with_cache_economics(…). If you'd rather the catalog live in core, that's your call — it changes the wiring.
2. One physical catalog, read by both languages; retire the litellm.model_cost lookup. Ship the vendored JSON as Python package data and read it in Python too, so Rust and Python can't drift. Bonus: drops Python's soft dependency on whatever litellm version is installed. Cost: ~1.4MB in the wheel — acceptable for a deploy artifact.
3. Shrink the per-provider dicts to a small anchor set; stop growing them per-model. Keep a handful of known-stable models per provider as an offline last-resort default and a test lock; route the primary lookup through the catalog. New models then need zero per-provider edits — the whole point.

I'm happy to drive this as the migration PRs above, starting with the Google bug fix (independent) and the per-model economics on top of #905. Does the shape work for you?