Description
CostTracker._get_cache_prices (headroom/proxy/cost.py) prices provider-reported cache-read (and cache-write) tokens at the full uncached input rate whenever LiteLLM lacks explicit per-token cache price fields for the model:
uncached = info.get("input_cost_per_token")
if not uncached:
return None
cache_read = info.get("cache_read_input_token_cost", uncached)
cache_write = info.get("cache_creation_input_token_cost", uncached)
return (cache_read, cache_write, uncached)LiteLLM carries those two fields for only a minority of its priced models. Measured against the bundled LiteLLM pricing DB, 1681 of the models that have an input_cost_per_token lack both cache_read_input_token_cost and cache_creation_input_token_cost (Bedrock, Mistral, Fireworks, several OpenAI-compatible gateways, and so on). For every one of those models, any provider-reported cache_read token is billed at the full uncached rate inside _get_cache_prices, even though cache reads are always a discounted rate (about 10 percent for Anthropic/Bedrock, about 50 percent for OpenAI), and cache writes are billed at the plain uncached rate even where the provider charges a write premium (about 125 percent for Anthropic/Bedrock).
Impact
_get_cache_prices feeds CostTracker.totals(), which drives the /stats dollar figures. The primary accounting path, estimate_cost -> litellm.cost_per_token, prices those same cache-read tokens natively; for a field-less model it charges 0 for the cache-read slice. So for the 1681 long-tail models the two paths disagree in opposite directions on every cache-warm request: totals() over-charges cache reads at the full rate while the budget path under-charges them at 0. The dashboard cache breakdown then does not reconcile with the primary cost accounting for the same traffic.
Concrete example (bundled LiteLLM DB): mistral/mistral-large-latest, input 5e-07/token, no cache fields. 10k cache-read tokens: litellm.cost_per_token returns 0.0; _get_cache_prices returns the full 0.005.
Suggested fix
When the LiteLLM cache field is absent, fall back to the provider cache economics the dashboard already uses (_CACHE_ECONOMICS in the same module): price cache reads at the provider read_multiplier and cache writes at the write_multiplier fraction of the uncached rate, defaulting to Anthropic's multipliers when the provider is unknown (matching the existing default in build_prefix_cache_stats). That is both closer to reality than the full uncached rate and closer to the primary litellm.cost_per_token path than the status quo.
Reported by @EvolveAegis in #2620 (comment) as a distinct follow-up to the OpenAI cache-write double-charge fixed there.
Description
CostTracker._get_cache_prices(headroom/proxy/cost.py) prices provider-reported cache-read (and cache-write) tokens at the full uncached input rate whenever LiteLLM lacks explicit per-token cache price fields for the model:LiteLLM carries those two fields for only a minority of its priced models. Measured against the bundled LiteLLM pricing DB, 1681 of the models that have an
input_cost_per_tokenlack bothcache_read_input_token_costandcache_creation_input_token_cost(Bedrock, Mistral, Fireworks, several OpenAI-compatible gateways, and so on). For every one of those models, any provider-reportedcache_readtoken is billed at the full uncached rate inside_get_cache_prices, even though cache reads are always a discounted rate (about 10 percent for Anthropic/Bedrock, about 50 percent for OpenAI), and cache writes are billed at the plain uncached rate even where the provider charges a write premium (about 125 percent for Anthropic/Bedrock).Impact
_get_cache_pricesfeedsCostTracker.totals(), which drives the/statsdollar figures. The primary accounting path,estimate_cost->litellm.cost_per_token, prices those same cache-read tokens natively; for a field-less model it charges 0 for the cache-read slice. So for the 1681 long-tail models the two paths disagree in opposite directions on every cache-warm request:totals()over-charges cache reads at the full rate while the budget path under-charges them at 0. The dashboard cache breakdown then does not reconcile with the primary cost accounting for the same traffic.Concrete example (bundled LiteLLM DB):
mistral/mistral-large-latest, input5e-07/token, no cache fields. 10k cache-read tokens:litellm.cost_per_tokenreturns0.0;_get_cache_pricesreturns the full0.005.Suggested fix
When the LiteLLM cache field is absent, fall back to the provider cache economics the dashboard already uses (
_CACHE_ECONOMICSin the same module): price cache reads at the providerread_multiplierand cache writes at thewrite_multiplierfraction of the uncached rate, defaulting to Anthropic's multipliers when the provider is unknown (matching the existing default inbuild_prefix_cache_stats). That is both closer to reality than the full uncached rate and closer to the primarylitellm.cost_per_tokenpath than the status quo.Reported by @EvolveAegis in #2620 (comment) as a distinct follow-up to the OpenAI cache-write double-charge fixed there.