fix: backfill input + cache token counts in OpenAI→Anthropic streaming message_delta#2350
Open
ajac-zero wants to merge 14 commits into
Open
fix: backfill input + cache token counts in OpenAI→Anthropic streaming message_delta#2350ajac-zero wants to merge 14 commits into
ajac-zero wants to merge 14 commits into
Conversation
…pic streaming message_delta The OpenAI→Anthropic streaming translator emitted message_start with input_tokens=0 (OpenAI does not report prompt tokens until the final usage chunk) and dropped input_tokens/cache_* from the closing message_delta. Anthropic clients that read usage from the streaming events (e.g. the Vercel AI SDK's @ai-sdk/anthropic provider) then recorded input_tokens=0 for every assistant turn, breaking context-usage UIs and token accounting for OpenAI-backed models routed through /anthropic/v1. Backfill input_tokens, cache_creation_input_tokens and cache_read_input_tokens in the final message_delta.usage. The input/output token totals are already available on the stream state by the time emitClosingEvents runs (captured from OpenAI's usage chunk); cache counts are now sourced from that same chunk's prompt_tokens_details (cached_tokens / cache_creation_input_tokens) instead of being hardcoded to zero. The AI SDK anthropic streaming handler overrides input_tokens, cache_read_input_tokens and cache_creation_input_tokens independently whenever the message_delta value is non-null, regardless of what message_start reported, so no client-side changes are required. This also fixes the gateway's own internal cost-tracking TokenUsage for this stream path, which previously always recorded zero cache tokens for OpenAI-backed models served through /anthropic/v1, even when the underlying request hit OpenAI's prompt cache. Signed-off-by: ajac-zero <ajcardoza2000@gmail.com>
dosubot
Bot
added
the
size:M
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2350 +/- ##
=======================================
Coverage 84.85% 84.86%
=======================================
Files 151 151
Lines 22086 22089 +3
=======================================
+ Hits 18742 18745 +3
Misses 2213 2213
Partials 1131 1131 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…e_delta usage Update tests/data-plane/testupstream_test.go expectations for the anthropic-openai streaming text response and streaming tool call cases to include input_tokens, cache_creation_input_tokens and cache_read_input_tokens in message_delta.usage, matching the fix in openai_helper.go. Signed-off-by: ajac-zero <ajcardoza2000@gmail.com>
ajac-zero
marked this pull request as draft
July 12, 2026 07:14
ajac-zero
force-pushed
the
fix-openai-anthropic-stream-usage
branch
from
July 12, 2026 08:33
0c2388e to
eb15a0e
Compare
…c streaming usage OpenAI's prompt_tokens_details.cached_tokens and cache_creation_input_tokens are a breakdown *within* prompt_tokens (analogous to text_tokens/audio_tokens in the same struct), not additive extras. Anthropic's native usage schema is the opposite: input_tokens excludes cache tokens, which are reported as separate, additive counters (cache_read_input_tokens, cache_creation_input_tokens). The previous commit forwarded OpenAI's cache token breakdown directly into Anthropic's additive cache usage fields without accounting for the semantic mismatch. Since Anthropic-consuming clients compute total context usage as input_tokens + cache_read_input_tokens + cache_creation_input_tokens, this caused the cached portion of every prompt to be counted twice: once inside input_tokens (via OpenAI's prompt_tokens) and again via the cache fields. Depending on the backend's exact reporting of cache_creation_input_tokens (a field with no documented meaning in OpenAI's real Chat Completions API, so its semantics are backend-defined), this could inflate the reported input token count far beyond the real prompt size, up to the point of clients rejecting requests as exceeding the model's context window. Revert to not forwarding OpenAI's cache token breakdown into Anthropic's cache usage fields, for both the client-facing message_delta.usage payload and the gateway's internal TokenUsage metrics. input_tokens/output_tokens continue to be backfilled correctly, which was the original motivating fix. The pre-existing ptr.To(int64(0)) cache args to ExtractTokenUsageFromExplicitCaching are left untouched to keep this change minimal. Signed-off-by: ajac-zero <ajcardoza2000@gmail.com>
ajac-zero
force-pushed
the
fix-openai-anthropic-stream-usage
branch
from
July 12, 2026 09:02
eb15a0e to
f876472
Compare
…gressions Strengthen the regression test for the cache-token double-counting bug fixed in the previous commit: - Drive the translator's emitted SSE stream through anthropic-sdk-go's own Message.Accumulate logic (the exact code path a real Anthropic client runs to build the final Message/Usage from a stream), instead of only asserting against our own emitted JSON. A test that merely snapshots our translator's own output can enshrine a bug as "expected" rather than catching it; this exercises the same client-side logic that actually surfaced the original bug during dogfooding. - Assert the invariant that must always hold (total client-visible input tokens equal the backend's real prompt_tokens) rather than a hardcoded expected value, so the test keeps guarding against regressions even if the implementation approach changes later. - Use realistic large-scale token counts (~97% cache hit rate, as in a long-running coding-agent session) instead of small arbitrary numbers, so a reintroduced bug produces an obviously-wrong total instead of an easy-to-miss discrepancy. Verified this test fails clearly (reports 378000 vs the real 190000) when the previous commit's fix is manually reverted, and passes with the fix in place. Signed-off-by: ajac-zero <ajcardoza2000@gmail.com>
ajac-zero
force-pushed
the
fix-openai-anthropic-stream-usage
branch
from
July 12, 2026 19:53
f876472 to
1a168f6
Compare
Cut the large explanatory comment blocks down to one or two lines each -- the rationale is already documented in the fix commit and in the assertion failure messages themselves, so repeating it at length in the test file added noise without adding information. Signed-off-by: ajac-zero <ajcardoza2000@gmail.com>
Shorten the comments explaining why we don't forward OpenAI's cache token breakdown, without changing behavior. Signed-off-by: ajac-zero <ajcardoza2000@gmail.com>
ajac-zero
force-pushed
the
fix-openai-anthropic-stream-usage
branch
from
July 12, 2026 20:00
1a168f6 to
6d93f89
Compare
ajac-zero
marked this pull request as ready for review
July 12, 2026 20:15
dosubot
Bot
added
size:S
missBerg
added
the
area/translation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds the input-token count from OpenAI's final usage chunk to the closing Anthropic
message_delta.usageevent. Anthropic clients use that event to update the final streamedMessage.usage, so the translated response now retains the backend's input-token count.OpenAI cache-token values are intentionally not forwarded. They are a breakdown already included in OpenAI
prompt_tokens, whereas Anthropic cache-token fields are additional toinput_tokens; forwarding them directly would double-count cached tokens.Related Issues/PRs (if applicable)
Fixes #2352
Special notes for reviewers (if applicable)
The regression test accumulates the translated stream with
anthropic-sdk-go, matching the behavior of an Anthropic client.AI usage disclosure
Portions of this change were developed with the assistance of Claude Sonnet 5. I reviewed, understood, and take full ownership of the change, and am able to address review feedback directly.