[BUG] Empty HTTP 200 returned after CCR buffered-stream conversion (proxy never contacts upstream, logs no error)

[Zcehtro]

Empty HTTP 200 returned after CCR buffered-stream conversion (proxy never contacts upstream, logs no error)

Description

When CCR converts a stream:true client request into a buffered stream:false upstream request so it can
handle headroom_retrieve server-side, the next request on that session aborts inside the proxy between
outbound_headers and beta_header_merge, and the client receives HTTP 200 with an empty body. The proxy
never opens a connection to api.anthropic.com for that request.

There are two defects here:

1. The abort itself. The request dies silently after the CCR buffered round-trip on the same session.
2. The reporting. The failure emits no ERROR line at all in proxy.log, and answers 200 rather
   than a 5xx. A failure that returns success and logs nothing took several sessions to localise.

Correlation in a single proxy.log: 6 CCR buffered conversions, 6 empty-200 deaths, perfectly paired in
both directions across three separate client sessions. No conversion without a death, no death without a
conversion.

The trigger appears to be headroom_retrieve being present in the tool list at all, not any particular
prompt. Note the proxy itself injects that tool
(event=tool_injection_decision ... decision=inject_sticky_replay tool_definition_bytes_count=464), so
removing the headroom MCP server client-side does not avoid it.

To Reproduce

Steps to reproduce the behavior:

1. Install headroom: uv tool install headroom-ai (version 0.35.0).

2. Start the proxy standalone, default transforms, CCR enabled:

   headroom proxy --port 8787

3. Run a Claude Code client through it:

   headroom wrap claude

   (this injects ANTHROPIC_BASE_URL=http://127.0.0.1:8787 per-process)

4. Work a long session until the proxy logs a CCR buffered conversion:

   CCR: stream:true request has headroom_retrieve available; using buffered stream:false upstream request for server-side retrieval handling
   

5. The next request on that session dies. The client prints the error below and the session must be
   exited.

Observed on requests that are not small: five of the six deaths landed at msgs=35, the sixth at msgs=140
with content_length=284080. Never on small requests.

To localise a death in proxy.log, grep for the signature:

PERF model=\S+ msgs=\d+ tok_before=0 tok_after=0 .* transforms=none

That pattern combined with no event=outbound_request matched exactly the six deaths and nothing else.

Expected Behavior

Either the request is forwarded upstream normally, or — if the proxy cannot process it — it returns a 5xx
with a body, and logs an ERROR line naming the failure.

Actual Behavior

The proxy returns status=200 in 17 ms with an empty body, having never opened an upstream connection, and
writes no ERROR line. The client cannot parse the response and drops the connection.

Code Sample

No Python API involved — this is the headroom proxy CLI in front of a Claude Code client.

# Not applicable. Reproduced via CLI:
#   headroom proxy --port 8787
#   headroom wrap claude
# The failing request is generated by Claude Code (claude-cli/2.1.232), not by user code.

Error Output

Client side (Claude Code):

API Error: API returned an empty or malformed response (HTTP 200) — check for a proxy or gateway intercepting the request

A subagent in the same session died with Agent terminated early due to an API error.

Proxy side — the entire log footprint of the failing request hr_1786680242_000147
(2026-08-13 23:04:02, proxy.log lines 6067–6071), client 127.0.0.1:61084, user-agent: claude-cli/2.1.232,
POST /v1/messages?beta=true, model=claude-opus-5, msgs=140, content_length=284080:

event=proxy_inbound_request  ... content_length=284080
event=outbound_headers forwarder=anthropic_messages stripped_count=1
PERF msgs=140 tok_before=0 tok_after=0 tok_saved=0 opt_ms=13 total_ms=13 tok_out=0 ttfb_ms=0 transforms=none client=claude-code
STAGE_TIMINGS "compression_first_stage": null, "memory_context": null, "upstream_connect": null, "upstream_first_byte": null, "total_pre_upstream": 14.97
event=proxy_inbound_response status=200 duration_ms=17.31

A healthy request on the same proxy logs outbound_headers → beta_header_merge → pipeline →
tool_injection_decision → cache_breakpoints → outbound_request. This one stops after outbound_headers.
upstream_connect: null and compression_first_stage: null place the bail-out after read_request_json +
deep_copy and before compression — roughly 15 ms in. That is the step that reads per-session sticky
beta / sticky tool state.

The immediately preceding request hr_1786680217_000146 (23:03:37, total_ms=24796) succeeded, and is
the CCR conversion:

CCR: stream:true request has headroom_retrieve available; using buffered stream:false upstream request for server-side retrieval handling
CACHE-BUST: expected_cached=98,632 actual_read=0 tokens_lost=98,632 tokens_saved=718
CACHE-MISS-ATTRIBUTION: reason=prefix_change idle=25s ttl=300s prefix_changed=True ttl_exceeded=False

The whole log file contains zero ERROR lines across its entire life. The only WARNINGs are upstream
429s, the Kompress/Magika startup notices, and Could not parse output_tokens from SSE, estimating 28 from 1144 bytes.

Environment

• Headroom version: 0.35.0 (installed via uv as headroom-ai)
• Python version: 3.14.6 (system interpreter; the uv tool environment may pin its own)
• OS: Windows 11 Pro 10.0.26200
• LLM Provider: Anthropic (api.anthropic.com, model=claude-opus-5)
• Client: Claude Code, claude-cli/2.1.232
• Proxy invocation: headroom proxy --port 8787, standalone, no config file under ~/.headroom,
  default transforms

Additional Context

Paired line numbers, single proxy.log, three separate client sessions:

CCR buffered conversion	empty-200 death	gap (lines)
1838	1848	+10
1918	1927	+9
1946	1955	+9
2235	2275	+40
2554	2594	+40
6027	6069	+42

Other CCR: lines (workspace unresolved, Proactively expanded N context(s),
skipping proactive expansion append) are common and harmless — only the stream:true → buffered
stream:false conversion pairs with a death.

Ruled out on our side:

• Double-bound proxy port — verified single instance,
  17432 (launcher) → 20032 → 24620 (python, owns port 8787).
• Version drift — installed headroom-ai matches the reported 0.35.0.
• Client-side hooks — they never touch the HTTP path; a blocked hook becomes ordinary tool_result content
  client-side, before any API call.

Wrong levers, for the record, in case they help narrow the code path:

• --no-ccr-proactive-expansion does not help — the conversion fires on headroom_retrieve being
  available, not on expansion occurring.
• --no-ccr avoids it but costs lossy compression with no recovery path.
• --lossless looks like the correct workaround: no CCR retrieval marker is emitted, so no
  headroom_retrieve is needed and the buffered stream:false path never runs. Not yet confirmed by a long
  session at the time of writing.

Log excerpt available. proxy.log lines 5996–6106 verbatim with original line numbers, 47 KB — covers
the last three healthy requests, the CCR/cache-bust predecessor, the failure, and two minutes of following
silence. authorization / x-api-key / sk-ant-* redacted. Happy to attach on request.

Note for maintainers: proxy.log carries verbatim prompt and tool-call content elsewhere in the file, so
it cannot be shared wholesale.

[PrakharSrivastav]

Independent reproduction on macOS, plus a version bisect that may narrow the search.

I hit this today and localised it separately before finding this issue. I arrived at the same
signature you published (tok_before=0 ... transforms=none with no upstream event), which is a
good sign the characterisation is right. Everything below is offered as extra data rather than a
second report.

Environment

OS	macOS 26.4.1, arm64
Install	pipx, Python 3.14.6
Client	Claude Code 2.1.232
Model	claude-opus-5 (yours was a different model, so this looks model-independent)
Proxy	headroom proxy, default transforms, CCR enabled

You reported from PowerShell, so between us the failure is not OS-specific.

The part that may help most: it does not reproduce on 0.34.0

The CCR buffered conversion is not sufficient on its own. On 0.34.0 the same conversion ran
constantly on this machine and never killed a request.

Version	CCR buffered conversions	Empty-200 deaths
0.34.0 (three days of logs, ending immediately before the upgrade)	3,144	0
0.35.0 (17 minutes, same machine, same sessions, same config)	6	6

Nothing in my configuration changed across that boundary. I diffed the client config, the client
settings, the Headroom settings and the MCP ledger after upgrading, and all four were byte-identical
to backups taken before it.

That points at a regression between 0.34.0 and 0.35.0 on the buffered-CCR path, rather than at a
long-standing flaw in the conversion itself. If that is right, the diff between those two tags
should contain the cause.

One honest caveat on the 0.34.0 column: it is not perfectly clean. Four requests there share the
tok_before=0 marker, but they look like a different event — all four fired inside a 23 ms window
on a different model at msgs=2, which reads like one client giving up rather than the failure
described here. I mention it so the zero is not overstated.

Worth excluding when counting: passthrough:count_tokens entries also carry tok_before=0
legitimately, and there are enough of them to hide the real signal if they are not filtered out.
That cost me an hour of chasing a phantom background rate.

Pairing, by conversation

I can confirm your one-to-one correlation. Six conversions, six deaths, each death the next request
in the same conversation, spread across three separate conversations:

CCR conversion	Conversation	Death	Gap
09:37:32	A	09:37:42	10 s
09:37:35	B	09:37:52	17 s
09:40:42	A	09:40:53	11 s
09:44:35	B	09:44:40	5 s
09:46:41	C	09:46:45	4 s
09:51:01	A	09:51:06	5 s

A note for anyone else trying to reproduce this pairing: the proxy's own session_id in
STAGE_TIMINGS is per-request, not per-conversation — mine had 103 distinct values across 103
requests — so it cannot be used to group them. The client's session header is the stable key.

Where it stops, from the stage timings

One death, in full:

POST /v1/messages  content_length=406700  model=claude-opus-5  msgs=113
tok_before=0  tok_after=0  transforms=none
opt_ms=4  total_ms=4  tok_out=0  ttfb_ms=0
→ status=200 in 5.09 ms

{
 "pre_upstream_wait": 0.0132,
 "read_request_json": 1.7625,
 "deep_copy": 0.5668,
 "compression_first_stage": null,
 "memory_context": null,
 "upstream_connect": null,
 "upstream_first_byte": null,
 "total_pre_upstream": 4.6118
}

It reads the body and deep-copies it, then every later stage is null. So it stops after
deep_copy and before compression_first_stage, having never opened an upstream connection — and
still answers 200. That matches your "dies between outbound_headers and beta_header_merge"
placement.

Sizes on my six deaths: msgs= 110, 140, 59, 140, 394, 113. The largest body was 406,700 bytes.
Consistent with your observation that small requests are spared, though 59 messages is fairly modest,
so the threshold may be lower than it looks.

Agreeing with the second defect

The silent-success part cost me more time than the abort. Because it answers 200 and writes no
ERROR line, headroom doctor reported zero failures while sessions were unusable, and the proxy
log looked healthy. A 5xx with a body, or even a single WARNING, would have made this obvious
immediately.

Offer

I have rolled back to 0.34.0 and the failure is gone, so I have a machine that reproduces on demand
by upgrading. Happy to run a patch, test a specific commit between the two tags, or pull any
additional log fields that would help. Just say what would be useful.

[PatrickLerner]

I can verify also that I am seeing this bug on 0.35 and that downgrading to 0.34 resolves it for me.

[Parideboy]

I think I have found where these turns are dying, and it is not where the "no upstream call was made" symptom first suggests. #3024 has a fix and regression tests. Below is the reasoning in full, including the part I could not confirm from the repository alone and a cheap experiment that would settle it.

Short version

The proxy is answering these turns from its own response cache. It replays the cached entry's stored response headers verbatim, including the header that describes how the original upstream connection framed its body. When that header is transfer-encoding: chunked and the replayed response is a fixed-length body, the client is told to parse plain JSON as chunked frames, finds no valid chunk-size line, and reads an empty body out of an HTTP 200. Nothing logs the cache hit, which is why the turn looks like it simply did nothing.

How I located it

Three things in the posted logs point at the response-cache hit specifically, and nowhere else.

The shape of the PERF line. Every field in the death lines matches the RequestOutcome constructed in the cache-hit block of headroom/proxy/handlers/anthropic.py: all-zero token counters, transforms=none, and the unusual property that opt_ms equals total_ms. Those two are set from the same value there. Searching every overhead_ms= emission across headroom/proxy turns up no other Anthropic site with that shape.

The log ordering. In the death logs, PERF appears before STAGE_TIMINGS. Every other exit path calls the pre-upstream finalizer first, which emits STAGE_TIMINGS and only then the outcome. The cache-hit return is the one path that inverts the order.

No outbound request, and a pre-upstream total around 15ms. The cache-hit return is the only place in the Anthropic handler that returns a 200 with a body before the upstream is contacted at all.

Why an empty body comes out of a 200

On a hit, the handler builds a fresh response from the stored bytes and the stored headers. It was removing content-encoding, content-length and content-type, but nothing else. Everything remaining describes the producing connection, and transfer-encoding: chunked is the damaging one. RFC 9112 §6.1 says Transfer-Encoding overrides Content-Length, so a client that receives both will use the chunked reader on a body that is not chunked and read nothing.

Worth noting: the Rust proxy already lists that header as hop-by-hop, and the OpenAI handler's shared sanitizer had one call site passing transfer-encoding and connection by hand, so the need to strip it was already known in the codebase. It just was not in the shared list, and the two cache-hit sites were the ones where the omission outlives the request that caused it, because a poisoned entry keeps answering later turns until it expires.

Why the failures track CCR activity

This is the part that explains why --lossless and --no-ccr make it stop.

The cache read is gated on the request not being a streaming request. The cache write, on the Anthropic path, was not. The cache key contains no streaming flag either. Ordinarily that gap is harmless, because a streaming turn returns through the streaming path and never reaches the store site.

CCR's buffered-stream conversion is the exception. It takes a request the client sent as stream: true, forces stream: false on the upstream call so it can buffer and rewrite the reply, and then falls through past the streaming return to the store site. The response it stores was shaped by that forced flip plus CCR tool injection, but the key cannot distinguish it from an ordinary non-streaming reply. A later non-streaming caller with a matching key is then served a response built for a request it never made.

So CCR is not itself broken here. It is the only thing that puts an entry of this kind into the cache in the first place.

Why nothing showed up as an error

A cache-served turn contacts no upstream. It has no outbound_request line, no upstream stage timings, and all-zero token counters. That is byte-for-byte identical to what a turn that died would look like, and the cache-hit block emitted no log line of its own. The PERF line also had no field for the cache flag, even though the outcome object already carried it. That is why headroom doctor reported zero failures while turns were dying.

What I could not confirm, and how you can

I cannot tell from the repository whether your upstream is actually sending transfer-encoding: chunked. Talking to Anthropic directly you would be on HTTP/2, where that header is forbidden. But any HTTP/1.1 hop in between reintroduces it: a corporate proxy, an --upstream pointed at a third-party gateway, a local relay, some VPN clients.

There is a cheap experiment that settles whether the cache path is involved at all, and it doubles as a better workaround than the ones found so far:

headroom proxy --no-cache

Run unmodified 0.35.0 with that flag. If the empty 200s stop, the localization is confirmed. It is a lighter workaround than --lossless or --no-ccr because compression and CCR both stay on; only the response cache is off. If the failures continue under --no-cache, my analysis is wrong and I would like to know, because the log ordering would then need another explanation.

What the PR changes

1. Moves the response-header sanitizer into a shared helper and adds the framing headers it was missing (transfer-encoding, connection, keep-alive), so both cache-hit sites strip exactly the same set as every other forwarding site.
2. Gates the Anthropic cache.set on the client's original streaming flag, mirroring the read side, so a buffered CCR turn no longer writes an entry a different caller can be served.
3. Logs RESPONSE-CACHE-HIT on a hit and adds cached=1 to the PERF line, so a cache-served turn is no longer indistinguishable in the logs from a dead one. The field is appended only on a hit, so existing PERF parsers are unaffected.

One related note on timing: #2953 is merged but not in a release yet, and it already narrows one half of this by refusing to store a response whose body did not parse as JSON. That stops an SSE body being cached. It does not stop a JSON-bodied response from carrying the producing turn's framing headers into the cache, and it does not add the streaming gate, so the two changes are complementary rather than overlapping.

[pyousefi]

Independent reproduction on Linux, which may be useful given the report is from PowerShell — the behaviour is identical, so it is not platform-specific.

Environment: headroom 0.35.0 (pipx, Python 3.13), Ubuntu 24.04, Claude Code 2.1.233, claude-sonnet-5, proxy started by headroom wrap claude (not standalone).

Match against the signature in this issue

Claim in the issue	This log
death is the request after a CCR buffered conversion	conversion on …_000010 at 21:00:20.719, death on …_000011 at 21:00:24.005
aborts between outbound_headers and beta_header_merge	…_000011 logs outbound_headers and then no beta_header_merge; …_000009 and …_000010 both log one
PERF … tok_before=0 tok_after=0 … transforms=none	matches, exactly once in the file
conversions and deaths pair 1:1	1 conversion, 1 death
proxy never contacts upstream	no outbound_request line for …_000011
returns 200	status=200 duration_ms=15.12 on a 216 KB request
no ERROR line	confirmed — the only WARNING/ERROR lines in the whole file are two unrelated upstream 429s

Counted over /v1/messages only: 15 inbound, 14 forwarded upstream, exactly one never forwarded. (The other unforwarded inbound requests in the file are /api/hello probes returning 404 — worth excluding when counting, or the drop looks larger than it is.)

One datapoint that widens the reported range

The issue notes "five of the six deaths landed at msgs=35, the sixth at msgs=140… Never on small requests."

This death landed at msgs=23, content_length=216304. Still not a small request by body size, but below the msgs=35 floor observed so far — so if there is a message-count threshold it is lower than 35, and body size looks like the better predictor than message count.

Log excerpt

Project name and session identifiers scrubbed; otherwise verbatim.

21:00:19,782 - headroom.proxy - INFO - event=outbound_headers forwarder=anthropic_messages stripped_count=1 request_id=hr_1786827619_000010
21:00:19,793 - headroom.proxy - INFO - event=beta_header_merge provider=anthropic session_id=<scrubbed> client_betas=12 sticky_betas=12 headroom_added= request_id=hr_1786827619_000010
21:00:20,719 - headroom.proxy - INFO - [hr_1786827619_000010] CCR: stream:true request has headroom_retrieve available; using buffered stream:false upstream request for server-side retrieval handling
21:00:20,720 - headroom.proxy - INFO - event=outbound_request forwarder=anthropic_messages method=POST path=/v1/messages body_bytes=216318 body_mutated=true mutation_reasons=structural_diff_vs_original,ccr_streaming_retrieve_buffered_non_stream source=passthrough request_id=hr_1786827619_000010
21:00:21,722 - headroom.proxy - INFO - event=proxy_inbound_response id=<scrubbed> method=POST path=/v1/messages status=200 duration_ms=1950.44

  (client sends the next request; this is the one that dies)

21:00:23,991 - headroom.proxy - INFO - event=proxy_inbound_request id=<scrubbed> method=POST path=/v1/messages query=beta=true client=127.0.0.1:42798 content_length=216304 ... "user-agent": "claude-cli/2.1.233 (external, cli)"
21:00:24,000 - headroom.proxy - INFO - event=outbound_headers forwarder=anthropic_messages stripped_count=1 request_id=hr_1786827623_000011
21:00:24,005 - headroom.proxy - INFO - [hr_1786827623_000011] PERF model=claude-sonnet-5 msgs=23 tok_before=0 tok_after=0 tok_saved=0 tok_inflated=0 tool_saved=0 total_saved=0 cache_read=0 cache_write=0 cache_hit_pct=0 opt_ms=12 total_ms=12 tok_out=0 ttfb_ms=0 transforms=none client=claude-code
21:00:24,005 - headroom.proxy - INFO - event=proxy_inbound_response id=<scrubbed> method=POST path=/v1/messages status=200 duration_ms=15.12

Client-side the session ends with:

API Error: API returned an empty or malformed response (HTTP 200) — check for a proxy or gateway intercepting the request

Triage shortcut

Since the failure logs nothing at ERROR and answers 200, the quickest way to confirm it in someone else's log is to check that inbound and forwarded counts disagree:

grep -c 'proxy_inbound_request .*path=/v1/messages' proxy.log
grep -c 'outbound_request' proxy.log

Agreed on the second defect being the expensive one — a failure that returns success and logs nothing cost us a while to localise too, and the 200 in particular means client-side retries do not kick in.