Feat/ccr reverse stats - #17
Merged
Merged
Conversation
…ssor entry point (headroomlabs-ai#2370) ## What Adds a pluggable compressor registry and a `headroom.compressor` entry-point group so compressors can be registered, discovered, and selected by name. - Pure-data contract (`CompressorDescriptor`, `CompressInput`, `CompressOutput`, `Compressor` Protocol). Only plain types (`str`/`int`/`bool`/`list`/`dict`) cross the boundary — no tokenizer, store, or config objects — so the same contract can be implemented outside Python. - `CompressorRegistry`: starts empty and accepts explicit registrations by name; discovers external compressors from the `headroom.compressor` group fail-open (mirrors the existing pipeline-extension discovery); resolves an opt-in selection (`select`/`active`) — nothing active by default, `"*"` for all, otherwise a name allowlist with unknown names logged and skipped. Discovery loads compressors but never invokes `compress`. ## Why Compressors are currently constructed and dispatched via a hardcoded chain in the content router; there is no way to add or select one without editing the router. This lands a name-addressable seam so that becomes possible. ## Behavior change None. Purely additive — not wired into `content_router`, the proxy server, or config, and constructing the registry has no global side effects. Router integration is a deliberate follow-up. ## Testing - `pytest tests/test_compressor_registry.py -q` → 11 passed (contract round-trip, registration, opt-in selection semantics, wildcard, unknown-name skip, monkeypatched entry-point discovery, discovery-never-runs-compress). - `ruff check` / `ruff format` → clean; `mypy` → no issues.
## What Adds `PrometheusMetrics.record_extension_savings(key, saved)` so proxy extensions can report the tokens they save, and surfaces the per-extension totals in the `/stats` payload. ## Why Proxy extensions that perform their own token reduction currently have no supported way to report their savings to the metrics object — there is no method for it, so that telemetry is silently dropped. This adds the recording method and exposes the aggregate alongside the existing per-strategy compression breakdown. ## How - New `extension_savings: dict[str, int]` counter on `PrometheusMetrics`, populated lazily per extension-supplied `key` (no hardcoded list of extensions). - `record_extension_savings(key, saved)` accumulates positive savings per key, mirroring how `record_compression` aggregates `tokens_saved_by_strategy` (lock-free `defaultdict(int)`, atomic under the GIL for these key types); non-positive values are ignored. - Cleared in `reset_runtime()` with the other in-memory counters. - Surfaced in `/stats` as `extension_savings`, next to `compressions_by_strategy` / `tokens_saved_by_strategy`. No new Prometheus series. ## Behavior change None to existing metrics. ## Testing - Two focused tests in `tests/test_compression_observability.py` (per-key accumulation incl. zero/negative ignored; surfaced in `/stats` via `create_app`) → 2 passed (13 in file). - `ruff check` / `ruff format` → clean; `mypy` → clean on changed source.
…inventory (headroomlabs-ai#2373) ## What - Adds an opt-in `--compressor` / `HEADROOM_COMPRESSORS` selection that narrows the active built-in compressors, mapped onto the existing `ContentRouterConfig` `enable_*` flags at the proxy config seam. Recognized names: `smart_crusher, kompress, code_aware, search, log, tabular, config, html, image`; `"*"` selects all. - Builds a name-addressable compressor registry in `ContentRouter`: a metadata-only descriptor per built-in plus opt-in discovery of `headroom.compressor` entry points (the seam added in headroomlabs-ai#2370). ## Why Built-in compressors were only reachable through a hardcoded if/elif; there was no supported way to select a subset (7 of the `enable_*` flags had no external surface) or to see the built-ins as a name-addressable set alongside third-party ones. ## Behavior change **None by default.** `--compressor` unset (the default) leaves every `enable_*` flag at its dataclass default, so the request path is byte-identical to today. The registry is inventory-only — built-ins are still constructed and dispatched by the existing if/elif; `_BuiltinCompressorEntry.compress` deliberately raises (never called), and registry construction is fail-open. Routing an external compressor *through* the pipeline is a deliberate follow-up. ## How - `server.py`: `BUILTIN_COMPRESSOR_FLAGS` map + `_apply_compressor_selection(router_config, compressors)` (no-op when `None`/empty; runs before the `disable_kompress` override so that stays authoritative). - `models.py`: `ProxyConfig.compressors: set[str] | None = None`. - `cli/proxy.py`: `--compressor` (repeatable, comma-split, `HEADROOM_COMPRESSORS`), mirroring `--proxy-extension`. - `content_router.py`: built-in descriptors + `_build_compressor_registry()` (register built-ins, then fail-open `discover()`), exposed as `self.compressor_registry`. Dispatch unchanged. ## Testing `tests/test_compressor_selection.py` — 23 tests: selection mapping (None/empty/whitespace = byte-identical defaults, single/multi/wildcard, external-only disables built-ins, unrecognized ignored), `ProxyConfig` field, registry inventory (descriptors cover the 9 names, valid cost tiers, router exposes registry, inventory doesn't auto-activate, built-in `compress` guard, discovery merges external, fail-open on discovery error). Local: 23 passed; ruff + mypy clean on changed files. Full suite runs in CI. Stacks conceptually on headroomlabs-ai#2370 (registry seam); rebased onto `main` after that merged. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
…droomlabs-ai#2385) ## Description A `--compressor` selection that matches no built-in name (e.g. `smart_krusher`, a typo of `smart_crusher`) silently disables **all** built-in compression: the proxy starts healthy, the dashboard shows ~0 savings, and nothing explains why. The all-off *semantics* is deliberate and stays untouched — `test_only_external_name_disables_all_builtins` pins the opt-in "exactly these" contract, and external/registry names are a legitimate input class. What's missing is any **signal**: a typo and an external compressor name are indistinguishable at this seam, and the registry's own unregistered-name warning (`CompressorRegistry.select`) never runs on this path. Fix: `_apply_compressor_selection` now logs one warning when the selection contains unmatched names — - **nothing matched** (the typo case): says plainly that every built-in compressor is now disabled and lists the valid names + `*`; - **mixed**: names the unmatched entries as assumed registry names. Selection results are byte-identical before/after. Fixes headroomlabs-ai#2384. ## Type of Change - [x] Bug fix (observability for a silent misconfiguration; no behavior change) ## Changes Made - `headroom/proxy/server.py`: `_apply_compressor_selection` computes the unmatched set and emits one `headroom.proxy` warning (two phrasings: nothing-matched vs mixed); docstring updated. - `tests/test_compressor_selection.py`: 3 new tests — typo-only selection warns (and flags stay all-off, pinning the unchanged contract), mixed selection warns only about the unmatched name, matched/wildcard selections stay warning-free. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff` + `mypy`, CI-pinned settings) - [x] Wrote the failing tests first, then the warning ### Test Output ```text $ .venv/bin/python -m pytest tests/test_compressor_selection.py -q 26 passed # Before the fix the two new warning tests fail (no log records emitted). $ ruff check headroom/proxy/server.py tests/test_compressor_selection.py # All checks passed! $ ruff format --check <both> # formatted $ mypy headroom/proxy/server.py --ignore-missing-imports # Success: no issues ``` ## Real Behavior Proof - Environment: macOS (Darwin), Python in a uv venv, branch `fix/compressor-selection-warn` off `main` (`56c7d4a5`). - Exact command / steps: configured stdlib logging at WARNING and called `_apply_compressor_selection(ContentRouterConfig(), {"smart_krusher"})` — the exact typo scenario from headroomlabs-ai#2384. - Observed result: `WARNING headroom.proxy: compressor selection smart_krusher matches no built-in compressor — every built-in compressor is now disabled. If this is a typo, valid names are: code_aware, config, html, image, kompress, log, search, smart_crusher, tabular (or '*' for all).` with `enable_smart_crusher = False` (contract unchanged). Before the fix the same call produced zero log output. - Not tested: a full `headroom proxy --compressor smart_krusher` process launch; the seam is exercised directly and the proxy wires it unconditionally. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — N/A (docstring updated) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A ## Additional Notes - Deliberately warn-only: erroring here would break legitimate external/registry selections and could brick startup on a stale `HEADROOM_COMPRESSORS` settings value. If you'd rather hard-fail just the CLI-typed path, happy to follow up.
…omlabs-ai#2383) ## Description `headroom wrap codex` with a custom provider emits `--config` overrides whose dotted key quotes **every** segment (`"model_providers"."litellm_prod"."base_url"=…`). Codex's override parser matches dotted segments literally and silently ignores quoted ones, so the overrides are dropped, the session keeps the provider's real `base_url`, and traffic **bypasses Headroom entirely** — the exact silent-bypass reported in headroomlabs-ai#2358 on Codex 0.144.5. I reproduced the parser behavior differentially on a local Codex **0.144.1** (see Real Behavior Proof): a bare key is parsed (Codex errors on the injected value), the same key quoted produces no error at all — the override is silently discarded. Fix: `_codex_dotted_key()` now emits segments **bare** whenever they are valid bare keys (`[A-Za-z0-9_-]+`, which covers `model_providers`, every normal provider id, and hyphenated header names like `X-Headroom-Base-Url`), and quotes only segments where bare emission would corrupt the dotted path (e.g. a provider id containing a dot). Fixes headroomlabs-ai#2358. ## Type of Change - [x] Bug fix (silent proxy bypass for custom-provider Codex wraps) ## Changes Made - `headroom/cli/wrap.py`: `_codex_dotted_key()` quotes only non-bare-key segments; docstring explains the observed Codex parser behavior. The default `openai` path (`openai_base_url=…`, already bare) is unchanged. - `tests/test_cli/test_wrap_codex.py`: the custom-provider launch test now pins the bare form for all three overrides (`base_url`, `supports_websockets`, `env_http_headers.X-Headroom-Base-Url`), plus 2 direct unit tests (bare-when-safe incl. hyphens, quote-only-unsafe). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff` + `mypy`, CI-pinned settings) - [x] Reproduced the parser behavior on a real Codex CLI first, then fixed ### Test Output ```text $ .venv/bin/python -m pytest tests/test_cli/test_wrap_codex.py -q 93 passed # Before the fix the updated assertions fail (generated args still fully quoted): # FAILED ...::test_codex_session_launch_settings_preserve_custom_provider_identity # FAILED ...::test_codex_dotted_key_emits_bare_segments_when_safe # FAILED ...::test_codex_dotted_key_quotes_only_unsafe_segments $ ruff check headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py # All checks passed! $ ruff format --check <both> # formatted $ mypy headroom/cli/wrap.py --ignore-missing-imports # Success: no issues ``` ## Real Behavior Proof - Environment: macOS (Darwin), Codex CLI **0.144.1** (`/opt/homebrew/bin/codex`), empty temp `CODEX_HOME` (no auth, no network side effects), branch `fix/codex-config-bare-keys` off `main` (`56c7d4a5`). - Exact command / steps: differential probe of the override parser — `codex exec --skip-git-repo-check -c 'profile="__nope__"' 'hi'` (bare key) vs `codex exec --skip-git-repo-check -c '"profile"="__nope__"' 'hi'` (quoted key), each run once against a fresh empty `CODEX_HOME`. - Observed result: bare key → Codex **parsed the override** and failed fast on it (`Error: legacy profile = "__nope__" config is no longer supported…`); quoted key → **no error referencing the override at all**, Codex proceeded to start a session (banner printed) — the quoted override was silently discarded. That is precisely the headroomlabs-ai#2358 bypass mechanism: every generated custom-provider override was quoted, hence dropped, hence traffic went straight to the real upstream. - Not tested: an end-to-end wrapped session against a live LiteLLM upstream on Codex 0.144.5 (the reporter's exact version); the parser behavior above is version-adjacent (0.144.1) and the argv shape is pinned by unit tests. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — N/A (docstring added) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A ## Additional Notes - Segments that genuinely need quoting (a provider id with a dot) keep their quotes: on parsers that ignore quoted segments those overrides still won't apply, but bare emission would corrupt a *different* key path, which is strictly worse. Such ids are rare; the common LiteLLM/custom-provider case is fully bare after this fix.
…kpoints (headroomlabs-ai#2382) ## Description `normalize_message_cache_control()` consolidates message-level `cache_control` breakpoints (strip all, re-place exactly one) to stay under Anthropic's 4-block limit. The re-placed marker was hardcoded to `{"type": "ephemeral"}`, so a client using 1-hour caching (`cache_control: {"type": "ephemeral", "ttl": "1h"}`) was silently downgraded to the 5-minute default on every consolidated turn — no error, no signal, just quietly worse cache economics. Fix: track the newest client marker while stripping, and re-place **that marker verbatim** (a copy). Headroom keeps owning *where* the breakpoint goes; the client keeps owning *what it says*. Older replayed markers don't win — if the client's newest marker has no `ttl`, we don't resurrect a stale `1h` (covered by a dedicated regression test). Fixes headroomlabs-ai#2375. ## Type of Change - [x] Bug fix (silent 1h→5m cache downgrade) ## Changes Made - `headroom/cache/prefix_tracker.py`: `normalize_message_cache_control()` records the last marker dict seen in message order and re-places a copy of it instead of a hardcoded `{"type": "ephemeral"}`; docstring documents the ownership split. - `tests/test_cache_control_move_bust.py`: 3 new tests — ttl preserved, newest-marker-wins over stale ttls, ttl survives an 8-turn conversation loop. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff` + `mypy`, CI-pinned settings) - [x] Reproduced the bug first (2 new tests failed on the old code), then verified the fix ### Test Output ```text $ .venv/bin/python -m pytest tests/test_cache_control_move_bust.py -q 10 passed # Before the fix, the two new ttl tests fail exactly as headroomlabs-ai#2375 describes: # FAILED ...::test_normalize_preserves_ttl_of_newest_marker # FAILED ...::test_normalize_ttl_survives_many_turns $ ruff check headroom/cache/prefix_tracker.py tests/test_cache_control_move_bust.py # All checks passed! $ ruff format --check <both files> # already formatted $ mypy headroom/cache/prefix_tracker.py --ignore-missing-imports # Success: no issues ``` ## Real Behavior Proof - Environment: macOS (Darwin), Python in a uv venv, branch `fix/cache-control-ttl-preserve` off `main` (`56c7d4a5`). - Exact command / steps: drove `normalize_message_cache_control` directly with a 2-message conversation whose marker carries `ttl: "1h"`, printed the re-placed marker before/after the fix, and ran the new regression tests against the unfixed code first. - Observed result: before — output marker `{'type': 'ephemeral'}` (ttl silently dropped); after — output marker `{'type': 'ephemeral', 'ttl': '1h'}` with marker count still exactly 1 (the ≤4-block guarantee is untouched). - Not tested: a live Anthropic round-trip asserting `cache_creation.ephemeral_1h_input_tokens` (needs a billed API call); the marker dict forwarded on the wire is what the assertion pins. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — N/A (docstring updated) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A ## Additional Notes - The `test_normalize_newest_marker_wins_over_stale_ttl` test also guards against over-fixing (e.g. "any 1h seen anywhere wins"), which would pin users to 1h pricing after they switch back to the default.
… (headroomlabs-ai#2378) ## Description Structured payloads still leave long prose leaves without a dedicated prose compressor. The Rust pipeline already handles top-level log, diff, search, and JSON-array shapes, and the existing structured recursion rewrites stringified JSON and opaque blobs, but a plain prose string leaf inside structured content still falls back to generic opaque long-string handling instead of query-aware extractive compression. That wastes prompt budget on fields like `summary`, `description`, and `analysis` even though `headroom-core` already ships the deterministic, query-aware `TextCrusher`. This PR adds a bounded prose-field path for structured leaves. It introduces a reusable `ProseFieldOffload` backed by `TextCrusher`, then wires that offload into `JsonOffload`'s structured recursion with conservative byte and segment thresholds. Only detector-confirmed `PlainText` leaves are eligible. When a leaf clears those gates and the marker-inclusive output still saves bytes, the exact original leaf is written to CCR and the inline output carries a prose marker keyed to that store entry. Short prose, low-segment prose, diff-shaped strings, stringified JSON, and opaque base64 or HTML keep their existing behavior. This stays inside the Rust transform stack. It does not add a PyO3 shim, ONNX runtime, live-zone prose handling, or any new Python dependency. It also keeps the existing wrapper-level `JsonOffload` CCR entry, so the full structured payload remains recoverable as before. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `ProseFieldOffload` as a `ContentType::PlainText` pipeline offload backed by `TextCrusher`, with conservative byte, segment, and target-ratio thresholds. - Thread the prose offload into the structured `JsonOffload` recursion so nested prose leaves can compress and recover through the orchestrator store. - Add a pipeline-aware `JsonOffload::from_pipeline` constructor so `offload.prose_field` overrides actually reach the live prose hook instead of falling back to embedded defaults. - Preserve current behavior for short prose, low-segment prose, diff-shaped leaves, stringified JSON containers, and opaque base64 or HTML leaves. - Add focused config, routing, determinism, and CCR roundtrip coverage for the new prose path. - Leave changelog generation to the repo's conventional-commit release flow rather than editing `CHANGELOG.md` directly. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core --lib transforms::pipeline::offloads::prose_field::tests`) - [x] Linting passes (`cargo clippy -p headroom-core -- -D warnings`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text cargo fmt --all -- --check cargo clippy -p headroom-core -- -D warnings cargo test -p headroom-core --lib transforms::pipeline::offloads::prose_field::tests test result: ok. 6 passed; 0 failed cargo test -p headroom-core --lib transforms::pipeline::offloads::json_offload::tests test result: ok. 17 passed; 0 failed cargo test -p headroom-core --lib transforms::smart_crusher::crusher::tests::default_crush_ignores_opt_in_prose_hook -- --exact test result: ok. 1 passed; 0 failed cargo test -p headroom-core --lib transforms::smart_crusher::crusher::tests::prose_hook_preserves_html_opaque_routing -- --exact test result: ok. 1 passed; 0 failed cargo test -p headroom-core --lib transforms::smart_crusher::crusher::tests::prose_hook_runs_for_dict_array_rows -- --exact test result: ok. 1 passed; 0 failed cargo test -p headroom-core --lib transforms::smart_crusher::crusher::tests::unchanged_stringified_json_container_skips_prose_hook -- --exact test result: ok. 1 passed; 0 failed cargo test -p headroom-core --test ccr_roundtrip nested_structured_prose_leaf_uses_ccr -- --exact test result: ok. 1 passed; 0 failed git diff --check ``` ## Real Behavior Proof - Environment: Windows 11, stable Rust toolchain, in-memory CCR store, no live provider - Exact command / steps: run the focused nested CCR roundtrip test through `CompressionPipeline::run` on a five-row structured payload containing a long prose leaf, then resolve the emitted prose marker key from the same orchestrator store - Observed result: the generic `CompressionPipeline` plus `JsonOffload` path applies, the nested prose leaf becomes shorter on the wire, and that prose key retrieves the byte-identical original leaf from the orchestrator store while HTML-shaped and diff-shaped leaves stay on their opaque marker routes - Not tested: live provider run ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - The upstream issue body originally parked PR3b behind a PyO3 shim or a later ONNX port. This PR takes the narrower Rust-native path instead by reusing the existing `TextCrusher` already in `headroom-core`. - This PR advances the pipeline-side PR3b slice from headroomlabs-ai#334. It does not close headroomlabs-ai#334, and it does not wire live-zone or PyO3 SmartCrusher callers to this path. - `CHANGELOG.md` is intentionally untouched because this repo's release pipeline generates changelog entries from conventional commits, and `repos/headroom/config.md` marks manual changelog edits as out of policy. - Python lint, type checking, and pytest are not part of the focused local proof for this slice because the change stays inside `crates/headroom-core`.
headroomlabs-ai#2377) ## Description Requests routed to the GitHub Copilot API travel on the OpenAI or Anthropic wire, so the proxy handlers stamp the *wire* provider (`openai` / `anthropic`) on the outcome. As a result, Copilot traffic is attributed to OpenAI/Claude in the dashboard's per-request provider stats, hiding the real upstream. (This is distinct from the existing **Copilot Quota** panel, which is separate from per-request provider attribution.) This labels Copilot traffic as `copilot` in the single outcome funnel. `build_copilot_upstream_url()` is already the one routing chokepoint every Copilot surface goes through (OpenAI `/chat/completions` + `/responses` and the Anthropic `/v1/messages` route all build their upstream URL there), so it flags the request via a task-local `ContextVar`; `emit_request_outcome()` reads the flag and relabels the provider. The relabel runs before the `>= 500` failed guard, so a failed Copilot request is attributed to `copilot` too. Non-Copilot traffic never sets the flag and is untouched. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - `headroom/copilot_auth.py`: add a task-local `_request_routed_to_copilot` `ContextVar` with `mark_request_routed_to_copilot()` / `request_routed_to_copilot()` helpers; set the flag in `build_copilot_upstream_url()` whenever the base is a Copilot API URL (the existing `is_copilot_api_url` check). `/v1` path normalization is unchanged. - `headroom/proxy/outcome.py`: in `emit_request_outcome()`, when the request was routed to Copilot and the wire provider is `openai`/`anthropic`, relabel the outcome provider to `copilot` (before the 5xx guard). - `tests/test_copilot_provider_label.py`: new tests for the chokepoint marking and the outcome relabel. ## Testing - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) — ran on the changed files only (clean) - [ ] Type checking passes (`mypy headroom`) — ran on the changed files only (clean) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_copilot_provider_label.py tests/test_outcome_records_5xx_as_failed.py -q tests/test_copilot_provider_label.py ..... [ 71%] tests/test_outcome_records_5xx_as_failed.py .. [100%] 7 passed $ python -m pytest tests/test_copilot_auth.py -k "build_copilot_upstream_url or copilot_api_url" -q 8 passed, 58 deselected # existing /v1-stripping behavior preserved $ python -m ruff check headroom/copilot_auth.py headroom/proxy/outcome.py tests/test_copilot_provider_label.py All checks passed! $ python -m mypy headroom/copilot_auth.py headroom/proxy/outcome.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: Python 3.11, headroom installed with the `proxy` extra. - Exact command / steps: the unit tests above drive `build_copilot_upstream_url()` followed by `emit_request_outcome()` in an isolated context and assert the recorded provider. - Observed result: an `anthropic`/`openai` outcome for a request routed to `https://api.githubcopilot.com` is recorded as provider `copilot`; a request not routed to Copilot is recorded under its wire provider unchanged. - Not tested: end-to-end against a live Copilot subscription (no live seat in the test environment). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes - The flag is a `ContextVar` (task-local), so it cannot bleed across concurrent requests; each request that is not routed to Copilot simply reads the `False` default. - No `CHANGELOG.md` edits (release-please generates it from the Conventional Commit PR title). --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
…router (headroomlabs-ai#2388) ## What Scope 3 of the pluggable-compressor system: a **selected external `headroom.compressor` plugin now compresses real traffic**. Opt-in via `--compressor` / `HEADROOM_COMPRESSORS` — external (non-built-in) names flow to `ContentRouterConfig.active_external_compressors`, resolved once against the registry in `__init__` (built-in inventory entries filtered out). ## How A single guarded branch at the top of `_apply_strategy_to_content`, immediately before the built-in if/elif. When a selected external compressor declares the block's detected content type (exact MIME, `text/*`, or `*` wildcard), the block runs through the pure-data `Compressor` contract; otherwise it falls through to the built-in path unchanged. ## Cache safety (by construction) The branch lives **inside the per-block strategy dispatch**, which only runs on non-frozen, already-compressible blocks — the frozen/cached prefix is split off upstream in `apply()`. So a selected external compressor **can never rewrite cached-prefix content and bust the prompt cache**; it inherits the exact same cache-preservation the built-ins have. ## Fail-open + fidelity Raise, malformed/non-`CompressOutput`, empty-from-non-empty, or expansion all fall back to the built-in path. Tokens are counted with the router's own estimator (never the compressor's self-report). Any `recoverable` (hash→original) map is mirrored to the CCR store like SmartCrusher, so `/v1/retrieve/{hash}` resolves. Reached only in lossy/CCR mode (lossless-only sessions return earlier), so it can't inject unrecoverable loss. ## Behavior change **None by default.** With no external compressor selected, the branch is a single cheap guard and everything below is byte-identical to today. ## Testing `tests/test_router_external_dispatch.py` — end-to-end dispatch of a selected external compressor, recoverable-map retrievability, non-hex-hash skip, fail-open on raise/malformed/empty/expansion, not-selected & non-matching-content-type leave the built-in path unchanged, wildcard selection. Offline suite: 84 passed (this file + selection + registry + settings_store). ruff + mypy clean. Note: the broad content-router/compression suite exercises real HF-Hub model downloads + local ONNX inference and is slow/flaky in some local envs — deferred to CI. Stacks on headroomlabs-ai#2370/headroomlabs-ai#2371/headroomlabs-ai#2373 (all merged).
…rner (headroomlabs-ai#2333) ## Description Addresses the chat/completions portion of headroomlabs-ai#2060. The live traffic learner is wired into the Anthropic `/v1/messages` handler and, since then, the OpenAI Responses HTTP handler (`_observe_openai_responses_traffic`, called from `handle_openai_responses`). But `handle_openai_chat` has **no** ingestion call site: ```text headroom/proxy/handlers/openai.py handle_openai_responses -> _observe_openai_responses_traffic (wired) handle_openai_chat -> (no traffic_learner call) (gap) ``` So OpenAI-compatible clients that route through `/v1/chat/completions` — GitHub Copilot CLI, opencode, OpenAI SDKs — run through an apparently healthy proxy with Learn enabled while producing no learned patterns: the learner starts, but it never receives their tool results or user messages. ## Fix Observe the original client payload (before memory/compression mutates it) at the top of `handle_openai_chat`, mirroring the Responses and Anthropic ingestion paths: ```python await self._observe_openai_chat_traffic(original_client_messages, request_id=request_id) ``` `_observe_openai_chat_traffic` is the chat counterpart of `_observe_openai_responses_traffic`: same lazy backend wiring, same `on_tool_result` / `on_messages` lifecycle, same fail-soft `try/except`. The one format-specific piece is tool-result extraction. chat/completions encodes tool calls differently from Anthropic — the call is on an assistant message's `tool_calls` array (`id` -> function `name` + `arguments`) and each result is a separate `role: "tool"` message keyed by `tool_call_id`, so the existing `extract_tool_results_from_messages` (which scans for Anthropic `tool_use`/`tool_result` blocks) finds nothing. A new `TrafficLearner.extract_tool_results_from_openai_messages`: - builds the `tool_call_id -> function` map from assistant `tool_calls`; - for each `role: "tool"` message, resolves the tool name and joins string-or-list content; - parses the OpenAI `arguments` JSON string into a dict, so the downstream environment/recovery extractors (which call `input.get("command")`, `input.get("file_path")`, ...) see the same shape as an Anthropic `tool_use.input` instead of a raw string; - sniffs `is_error` from the output (chat tool messages carry no error flag). It returns the same `{tool_name, input, output, is_error}` shape as the Anthropic extractor, so `on_tool_result` stays format-agnostic. User-message preference extraction (`on_messages`) already reads plain `role`/`content`, so it consumes chat messages unchanged. Scope: this wires the **chat/completions** path. Codex WebSocket ingestion (`handle_openai_responses_ws`) additionally needs per-`response.create` evaluation plus transcript-replay baselining on reconnect, so it is intentionally left as a follow-up rather than half-implemented here. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/memory/traffic_learner.py`: add `extract_tool_results_from_openai_messages` (OpenAI chat tool-result extraction with `arguments` JSON parsed to a dict). - `headroom/proxy/handlers/openai.py`: add `_observe_openai_chat_traffic` and call it from `handle_openai_chat` on the original client payload. - `tests/test_memory/test_traffic_learner.py`: cover the OpenAI extractor (name resolution, arguments parsing, list content, error sniff, malformed/orphan handling, empty case). - `CHANGELOG.md`: Bug Fixes entry. ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/memory/traffic_learner.py headroom/proxy/handlers/openai.py tests/test_memory/test_traffic_learner.py All checks passed! $ uvx ruff@0.15.17 format --check <same files> 3 files already formatted $ uvx mypy@1.20.2 --ignore-missing-imports headroom/memory/traffic_learner.py # clean for this file (the one reported error is a pre-existing # headroom/_subprocess.py:18 no-any-return, unrelated to this change and # present on main with these edits stashed) ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx mypy@1.20.2`. A full `pytest` OOMs this box (ML-stack import), so I reproduced the extractor with a dependency-free script and left the full pytest to CI. - Exact command / steps: replicated `extract_tool_results_from_openai_messages` and ran it over a typical chat round-trip (assistant `tool_calls` for `bash` + `read_file`, then two `role: "tool"` results, one erroring and one with list content), plus malformed-`arguments`, orphan-`tool_call_id`, and no-tool cases. - Observed result: tool names resolved from the call-id map; `arguments` parsed to a dict so `input.get("command")` works; list content joined; `is_error` sniffed from output; malformed arguments degrade to `{}` and an orphan id yields `unknown` without raising. The added unit tests assert the same through a real `TrafficLearner`. - Not tested: a live Copilot CLI session end to end; the added tests drive `TrafficLearner.extract_tool_results_from_openai_messages` directly, matching the existing `test_extract_tool_results_from_messages` pattern. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes The "unit tests pass locally" box is unchecked because a local pytest run imports the ML stack and OOMs this box; the added tests reuse the existing `TrafficLearner(backend=None, ...)` harness in `test_traffic_learner.py` (no real backend) and run under the normal CI pytest job, and the extractor behavior is corroborated by the standalone proof above. This PR is deliberately scoped to `/v1/chat/completions`; I'm happy to follow up with the Codex WebSocket ingestion path (which needs the transcript-replay baselining discussed in the issue) as a separate change if useful. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
…ations (adapters) (headroomlabs-ai#2391) ## What Turns each built-in registry entry into a working `Compressor` (the `compressor_registry` contract): `compress(CompressInput) -> CompressOutput` delegates to the same underlying built-in method the content router already invokes in `_apply_strategy_to_content`, reached through the router's own `_get_*` getter so config flows through identically. Token counts use the router's `_estimate_tokens`; `lossless` mirrors the descriptor; `recoverable` is `{}` (built-ins persist CCR recovery to the store as a side effect, not via their return value). Adapted: `smart_crusher, code_aware, search, log, tabular, config, html, kompress`. ## Behavior change **None — additive by construction.** Dispatch, the `_get_*` getters, fallback chains, the reversibility gate, and config are all unchanged. The router still dispatches built-ins via its existing if/elif and never routes a request through the registry; `_resolve_active_external_compressors` filters built-in entries out of the opt-in external-dispatch path *by type* (the class name `_BuiltinCompressorEntry` is load-bearing). A default request is byte-identical: `_active_external_compressors == []`, external dispatch is an inert guard, and adapters are reachable only via `compressor_registry.get()/active()`. ## `image` — documented passthrough (not a guess) `ImageCompressor.compress(messages)` operates on image blocks inside message dicts, not `str` content, and isn't on the `_apply_strategy_to_content` path, so there's no faithful `str→str` delegation. Its adapter is a documented non-raising passthrough rather than a fabricated one. ## Testing `tests/test_builtin_compressor_adapters.py` — differential tests asserting each adapter's output matches the built-in's direct output (JSON→smart_crusher, CSV→tabular, log lines→log, grep→search, config→config, Python→code_aware, HTML→html); kompress is mocked (no ML inference); every registry entry has a working non-raising `compress`. Updated the obsolete guard test in `test_compressor_selection.py`. Offline suite: 72 passed; ruff + mypy clean. (Broad content-router/compression suite deferred to CI — it needs HF-Hub/ONNX model loads.) This is PR-A of the adapter phase (built-ins become Compressor implementations); flipping the router's dispatch to registry-resolved is the follow-up. Builds on headroomlabs-ai#2370/headroomlabs-ai#2371/headroomlabs-ai#2373/headroomlabs-ai#2388.
…pressor registry (headroomlabs-ai#2399) ## What Second increment of the adapter phase (builds on headroomlabs-ai#2391). Flips the content router's per-strategy dispatch in `_apply_strategy_to_content` from the hardcoded if/elif to **registry-resolved** — but only for the *clean, single-compressor* strategies: **SEARCH, LOG, TABULAR, CONFIG**. Each resolves its compressor by name from `compressor_registry` and runs it over the pure-data `CompressInput`/`CompressOutput` contract via a shared `_registry_compress_content` helper, then maps back to the branch's exact historical return shape. ## Byte-identical by construction - The built-in adapter delegates to the SAME `_get_<name>()` getter + method with the same args (`context`→query, `bias`→budget), so returned content is identical to the old direct call. - Each flipped branch **keeps its `enable_*` gate and `_get_*` availability guard** — so the built-in-unavailable → passthrough behavior is preserved and the adapter's `None`→content collapse is never reached. - Each branch **recomputes its token count with its own historical metric** (`_estimate_tokens` for search/log/tabular; `len(split())` for config). - `content_type` in `CompressInput` is inert (built-ins don't consume it), so it can't shift output. ## Deferred (left byte-for-byte as-is) — and why - **CODE_AWARE** — has a Kompress/ML fallback chain (`compressed is None` → `_try_ml_compressor`, plus a `lossless_then_lossy` no-shrink retry) that mutates `strategy`/`strategy_chain`. Not a clean single call. - **HTML** — uses `.extract().extracted` (different shape) and relies on `None` extraction falling through to bottom passthrough (`[html, passthrough]`); the adapter's `None`→content collapse would change the chain. Not byte-identical through the entry. - **SMART_CRUSHER** (fallback chain), **KOMPRESS/TEXT** (ML boundary), **PASSTHROUGH**, **DIFF** — untouched per plan. The reversibility gate, external-compressor dispatch (headroomlabs-ai#2388), and default (nothing-selected) behavior are unchanged. No new config/env. ## Testing New `tests/test_router_registry_dispatch.py` (6 tests): differential test per flipped strategy asserting registry-dispatch output == old direct-dispatch output (content + branch token metric + `[strategy]` chain), plus assertions that deferred SMART_CRUSHER and KOMPRESS are unchanged. Offline suite: 78 passed; ruff + mypy clean. The broad content-router suite (HF-Hub/ONNX) is deferred to CI — **that full suite is the authoritative byte-identical gate for the flipped strategies.**
…ai#2402) ## Description `/readyz` can keep reporting Kompress as `{"ready": false, "status": "unhealthy", "backend": null}` after the live compressor has already become ready. Startup intentionally records Kompress as `deferred` without loading the model, `WarmupRegistry.merge_transform_status()` stores that only as metadata, and the health check later serializes the stale warmup slot instead of the live runtime compressor state. The request path can already see the real readiness signal through `KompressCompressor.is_ready()`, but nothing promotes the health surface after startup. This change keeps startup behavior untouched and reconciles Kompress health from the live compressor right before `/readyz` serializes component state. It adds side-effect-free runtime backend accessors for local and remote Kompress implementations, promotes the warmup slot only when the runtime compressor is ready, preserves loaded state on transient inspection failures, and keeps Kompress excluded from aggregate readiness. Closes headroomlabs-ai#2386 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/kompress_compressor.py`: add a side-effect-free `ready_backend()` accessor that returns the cached backend for the current model or `None`. - `headroom/transforms/kompress_remote.py`: add `ready_backend()` returning `"remote"` for the always-ready remote adapter. - `headroom/proxy/server.py`: derive Kompress health from the live enabled `ContentRouter` instances, promote the warmup slot only when runtime readiness is real, respect per-provider re-enable overrides, and preserve loaded state on transient inspection failures. - `tests/test_proxy_health.py`: add focused regression, override, pending, remote, no-instantiation, disabled, fail-open, and aggregate-readiness coverage. - `tests/test_kompress_preload_deferral.py`: keep startup-deferral proof current if a helper needs the new accessor surface. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy_health.py tests/test_kompress_preload_deferral.py tests/test_kompress_request_nonblocking.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/server.py headroom/transforms/kompress_compressor.py headroom/transforms/kompress_remote.py tests/test_proxy_health.py tests/test_kompress_preload_deferral.py`) - [x] Formatting passes (`uv run ruff format headroom/proxy/server.py headroom/transforms/kompress_compressor.py headroom/transforms/kompress_remote.py tests/test_proxy_health.py tests/test_kompress_preload_deferral.py --check`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_proxy_health.py tests/test_kompress_preload_deferral.py tests/test_kompress_request_nonblocking.py -q ................................ [100%] 32 passed, 1 warning in 2.06s $ uv run ruff check headroom/proxy/server.py headroom/transforms/kompress_compressor.py headroom/transforms/kompress_remote.py tests/test_proxy_health.py tests/test_kompress_preload_deferral.py All checks passed! $ uv run ruff format headroom/proxy/server.py headroom/transforms/kompress_compressor.py headroom/transforms/kompress_remote.py tests/test_proxy_health.py tests/test_kompress_preload_deferral.py --check 4 files already formatted ``` ## Real Behavior Proof - Environment: Windows host, local FastAPI test app with the same `HeadroomProxy`, `WarmupRegistry`, and `/readyz` route used in production - Exact command / steps: run `uv run pytest tests/test_proxy_health.py tests/test_kompress_preload_deferral.py tests/test_kompress_request_nonblocking.py -q`, covering a deferred startup slot, a pending resident compressor, a global-disable plus `disable_kompress_anthropic=False` override, and a router whose lazy getters would raise if health instantiated them - Observed result: deferred runtime readiness promotes to `{"enabled": true, "ready": true, "status": "healthy", "backend": "onnx"}`, a pending resident compressor stays `{"ready": false, "backend": null}`, a per-provider override re-enables health even when the global flag is off, and the health path never instantiates Kompress - Not tested: live remote Kompress endpoint behavior beyond the local remote-adapter contract ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have made corresponding changes to the documentation if needed - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes ## Additional Notes - `CHANGELOG.md` stays untouched because Headroom generates changelog entries from conventional commits. - Kompress remains a soft component already excluded from aggregate readiness. This PR fixes only the per-component health report. - The health path must remain read-only; it must not call `preload()`, `ensure_background_load()`, `compress()`, or any network or model I/O.
…adroomlabs-ai#2401) ## Description OpenAI-format `POST /v1/chat/completions` requests routed through `--backend litellm-vertex` fail when the client includes `max_tokens`. The proxy currently runs its direct-OpenAI compatibility shim before backend dispatch, renames `max_tokens` to `max_completion_tokens`, then the LiteLLM path no longer recognizes that field as standard and sweeps it into `extra_body`. Vertex rejects the resulting request with `extra_body: Extra inputs are not permitted`. This change scopes the rename shim to the direct OpenAI path only. Backend-routed chat requests now keep `max_tokens`, which LiteLLM already forwards correctly for the Vertex Anthropic path. Direct GPT-5 and o-series compatibility stays unchanged. Closes headroomlabs-ai#2392. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Thread a backend-owned translation flag into `_normalize_openai_max_tokens`. - Skip the legacy-to-completion-token rename on backend-routed OpenAI chat requests. - Keep the direct OpenAI compatibility path covered with a backend-owned translation no-op test. - Add buffered and streaming handler-level regressions for the exact `litellm-vertex` request shape, proving the request survives the `/v1/chat/completions` normalization boundary with vendor fields intact. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py -q ......sss............ [100%] 20 passed, 3 skipped, 1 warning in 42.13s $ uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py All checks passed! $ uv run ruff format headroom/proxy/handlers/openai.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py --check 5 files already formatted ``` ## Real Behavior Proof - Environment: Windows, synced Headroom development environment, mocked LiteLLM provider boundary, no paid GCP credentials required - Exact command / steps: run `uv run pytest tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py -q`, using the issue payload shape `{"model":"claude-sonnet-4-6","max_tokens":32,"messages":[{"role":"user","content":"hi"}],"chat_template_kwargs":{"enable_thinking":false}}` through `POST /v1/chat/completions` - Observed result: buffered and streaming `litellm-vertex` requests keep `max_tokens` as a named backend kwarg, preserve `chat_template_kwargs` in `extra_body`, omit `max_completion_tokens` from `extra_body`, and return success through the handler boundary. Direct-path normalization still renames legacy `max_tokens`. - Not tested: live Vertex AI request ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - `CHANGELOG.md`: N/A, the release pipeline generates it from the conventional-commit subject. - Scope is intentionally narrow: this fixes the exact backend-routed `max_tokens` failure and does not broaden `extra_body` hardening for unrelated OpenAI fields.
…message (headroomlabs-ai#2389) ## Description `CCRResponseHandler._extract_assistant_message` extracts the assistant message from an upstream response while building the CCR retrieval-continuation history. The OpenAI branch is not defensive about an empty or malformed `choices` array: ```python elif provider == "openai": message = response.get("choices", [{}])[0].get("message", {}) ``` `response.get("choices", [{}])` only falls back to `[{}]` when the key is **absent**. When `choices` is present but empty (`[]`) or carries a null first element (`[null]`), this raises on the success path: - `choices: []` → `[][0]` → `IndexError` - `choices: [null]` → `None.get(...)` → `AttributeError` OpenAI-compatible gateways can return those shapes on content-filtered or usage-only responses. The sibling **Google** branch a few lines below already guards this (`candidates = response.get("candidates", []); if candidates: ... else: parts = []`), and so does `ccr/tool_calls.py` (it checks `isinstance(choices, list)`, non-empty, and `isinstance(first_choice, dict)`). Only this OpenAI branch was missed. ## Fix Guard the list and the first element the same way the siblings do: ```python elif provider == "openai": choices = response.get("choices") first = choices[0] if isinstance(choices, list) and choices else {} message = first.get("message", {}) if isinstance(first, dict) else {} return { "role": "assistant", "content": message.get("content"), "tool_calls": message.get("tool_calls"), } ``` A well-formed response is unaffected; an empty/null/absent `choices` now yields `{"role": "assistant", "content": None, "tool_calls": None}` instead of raising. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/ccr/response_handler.py`: guard empty/non-list `choices` and a non-dict first element in the OpenAI branch of `_extract_assistant_message`. - `tests/test_ccr_response_handler.py`: add `TestExtractAssistantMessageEdgeCases` (empty `choices`, `[null]`, absent, and the normal case). - `CHANGELOG.md`: Bug Fixes entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/ccr/response_handler.py tests/test_ccr_response_handler.py All checks passed! $ uvx ruff@0.15.17 format --check headroom/ccr/response_handler.py tests/test_ccr_response_handler.py 2 files already formatted # Verified against the REAL imported module (headroom.ccr.response_handler is # light — no ML imports), so this ran locally in the project venv: $ python -c "from headroom.ccr.response_handler import CCRResponseHandler as H; h=H(); \ assert h._extract_assistant_message({'choices': []}, 'openai') == {'role':'assistant','content':None,'tool_calls':None}" # (no IndexError; normal case still extracts content/tool_calls) ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, project venv (`uv sync --extra proxy`), `uvx ruff@0.15.17`. - Exact command / steps: imported the real `CCRResponseHandler` and called `_extract_assistant_message` with `{"choices": []}`, `{"choices": [null]}`, `{}` (absent), and a normal `{"choices": [{"message": {...}}]}`. - Observed result: the OLD code raised `IndexError` on `[]` and `AttributeError` on `[null]`; the NEW code returns `{"role": "assistant", "content": None, "tool_calls": None}` for all three malformed shapes and still extracts `content`/`tool_calls` from a well-formed response. Because `response_handler` has no ML imports, this ran against the actual module, not a replica. - Not tested: a live CCR retrieval round trip through a gateway that emits empty choices; the added unit tests drive `_extract_assistant_message` directly. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes `headroom/ccr/response_handler.py` is a light module (no ML imports), so unlike most of my recent PRs I verified the fix by importing the real class in the project venv (output above), in addition to the added unit tests. This aligns the OpenAI branch with the already-defensive Google branch and `ccr/tool_calls.py`.
…ff via registry (headroomlabs-ai#2400) ## What Third increment of the adapter phase (builds on headroomlabs-ai#2391/headroomlabs-ai#2399). Adds a `compressed: bool` field to `CompressOutput` and uses it to flip the **fallback/passthrough** strategies — CODE_AWARE and HTML (and DIFF where clean) — to registry-resolved dispatch, byte-identically. ## The contract addition (the enabling piece) `CompressOutput.compressed: bool = True` — lets a compressor signal **passthrough** (did-not-compress, `content` is the original unchanged) vs a real result. This is what the router's `None`-driven fallback/passthrough branches needed to move to the registry without changing behavior. Default `True`, so existing and external compressors are unaffected. ## How (byte-identical) A new `_registry_compress` helper returns the `CompressOutput` (or `None` when the built-in is unavailable, preserving the `_get_*` guard's passthrough). The flipped branches map that back to their historical `compressed is None` semantics: - **CODE_AWARE:** a passthrough (`not output.compressed` / `None`) sets local `compressed = None`, so the existing `_try_ml_compressor` Kompress fallback + `lossless_then_lossy` no-shrink retry + `strategy`/`strategy_chain` mutations run **verbatim**. - **HTML:** a `None`/passthrough falls through to the bottom passthrough exactly as before (`strategy_chain == [html, passthrough]`). ## Deferred SMART_CRUSHER, KOMPRESS, TEXT, PASSTHROUGH — the SmartCrusher→Kompress→Log fallback chain + the ML boundary — are the next (final) increment, left byte-for-byte here. Reversibility gate, external dispatch (headroomlabs-ai#2388), default behavior unchanged. No new config/env. ## Testing `tests/test_router_registry_dispatch.py` + `tests/test_builtin_compressor_adapters.py` extended: differential tests for CODE_AWARE (success AND None→Kompress-fallback with matching `strategy_chain`, ML mocked), HTML (success AND None→`[html, passthrough]`), and the adapter `compressed=False`-on-None mapping. Offline suite: 88 passed; ruff + mypy clean. The full content-router suite in CI is the authoritative byte-identical gate.
…(defer kompress/text ML boundary) (headroomlabs-ai#2404) ## What Final increment of the adapter phase (builds on headroomlabs-ai#2391/headroomlabs-ai#2399/headroomlabs-ai#2400). Flips the **SMART_CRUSHER** primary `.crush()` invocation in `_apply_strategy_to_content` to registry-resolved dispatch, following the CODE_AWARE/HTML pattern. The shared SmartCrusher→Kompress→Log fallback block is unchanged. ## Byte-identical (SMART_CRUSHER) The `smart_crusher` adapter delegates to the same `_get_smart_crusher().crush(content, query=context, bias=bias)` (same cached getter, same method), so `output.content == result.compressed`; the branch recomputes the same `_estimate_tokens` metric; the `if crusher:` guard and the entire fallback chain / `strategy_chain` / `decision_reason` mutations are preserved verbatim. ## Deferred — KOMPRESS and TEXT (honest contract limitation) The `kompress` adapter can't reproduce the direct `_try_ml_compressor(content, context, question)` byte-for-byte, for two independent reasons: 1. **`question` is dropped** — the adapter hardcodes `None`, so QA-aware compression content would diverge. 2. **Token count differs** — the historical branch returns Kompress's own `compressed_tokens` (a word count taken *before* the CCR marker is appended), while the registry path recomputes `_estimate_tokens` over the marker-augmented output. Structurally different numbers whenever Kompress actually compresses. Flipping them would require evolving the adapter/`CompressOutput` contract (forward `question`; carry the compressor's own token count), which is a separate change and would touch the ML boundary — so they're left byte-for-byte here. ## Testing New `tests/test_router_registry_smartcrusher.py`: SMART_CRUSHER success (differential vs a real crush), query/bias forwarding, Kompress-fallback (`[smart_crusher, kompress]`) and Log-fallback (`[smart_crusher, kompress, log]`) chains; plus KOMPRESS/TEXT tests that *pin the deferral facts* (token mismatch + `question` forwarding). Offline suite: 94 passed; ruff (0.15.17) + mypy clean. Full content-router CI suite is the authoritative byte-identical gate. No new config/env. Reversibility gate, external dispatch (headroomlabs-ai#2388), default behavior unchanged.
…roomlabs-ai#2407) ## Description The autouse `_reset_copilot_routing_flag` fixture in `tests/conftest.py` did an unconditional `from headroom.copilot_auth import reset_request_routed_to_copilot` for **every** test. That import pulls in the whole package (`headroom/__init__` → `compress.py` → `observability` → `opentelemetry`). The `macos-native-wrapper` and `windows-native-wrapper` CI jobs run `tests/test_install/test_native_installers.py` with **only `pytest` installed** (see `.github/workflows/ci.yml` — those jobs `pip install pytest` and nothing else). Those tests drive the installer shell scripts via `subprocess` and never import headroom, so the autouse fixture errored at setup: ``` tests/conftest.py:40: in _reset_copilot_routing_flag from headroom.copilot_auth import reset_request_routed_to_copilot headroom/__init__.py:86: from .compress import ... headroom/compress.py:65: from .observability import get_otel_metrics headroom/observability/metrics.py:11: from opentelemetry import metrics E ModuleNotFoundError: No module named 'opentelemetry' ``` Guard the import: when headroom isn't importable there is no routing flag to reset, so the fixture is a no-op. No production code changes; behavior is unchanged whenever headroom is installed (all other jobs). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `tests/conftest.py`: wrap the `_reset_copilot_routing_flag` fixture's `headroom.copilot_auth` import in `try/except ModuleNotFoundError` → yield-and-return when headroom is absent. ## Testing - [x] Ran the exact CI command locally - [x] Linting passes (`ruff check`) ### Test Output ```text $ ruff check tests/conftest.py All checks passed! $ pytest tests/test_install/test_native_installers.py -q collected 2 items tests/test_install/test_native_installers.py ss [100%] ============================== 2 skipped in 0.11s ============================== ``` (2 skipped = Docker not available on the local box; the point is **no more "ERROR at setup"**. Before this change the same run reported `1 error in 0.11s` with the `opentelemetry` traceback above.) ## Real Behavior Proof - Environment: macOS, Python 3.12, headroom installed (normal path exercised). - Exact command / steps: `pytest tests/test_install/test_native_installers.py -q` - Observed result: no setup error; fixture takes the normal (headroom-present) path — 2 tests skipped for lack of Docker. - Not tested: the headroom-absent branch can't be reproduced locally (headroom is installed here); it is exactly the CI job's environment, which this PR's CI run will exercise. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes Scope is intentionally the native-wrapper failures only. The separate `test-dashboard-ui` red X is unrelated (a stale UI-text assertion: `element(s) not found — "Completed 128 Failed 0 Rate Limited 0 Cached 96"`) and is not addressed here. Checklist items about docs/CHANGELOG/new-tests are N/A — this is a test-harness resilience fix, not a behavior change.
## Description Ruff currently has three independent versions: `uv.lock` resolves `0.14.14`, pre-commit runs `0.9.4`, and CI installs `0.15.17`. Contributors can therefore pass one formatter path and fail another. Make the exact Ruff pin in `pyproject.toml` the source of truth, align the lockfile and pre-commit hook to it, and make CI read that pin through a deterministic consistency verifier instead of carrying another hardcoded version. Closes headroomlabs-ai#2398 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that causes existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Pin the existing `[dev]` Ruff dependency to `0.15.17`, the formatter baseline already used by CI. - Refresh only Ruff in `uv.lock` with `uv 0.11.29`. - Align `ruff-pre-commit` to `v0.15.17`. - Add `scripts/verify-ruff-version.py` and run it from pre-commit and CI. - Make CI install the verified version read from `pyproject.toml` rather than a separate literal. ## Testing - [ ] Unit tests pass (`pytest`) — not run; no runtime source or test behavior changed. - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New deterministic guard proves the configuration fix - [x] Manual testing performed ### Test Output ```text # Before: run the verifier with the patched pyproject pin but base-branch # uv.lock, pre-commit config, and workflow. Ruff version mismatch detected: uv.lock uses Ruff 0.14.14, expected 0.15.17 .pre-commit-config.yaml uses Ruff 0.9.4, expected 0.15.17 ci.yml does not run 'python scripts/verify-ruff-version.py --print-version' ci.yml does not install Ruff from 'steps.ruff-version.outputs.version' $ python3 scripts/verify-ruff-version.py Ruff versions aligned at 0.15.17 $ uvx uv@0.11.29 lock --check Resolved 269 packages $ uvx uv@0.11.29 tree --locked --package ruff ruff v0.15.17 $ uvx ruff@0.15.17 check . All checks passed! $ uvx ruff@0.15.17 format --check . 1322 files already formatted $ uvx mypy@1.20.2 headroom --ignore-missing-imports Success: no issues found in 505 source files $ uvx --with tomli mypy@1.20.2 scripts/verify-ruff-version.py --ignore-missing-imports Success: no issues found in 1 source file $ uvx pre-commit run ruff --all-files Passed $ uvx pre-commit run ruff-format --all-files Passed $ uvx pre-commit run verify-ruff-version --all-files Passed ``` ## Real Behavior Proof - Environment: macOS 26.5, Python 3.14.6 locally; Python 3.10 fallback also exercised; `uv 0.11.29`, Ruff `0.15.17`, mypy `1.20.2`. - Exact command / steps: reproduced the mismatch using the base branch's real `uv.lock`, `.pre-commit-config.yaml`, and `.github/workflows/ci.yml`; then ran the verifier, locked Ruff tree, full Ruff check/format, mypy, and actual pre-commit hooks after the patch. - Observed result: the base state fails with all four drift points listed; the patched state reports one aligned Ruff version (`0.15.17`) and every formatter path passes. - Not tested: runtime proxy behavior and the pytest suite, because the change is limited to development-tool configuration, lock metadata, pre-commit, and CI wiring. ## Dependency / Supply-Chain Justification - Ruff is an existing development-only formatter maintained by Astral; this PR adds no new package. - `0.15.17` is required to fix local/CI reproducibility and has already been the repository's CI formatter baseline since headroomlabs-ai#1295. - Install surface is limited to the `[dev]` extra, lint CI job, and pre-commit environment. Production/runtime dependencies are unchanged. - The `uv.lock` refresh updates only Ruff; no unrelated dependency upgrades are included. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented the non-obvious consistency checks - [x] Documentation changes are N/A; contributor commands are unchanged - [x] My changes generate no new warnings - [x] The guard fails on the real base-state mismatch and passes after the fix - [ ] New and existing unit tests pass locally — not run; no runtime code changed - [x] I did not edit `CHANGELOG.md`; release-please will use the conventional PR title ## Additional Notes No formatter-driven source changes are included. AI assistance was used to inspect configuration, implement the verifier, and run validation.
headroomlabs-ai#112) (headroomlabs-ai#2405) Automated by Headroom + Kimi (Fireworks) in a Modal Sandbox. Request: Make ONLY this one change, nothing else: in .github/workflows/release.yml, in the publish-pypi job's step that uses pypa/gh-action-pypi-publish, add exactly one line 'attestations: false' immediately after the 'skip-existing: true' line in its with: block (same indentation). Do NOT edit any other file, do NOT investigate the codebase, do NOT make any other change. Then finish. Co-authored-by: Headroom Kimi <kimi@headroom.dev>
…+ forward question (headroomlabs-ai#2411) ## What Completes the if/elif → registry migration in the content router: **KOMPRESS and TEXT** now dispatch through the `kompress` built-in adapter (`_registry_compress`), like every other strategy. Also **fixes a latent bug** in `_invoke_kompress` that dropped the QA-aware `question` argument (hardcoded `None`) — `question` now rides `CompressInput.config['question']` and is forwarded into `_try_ml_compressor`, so QA-aware compression content is preserved. ## Intentionally NOT byte-identical (one approved change) The sole behavior change is the KOMPRESS/TEXT **token metric**: reported `compressed_tokens` is now `_estimate_tokens(output.content)` — the router's calibrated estimate, consistent with `original_tokens` and every other registry-dispatched strategy — instead of the Kompress model's own tuple count. **Compressed content is preserved byte-for-byte in all paths.** ## Decision-impact analysis (traced every reader of `compressed_tokens`) No content, routing, keep/drop, fallback, or lossless-then-lossy decision reads this metric for KOMPRESS/TEXT: they're not in `fallback_eligible_strategy` nor `{SEARCH,LOG,HTML}`, and the STAGE-0/general layering calls `_try_ml_compressor` directly (unchanged, already forwards `question`). The only downstream value-reader is `_record_to_toin`'s skip gate (`original_tokens <= compressed_tokens`) — **telemetry/learning only**, never affects returned content or routing, and arguably more correct now (both sides on the same `_estimate_tokens` scale). Consciously accepted. ## Tests Rewrote the PR-C2 deferral-pinning tests → registry-dispatch-matches-direct (content matches the direct `_try_ml_compressor(..., question)` call; token assertion switched `==<model count>` → `==_estimate_tokens(output)`, the only assertion change, solely due to the approved metric switch). Added a QA-differential test (question changes content) + an adapter-level `question`-forwarding test. Offline suite: 96 passed; ruff 0.15.17 + mypy clean. **Note:** the full content-router CI suite may require further test updates for any test that exercises the real KOMPRESS/TEXT branch and asserts the returned count equals the model's tuple `compressed_tokens` — those should switch to `_estimate_tokens(output)`. (The broad content_router/compression selection wasn't run locally — it needs ONNX/HF.) After this, the router's per-strategy dispatch is fully registry-resolved.
…ind unified --code-memory (headroomlabs-ai#2413) ## What Two commits: 1. **Unify code-memory MCP selection behind `--code-memory {tokensave|serena|none}`** (+ `HEADROOM_CODE_MEMORY`), collapsing the `--serena`/`--no-serena`/`--no-tokensave` flag tangle into one selector. Old flags remain as hidden deprecated aliases that map into it. Shared across the code-memory-capable subcommands (claude/codex/grok). 2. **Default the engine to Serena**, with its **dashboard browser suppressed**. ## Why Serena as default Serena is a mature, offline, symbol-level code-navigation MCP with broad language coverage (LSP-backed) — the strongest zero-account default for reducing tokens by letting the agent query symbols/definitions/references instead of reading whole files. It attacks the *protected-reads* volume the proxy deliberately doesn't compress, so it's complementary to the pipeline compressors. ## Browser suppression (in Serena's own settings) `_ensure_serena_dashboard_disabled()` sets `web_dashboard_open_on_launch: false` in `~/.serena/serena_config.yml` when Serena is set up, so wrapped sessions don't spawn a browser tab. The dashboard backend stays reachable manually at `localhost:24282`. This lives in Serena's config (authoritative), not just a startup flag. ## Schema-overhead note Serena injects tool schemas per request; that cost is deferred by the tool-search deferral the coding profile already enables (`HEADROOM_TOOL_SEARCH=1`), so tools load on demand — the navigation benefit without a standing schema tax on turns that don't navigate. ## Selection / escape hatches `--code-memory serena` (default) · `tokensave` (lighter/faster) · `none` (disable). Deprecated `--serena`/`--no-serena`/`--no-tokensave` still work. ## Testing Updated the primary/backup policy test to the serena-primary default; code-memory selector + serena disable/migrate tests pass. Local: 21 passed (policy + code-memory); ruff + mypy clean. Full suite in CI.
## Description
ContentRouter ran the native content detector two to three times on
identical content, on the hottest path in the proxy (every compressed
message, every request). This cuts it to once.
`_detect_content` isn't cheap and isn't memoized. It strips a detection
envelope, runs the Rust/Magika ONNX classifier, then several regex
passes. `compress()` ran it once for debug logging that's off by
default, then `_determine_strategy()` recomputed it (plus
`is_mixed_content`) on the same content. That's twice per `compress()`,
and three times on the `apply()` cache-miss path.
Closes: N/A (no filed issue, surfaced by an internal
contribution-backlog audit).
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `compress()` computes `is_mixed_content` and `_detect_content` once,
then threads both into `_determine_strategy` through new optional params
(`mixed`, `detection`).
- `_determine_strategy` uses the passed values when present, and
computes them itself when they're `None`. Its one private caller
changes. Any other caller keeps working.
- Added `tests/test_content_router_detection_dedup.py`. One test asserts
`compress()` detects exactly once (it fails before the fix at `assert 2
== 1`). The other asserts the threaded result routes the same as the
recomputed one across content types.
- Updated two existing `_determine_strategy` test doubles to take the
new kwargs.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_content_router_detection_dedup.py tests/test_transforms_content_router.py \
tests/test_transforms/test_content_router.py tests/test_transforms_content_detection.py -q
135 passed in 8.98s
$ pytest tests/test_transforms/ tests/test_content_router_*.py tests/test_router_*.py \
tests/test_lossless_excluded_compaction.py -q
423 passed, 62 skipped in 54.93s
$ ruff check .
All checks passed!
$ mypy headroom
Success: no issues found in 505 source files
```
## Real Behavior Proof
- Environment: macOS (Darwin), Python 3.13, headroom worktree on this
branch off `upstream/main`, `HF_HUB_OFFLINE=1
LITELLM_LOCAL_MODEL_COST_MAP=true`. A counter wraps the real
`_detect_content` and delegates to it, so real routing and compression
run.
- Exact command / steps: run the real router over one representative
message and count `_detect_content` calls on the fixed tree, then `git
stash` the source and count again on the unfixed tree. Covered
`router.compress(blob)` and `router.apply([tool_msg])`.
- Observed result: `compress()` dropped from 2 detection calls to 1, and
`apply()` dropped from 3 to 2, on the same input with the same routing
strategy (`text`) and the same output. The once-only test flips from
`assert 2 == 1` before to passing after.
- Not tested: production Magika ONNX timing. This dev env has no
onnxruntime, so the detector ran its regex fallback tier, which makes
the saved cost a floor, not a ceiling. I also scoped out the Tier B
extension (threading the `apply()` Pass-1 detection into `compress()`)
on purpose.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
Scope is the default routing path. `force_kompress` already uses the
cheaper regex detector, so it never paid the redundant native cost.
`_compress_mixed` re-detects per split section, but that's different
content (sub-sections), so it's out of scope.
The `apply()` Pass-1 detection stays. It gates the `is_code` protection
check for every message, including cache hits that never reach
`compress()`. Threading it into `compress()` would widen a shared task
tuple and change the public `compress()` signature, all for a
cache-miss-only save, so I left it as a possible follow-up.
Doc checklist item is N/A (internal perf dedup, no user-facing docs
change). This is a Python-only change, so the first push will use
`--no-verify` for the known `ci-precheck` Rust-latency bench flake
(`classify_under_10us_per_call`), which runs clean in CI.
…ion savings on the dashboard (headroomlabs-ai#2248) (headroomlabs-ai#2424) ## Description Users upgrading 0.27.0 → 0.31.0 report that the dashboard's compression / "Tokens Saved" figures drop to ~0 and conclude Headroom stopped working. The headroomlabs-ai#2248 reporter ran the same prompt on both versions and captured the telltale detail: **0.31.0 actually spent fewer total tokens than 0.27.0, despite showing 0 saved.** This is a default-mode change, not a regression. 0.31.0 ships the `coding` savings profile as the out-of-box default (`headroom/agent_savings.py`: `DEFAULT_PROFILE = "coding"`), and `coding` sets `proxy_mode="cache"`. Cache mode freezes the provider prefix and compresses only the newest turn *delta* — deliberately, to avoid busting the prompt cache — so the **compression** number is small while savings shift to **cheaper prefix-cache reads**. On a short prompt there's little delta to compress, so the compression tile reads ~0 even as real cost drops. The reference behavior is already documented in the proxy docs' [Savings profiles](/docs/proxy#savings-profiles) section, but there was no discoverable troubleshooting entry connecting the alarming "0 saved after upgrade" symptom to this cause — so it gets filed as a bug. Closes headroomlabs-ai#2248 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made `docs/content/docs/troubleshooting.mdx` only — a new `### Dashboard shows 0 compressed/saved tokens after upgrading to 0.31.0` subsection appended to the existing `## No Token Savings` section: - **Symptom** — compression figures ~0 after upgrade, while total spend is flat or lower (so users can match it by search). - **Cause** — the `coding`/cache-mode default and why delta-only compression makes the compression tile small. - **Where the savings show up** — the **Prefix Cache Impact** panel and **Compression vs Cache** tile, which reflect cache-read savings; the headline "Tokens Saved" tile counts compression only and understates the benefit in cache mode. - **How to get 0.27.0-style numbers back** — `--mode token`, or `HEADROOM_SAVINGS_PROFILE=balanced` / `agent-90`, with the explicit trade-off that token mode raises visible compression but can reduce prefix-cache hits. Placed under the existing `## No Token Savings` heading (which covers the separate SDK/library case: audit mode, sub-threshold tool outputs) rather than rewriting it. Cross-links to the existing Savings-profiles reference instead of restating the profile table, keeping one source of truth. No code change. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed ### Test Output Docs-only; verification is fact cross-check against source plus MDX sanity: ```text $ grep -n 'DEFAULT_PROFILE = \|proxy_mode="cache"' headroom/agent_savings.py 18:DEFAULT_PROFILE = "coding" 173: proxy_mode="cache", # delta-only compression at ~0 prefix-cache busts $ grep -n "_estimate_cache_savings_usd" headroom/proxy/savings_tracker.py 248:def _estimate_cache_savings_usd(model: str, cache_read_tokens: int) -> float: $ grep -c "Prefix Cache Impact" headroom/dashboard/templates/dashboard.html # 2 $ grep -c "Compression vs Cache" headroom/dashboard/templates/dashboard.html # 1 $ grep -n "### Savings profiles" docs/content/docs/proxy.mdx 94:### Savings profiles # cross-link target for /docs/proxy#savings-profiles # placement: new "### Dashboard shows 0 compressed..." (line 145) sits between # "## No Token Savings" (89) and "## Claude Code context window..." (166) # MDX sanity: code fences balance (even count) ``` ## Real Behavior Proof - **Environment:** Docs source verified against the current `main` base (`718c8dc5`). - **Exact command / steps:** Issue headroomlabs-ai#2248 contains a complete reproduction — the same prompt run under 0.27.0 and 0.31.0 via `headroom wrap claude --dangerously-skip-permissions` (Sonnet 5, same files, same Claude Code version, reproduced on macOS and Debian 12), with dashboard screenshots showing savings on 0.27.0 and ~0 on 0.31.0. Every claim in the new section is verified against the tree with the greps above: the `coding` default and its `proxy_mode="cache"`, the cache-read savings estimator, and both dashboard panel/tile labels users are pointed to. - **Observed result:** The documented cause matches the code — the compression tile legitimately reads ~0 in cache mode while cache-read savings accrue in the Prefix Cache Impact panel, which explains the reporter's own observation that 0.31.0 spent *fewer* tokens while showing 0 saved. - **Not tested:** I did not re-run a live 0.27.0-vs-0.31.0 dashboard comparison (that requires installing an old release and generating real provider traffic); the reporter's reproduction with screenshots already establishes the symptom, and the cause is verified in source. No local Fumadocs site build was run, so the section is validated by MDX syntax checks rather than a rendered preview. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A (troubleshooting prose addition). ## Additional Notes - Test/tests-added and CHANGELOG checklist items are N/A — documentation-only change, kept to a single file (matching the merged headroomlabs-ai#2031 and headroomlabs-ai#2237 precedent). - If maintainers would rather resolve this in the UI than the docs, an alternative is a dashboard hint shown when mode is `cache` and compression savings are ~0 (pointing at the Prefix Cache Impact panel). That touches `dashboard.html` and has UX implications, so it's intentionally not attempted here. - This is the second report rooted in the cache-mode default (following the confusion behind headroomlabs-ai#2031), which is why it's framed as a searchable troubleshooting entry rather than another reference-section edit.
…ive passthrough (headroomlabs-ai#2422) ## Description Documents the Vertex AI proxy backend properly, fixing headroomlabs-ai#2393. Following the docs verbatim (`pip install "headroom-ai[proxy]"` + `headroom proxy --backend vertex_ai`) currently fails with `vertexai import failed`, and the LiteLLM-specific `VERTEXAI_PROJECT`/`VERTEXAI_LOCATION` env vars are documented nowhere — risking requests silently resolving against the ADC default quota project and billing the wrong GCP project. All documented behavior was verified against source: alias normalization in `headroom/providers/registry.py` (`vertex`/`google-vertex`/`googlevertex` → `vertex_ai`), the always-registered native publisher passthrough routes in `headroom/providers/proxy_routes.py`, and `pyproject.toml` (no extra pulls in `google-cloud-aiplatform`). ## Type of Change - [ ] Bug fix - [ ] New feature - [x] Documentation update - [ ] Refactor - [ ] Other ## Changes Made - `docs/content/docs/proxy.mdx`: new **Google Vertex AI** subsection under Cloud providers — `google-cloud-aiplatform>=1.38` requirement (not in any extra or Docker image), `VERTEXAI_PROJECT`/`VERTEXAI_LOCATION` env vars with a warning about silent ADC quota-project fallback and their distinction from the standard `GOOGLE_CLOUD_PROJECT`/`GOOGLE_CLOUD_LOCATION` vars, backend name alias equivalence (`vertex_ai` / `vertex` / `google-vertex` / `googlevertex` / `litellm-vertex` / `litellm-vertex_ai`), and cross-links to the Claude Code on Vertex page and the LiteLLM callback page. - `docs/content/docs/proxy.mdx`: new **Native Vertex passthrough routes** subsection documenting the unconditionally registered `/{api_version}/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:*` routes and the `publisher=google` (Gemini handler) vs `publisher=anthropic` (LiteLLM-Vertex path) branching. - `docs/content/docs/installation.mdx`: added `VERTEXAI_PROJECT` and `VERTEXAI_LOCATION` rows to the LLM provider keys table, plus a pointer to the new Vertex section for the SDK dependency. - `docs/content/docs/litellm.mdx`: cross-reference callout distinguishing the LiteLLM callback integration from the proxy's `litellm-*` backends (issue gap #5). ## Testing - [x] Docs build passes locally **Test Output** ``` $ npm run build # docs/ — same as CI validate-nextjs ✓ Static + SSG pages generated (exit code 0), /docs/proxy, /docs/installation, /docs/litellm prerendered $ mkdocs build # same as CI validate-mkdocs INFO - Documentation built in 8.32 seconds ``` ## Real Behavior Proof - Environment: Windows 11, Node 20, npm 10, Python 3.13, mkdocs-material (latest), branch `docs/2393-vertex-ai-backend` off `upstream/main`. - Exact command / steps: `cd docs && npm ci && npm run build`; `mkdocs build` from repo root; manually re-verified each documented claim against `headroom/providers/registry.py` (alias normalization), `headroom/providers/proxy_routes.py` (publisher passthrough routes), and `pyproject.toml` `[project.optional-dependencies]` (no vertex SDK in any extra). - Observed result: Both docs builds succeed; new sections render with valid internal anchors (`/docs/proxy#google-vertex-ai`, `/docs/proxy#cloud-providers`, `/docs/claude-code-vertex`, `/docs/litellm`). - Not tested: Live end-to-end Vertex AI request through the proxy (no GCP project available); error messages and env-var behavior are taken from the issue reporter's verified reproduction on v0.32.0 and cross-checked against LiteLLM's Vertex provider docs. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Description The OpenAI chat path caches responses under a different key than it looked them up by. `handle_openai_chat` calls `cache.get(messages, ...)` at request start, then the `pre_compress` hook reassigns `messages` before `cache.set(messages, ...)`. When a deployment configures a message-rewriting hook, the handler stores every response under a key no future lookup can produce. The response cache never hits and fills with unreachable entries until eviction, with no error signal. This is the OpenAI twin of the anthropic fix in headroomlabs-ai#2124 (which closed headroomlabs-ai#327). Same snapshot pattern: capture the lookup messages once before the hook runs, reuse them verbatim at `cache.set`. Related to headroomlabs-ai#327, follow-on to headroomlabs-ai#2124 (which fixed the anthropic side only). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Snapshot `cache_lookup_messages = messages` before the `pre_compress` hook in `handle_openai_chat`, and cache the response under that snapshot at `cache.set`. Mirrors the shipped anthropic pattern in `handlers/anthropic.py`. - Add `tests/test_openai_response_cache_key.py`: drives two identical `/v1/chat/completions` requests through a message-rewriting `pre_compress` hook against the real `SemanticCache`, and asserts the repeat is served from cache (upstream called once) rather than re-sent. This exercises the real cache-key function, which a get/set-argument check does not. - Document the ordering invariant at the snapshot: image compression also rebinds `messages` but runs upstream of the snapshot, so a future reorder that moved it below would reintroduce the drift. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_openai_response_cache_key.py tests/test_proxy_openai_cache_key_integration.py tests/test_backend_nonstreaming_cache_metrics.py tests/test_openai_codex_routing.py -q 31 passed, 1 warning in 17.43s $ ruff check headroom/proxy/handlers/openai.py tests/test_openai_response_cache_key.py All checks passed! $ mypy headroom Success: no issues found in 505 source files ``` ## Real Behavior Proof - Environment: headroom at `upstream/main` 6e4425a plus these commits, Python 3.13, macOS, uv venv. Drove the real `handle_openai_chat` through `create_app` + `TestClient` posting `/v1/chat/completions`, cache enabled (the real `SemanticCache`), a message-rewriting `pre_compress` hook, and a stubbed 200 upstream returning a unique body per call. - Exact command / steps: the regression test POSTs two identical requests and counts upstream calls. Ran it in-tree (with `conftest`) on the unpatched handler and again with the fix. - Observed result: on the unpatched handler the repeat request misses the cache and is re-sent upstream (served `resp-2`, upstream called twice). With the fix the repeat is served from cache (`resp-1`, upstream called once). Fails on the unpatched handler, passes with the fix, verified in-tree. A standalone key-hash demo corroborates: pre-fix `stored=[MUTATED]` != `lookup=[hello]` -> DRIFT, post-fix they match -> MATCH. - Not tested: only exercised the drift under a synthetic message-rewriting hook (the OSS default `CompressionHooks` is a no-op, so no user hits this without a custom hook). Did not measure real-world cache-hit-rate recovery on a production workload, and did not touch the streaming path (the response cache is non-streaming only). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes - No `CHANGELOG.md` edit. release-please owns it (`changelog-guard.yml`), and the entry comes from the Conventional Commit title `fix(proxy/openai): ...`. - Scope is latent in OSS: the default `CompressionHooks` is a no-op and no shipped subclass rewrites `messages`, so this only bites deployments that provide a custom message-rewriting `pre_compress` hook. It ships at parity with the anthropic side (headroomlabs-ai#2124). - Image compression on this path does rebind `messages`, but it runs upstream of the cache lookup and the snapshot, so it is not a between-lookup-and-store drift vector. The one live vector is the `pre_compress` hook. Anthropic differs: it runs image compression and a security scan after its lookup, so it snapshots against three vectors. The snapshot comment documents this ordering as a tripwire (a self-correction: an earlier commit message imprecisely said image compression "never rebinds messages"). - Pushed with `--no-verify`: the `ci-precheck-python` pre-push hook false-fails in a uv worktree venv (no `pip`), and the Rust latency benchmark flakes under local load. Python and Rust tests pass in the same run, and CI runs them on clean hardware.
…, repo-language scoping (headroomlabs-ai#2425) When Serena is the active code-memory engine, `headroom wrap` now does three things (all best-effort, timeout-guarded, non-fatal, and fully inert when Serena/uvx are absent — mirroring the existing RTK/tokensave patterns): 1. **Symbol-first guidance** — injects a marker-guarded, idempotent block into the agent's hint file (`CLAUDE.md` for Claude; `AGENTS.md` for Codex/Grok/OpenCode) steering it to prefer Serena's `get_symbols_overview` / `find_symbol` / `find_referencing_symbols` / `find_declaration` over whole-file reads. This is the highest-leverage change — Serena only saves tokens if the agent actually uses it. 2. **Repo-language scoping** — detects the languages present in the repo (extension scan, pruning `.git`/`node_modules`/`.venv`/etc.) and pins them into `.serena/project.yml`'s `languages` list, so Serena doesn't spin up superfluous language servers. Conservative: only rewrites a single-line flow list or creates a minimal `project.yml`; a custom/block-style entry is left untouched to avoid corrupting hand-authored config. 3. **Wrap-time pre-index** — runs `serena project index` so the first symbol query isn't cold. Order is inject → scope → index (scope before index so the pre-index respects the scope). No new env vars, no settings_store drift, no behavior change outside the Serena path. The `languages` key and extension→language mapping were verified from Serena's local source (`project.template.yml`, `ProjectConfig`, `solidlsp/ls_config.py`), not the web. ## Testing New `tests/test_cli/test_wrap_serena_boost.py` (16 tests: injection idempotency + content, language detection incl. ignore-dirs, mocked pre-index/project.yml write incl. failure/timeout no-op). Updated `test_serena_migrate.py`'s fixture to neutralize the new side-effecting calls. Offline: 46 passed; ruff 0.15.17 + mypy clean.
…k time (headroomlabs-ai#2428) ## Description The shared `test` job is currently failing on every open PR because of a wall-clock time-bomb in the DeepSeek pricing tests, not because of any code change. `tests/test_providers/test_deepseek.py::TestDeepSeekPricingModule::test_registry_staleness_and_source_url` asserted: ```python assert not registry.is_stale() ``` `PricingRegistry.is_stale()` returns `(date.today() - last_updated) > timedelta(days=30)`. The DeepSeek registry ships `LAST_UPDATED = date(2026, 6, 19)`, so this assertion holds only while the current date stays within 30 days of that constant. Once it lapses, the test fails on time alone, turning the `test` shard red for every unrelated PR in the repo. It is failing right now (31 days past `LAST_UPDATED`). This is not testing code behavior: it only checks that the machine's clock is within 30 days of a hardcoded date. The sibling Anthropic and OpenAI registries are 560 days old and make no such assertion, so DeepSeek is the odd one out here rather than a deliberate freshness gate. ## Fix Drop the freshness assertion and keep the meaningful `source_url` check, renaming the test to `test_registry_source_url` to match what it now verifies. The staleness mechanism stays fully and time-independently covered by `tests/test_pricing.py::test_registry_staleness_and_warning`, which builds registries with `date.today() - timedelta(days=30)` (asserts not stale) and `date.today() - timedelta(days=31)` (asserts stale) plus the warning text. So this removes a fragile environmental assertion without reducing real coverage, and aligns DeepSeek with the Anthropic/OpenAI registries. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `tests/test_providers/test_deepseek.py`: remove the wall-clock-dependent `assert not registry.is_stale()`, keep the `source_url` assertion, rename the test to `test_registry_source_url`, and add a comment explaining why freshness is not asserted here. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check tests/test_providers/test_deepseek.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, project venv (`uv sync --extra proxy`), `uvx ruff@0.15.17`. - Exact command / steps: with the current date at 31 days past `LAST_UPDATED`, ran the registry's `is_stale()` and the fixed test body against the real modules, plus the mechanism test from `tests/test_pricing.py`. - Observed result: `get_deepseek_registry().is_stale()` is `True` on the current date (which is exactly what broke the old assertion); the fixed `test_registry_source_url` body passes regardless of the date; and `test_registry_staleness_and_warning` still passes, so the staleness mechanism remains covered. - Not tested: a live DeepSeek pricing fetch (out of scope; pricing values are unchanged). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable
…epo (headroomlabs-ai#2427) ## Description Fixes headroomlabs-ai#2426. Persistent Docker deployments store their image in the deployment manifest. The image org moved from the personal `ghcr.io/chopratejas/headroom` repo to the project org `ghcr.io/headroomlabs-ai/headroom`, and the personal repo is frozen at 0.27.0. Because the manifest image is only ever read back verbatim (`build_runtime_command`, `docker run`, status output), a deployment created before the move keeps pulling 0.27.0 forever, several minor versions behind the CLI, with no drift signal to the user. Two related gaps: - `headroom/install/state.py` reads the recorded image straight back with no migration, so an old manifest is stuck on the dead repo. - `headroom/cli/install.py` `deploy --image` still defaulted to `ghcr.io/chopratejas/headroom:latest`, so brand new deploys through that command also pinned the retired repo (the `install-apply` default was already correct). ## Fix - Rewrite the retired repo to the org repo when a manifest is loaded, in both `load_manifest` and `list_manifests`, preserving whatever tag was recorded. The rewrite is surgical: it only matches the exact retired `ghcr.io/chopratejas/headroom` repo and leaves already-current images and any third-party image untouched. The migrated value persists on the next apply/save. - Change the `deploy --image` default to `ghcr.io/headroomlabs-ai/headroom:latest` so it matches `install-apply`. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/install/state.py`: add `_migrate_deprecated_image` and apply it in `load_manifest` and `list_manifests` before constructing the manifest. - `headroom/cli/install.py`: `deploy --image` default now points at the org repo. - `tests/test_install/test_state.py`: new tests covering load and list migrating the retired repo (tag preserved) and leaving current/third-party images untouched. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/install/state.py headroom/cli/install.py tests/test_install/test_state.py All checks passed! $ uvx mypy@1.20.2 --ignore-missing-imports headroom/install/state.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, project venv (`uv sync --extra proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`. - Exact command / steps: wrote a manifest.json pinning `ghcr.io/chopratejas/headroom:latest` (and `:0.27.0`) under a temp home, then called the real `load_manifest` and `list_manifests`. - Observed result: both returned a manifest with `image == ghcr.io/headroomlabs-ai/headroom:latest` (tag preserved on the `0.27.0` case too); an already-current image and a third-party image passed through unchanged. Ran against the actual module. - Not tested: a live `docker run` against the migrated image. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable
…adroomlabs-ai#2431) ## Description `handle_openai_chat` reads token counts from the response usage to record metrics and update the prefix tracker: ```python usage = backend_response.body.get("usage", {}) output_tokens = usage.get("completion_tokens", 0) total_input_tokens = usage.get("prompt_tokens", optimized_tokens) ``` `.get(key, default)` only falls back when the key is **absent**. When an OpenAI-compatible backend emits a key with a **null** value (providers do this on a stopped or empty turn, the same shape that caused the Gemini crash in headroomlabs-ai#2347), `.get` returns `None`. That `None` then flows into: - `_infer_openai_cache_write_tokens(total_input_tokens, cache_read_tokens)` → `max(input_tokens - cache_read_tokens, 0)` (a `None - int` → `TypeError`), - `uncached_input_tokens = max(0, total_input_tokens - cache_read_tokens - cache_write_tokens)`, and - `RequestOutcome(output_tokens=..., optimized_tokens=...)`, whose fields are `int` and which the metrics recorder increments. Both chat usage-extraction sites are affected. On the direct-provider branch the arithmetic runs **outside** the surrounding `try`, so a single such response raises an uncaught `TypeError` and 500s the request; on the backend branch it corrupts outcome recording. ## Fix Coerce the three counts with the existing module-level `_usage_int` guard (`max(int(value), 0)`, 0 on failure) at both sites, matching the streaming path, the already-guarded cache keys in the same block (`usage.get("cache_read_input_tokens", 0) or 0`), and the Gemini fix in headroomlabs-ai#2347. A normal integer usage is unchanged; only a null (or absent) value now becomes the fallback/0. `prompt_tokens` keeps its `optimized_tokens` fallback so our own input estimate is used when the count is missing. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/openai.py`: `_usage_int`-guard `completion_tokens` / `prompt_tokens` / `cached_tokens` at both non-streaming usage-extraction sites in `handle_openai_chat`. - `tests/test_proxy/test_openai_chat_savings_profile.py`: regression driving a `/v1/chat/completions` request whose backend usage reports null `prompt_tokens` / `completion_tokens`, asserting a 200 instead of a crash. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_proxy/test_openai_chat_savings_profile.py -q 2 passed # with the fix reverted, the new test fails (the null-usage response 500s): $ git stash push -- headroom/proxy/handlers/openai.py $ python -m pytest tests/test_proxy/test_openai_chat_savings_profile.py::test_chat_completions_survives_null_usage_token_counts -q 1 failed $ uvx ruff@0.15.17 check headroom/proxy/handlers/openai.py tests/test_proxy/test_openai_chat_savings_profile.py All checks passed! $ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/handlers/openai.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, project venv (`uv sync --extra proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv. - Exact command / steps: ran the new FastAPI `TestClient` regression, which drives the real `handle_openai_chat` through a mock backend returning `usage: {prompt_tokens: null, completion_tokens: null, total_tokens: null}`; then reverted only `openai.py` and re-ran the same test. - Observed result: with the fix the request returns 200; with the fix reverted the same request fails (the null count reaches the `max(...)` arithmetic and outcome recording). Ran against the actual handler via the app. - Not tested: a live third-party OpenAI-compatible gateway emitting null usage. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable
…ployments (headroomlabs-ai#2429) ## Description Fixes headroomlabs-ai#2240. `headroom install apply` builds the persistent deployment's environment from the `HEADROOM_*` family plus any explicit `--env KEY=VALUE`. It never captured the provider upstream-routing overrides that the interactive `headroom proxy` reads from the environment through `resolve_api_overrides` (`ANTHROPIC_TARGET_API_URL` and its `*_TARGET_API_URL` siblings). A supervised runner (launchd, systemd, cron, Windows service/task) starts from a bare environment, so those exports never reach the persistent proxy. The result: a user who exports `ANTHROPIC_TARGET_API_URL` pointing at their gateway and runs `install apply` gets a proxy that silently forwards to the default Anthropic endpoint instead. That is both a correctness bug and a routing surprise (traffic and keys can go to the wrong host). ## Fix Capture the documented `*_TARGET_API_URL` overrides from the current environment and merge them into the manifest env underneath the explicit `--env` map, so an explicit `--env` still wins. Scope notes: - Only URL overrides are auto-captured. The `*_TARGET_API_HEADERS` variables can carry bearer tokens, so those are deliberately left to an explicit `--env` rather than being persisted into the on-disk manifest implicitly. - The proxy already resolves these vars correctly at runtime; this only makes `install apply` hand them to the supervised process the same way the interactive proxy would inherit them. - `headroom deploy` (the Docker path) is left unchanged here; this targets the exact reported `install apply` flow. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cli/install.py`: add `_PASSTHROUGH_URL_ENV_VARS` and `_capture_passthrough_env`, and merge the captured overrides under the parsed `--env` map in `install_apply` before building the manifest. - `tests/test_cli/test_install_cli.py`: unit test for the capture helper (skips empty/unrelated vars), plus CliRunner tests that a set `ANTHROPIC_TARGET_API_URL` reaches `build_manifest`'s env and that an explicit `--env` overrides the captured value. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_cli/test_install_cli.py -k "capture or captures or overrides" -q 3 passed $ uvx ruff@0.15.17 check headroom/cli/install.py tests/test_cli/test_install_cli.py All checks passed! $ uvx mypy@1.20.2 --ignore-missing-imports headroom/cli/install.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, project venv (`uv sync --extra proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv. - Exact command / steps: ran the new CliRunner tests, which export `ANTHROPIC_TARGET_API_URL` via monkeypatch, invoke `install apply` with the supervisor side effects stubbed, and capture the kwargs handed to `build_manifest`. Also called the real `_capture_passthrough_env` and real `build_manifest` directly to confirm the value lands in `manifest.base_env`. - Observed result: with the var exported, `build_manifest` received it in `extra_env` and `manifest.base_env["ANTHROPIC_TARGET_API_URL"]` held the gateway URL; with an explicit `--env ANTHROPIC_TARGET_API_URL=...` the explicit value won; empty and unrelated vars were skipped. Ran against the actual modules. - Not tested: a live launchd/systemd run forwarding to a real gateway. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable
headroomlabs-ai#2410) (headroomlabs-ai#2415) ## Description Fixes headroomlabs-ai#2410. When a streaming `/v1/responses` request has `headroom_retrieve` available, Headroom forces a non-streaming (`stream:false`) upstream call so CCR retrieval can be resolved server-side, then reconstructs the complete response as SSE for the client. GitHub Copilot returns 200 with real output tokens, but OpenCode shows no assistant response. Root cause: `_openai_responses_to_sse` emitted only two events — `response.created` and `response.completed`: ```python created_response = {**response, "status": "in_progress", "output": []} events = [("response.created", created_response), ("response.completed", response)] ``` Clients that read the whole answer off the terminal `response.completed` event work, but OpenCode and the Vercel AI SDK render output from the **incremental** item/text events (`response.output_item.added`, `response.output_text.delta`, ...). With those absent, the SDK displays nothing. ## Fix Reconstruct the real Responses event sequence: ``` response.created (status in_progress, empty output) response.in_progress for each output item: response.output_item.added (message items start with empty content) for each message content part: response.content_part.added (text blanked) response.output_text.delta (the text) response.output_text.done response.content_part.done response.output_item.done (full item) response.completed (full response) data: [DONE] ``` Non-message items (reasoning, function_call, ...) get `output_item.added` + `output_item.done` with the full item. Every event carries a contiguous `sequence_number`. The terminal `response.completed` still carries the full response, so clients that key off it are unaffected; clients that stream now receive the deltas they need. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/openai.py`: rewrite `_openai_responses_to_sse` to replay the incremental output-item/content-part/output-text events between `response.created`/`response.in_progress` and `response.completed`. - `tests/test_openai_responses_buffered_sse.py`: new test asserting the incremental `output_text.delta` (visible text), the per-item sequence for message vs non-message items, the empty-output case, and contiguous sequence numbers. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/proxy/handlers/openai.py tests/test_openai_responses_buffered_sse.py All checks passed! $ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/handlers/openai.py # no errors in the changed file # _openai_responses_to_sse is a pure module-level function, so I ran the new # tests against the real code in the project venv (uv sync): 3 passed. ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, project venv (`uv sync --extra proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`. - Exact command / steps: fed the real `_openai_responses_to_sse` a completed response with a reasoning item and a message item whose content is `output_text: "Hello world"`, plus an empty-output response and a function_call-only response. - Observed result: the stream now contains `response.output_text.delta` with `"Hello world"` at `output_index=1, content_index=0`, wrapped by `content_part.added/done` and `output_item.added/done`, with the reasoning and function_call items emitted as `output_item.added/done` and preserved whole; `response.created`/`in_progress` carry empty output while `response.completed` carries the full output; sequence numbers are `0..n`. Ran against the actual module. - Not tested: a live OpenCode -> Copilot Responses round trip; the added tests assert the event stream directly. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes `_openai_responses_to_sse` is a pure function, so I verified the fix against the real code in the venv (output above) in addition to the unit tests. This mirrors the incremental replay the Anthropic buffered path already does in `StreamingMixin._response_to_sse` (content_block_start/delta/stop), bringing the Responses buffered-CCR path to the same fidelity.
…eadroomlabs-ai#2409) (headroomlabs-ai#2414) ## Description Fixes headroomlabs-ai#2409. GitHub Copilot Claude requests routed through Headroom return `404 page not found`. Copilot serves Claude models at `/v1/messages`, but Headroom forwards them to `/messages`, so the upstream 404s (observed on OpenCode's GitHub Copilot provider for `claude-haiku-4.5` / `claude-sonnet-4.6`, Headroom 0.32.0). ## Root cause `build_copilot_upstream_url` strips the `/v1` prefix from every Copilot path: ```python if normalized_path.startswith("/v1/"): normalized_path = normalized_path[3:] ``` That is correct for Copilot's **OpenAI-compatible** surface, which has no `/v1` (`/chat/completions`, `/responses`, `/embeddings`). But Copilot's **Anthropic** surface for Claude models is `/v1/messages` — with the `/v1`. Stripping it produces `https://api.githubcopilot.com/messages`, which 404s. Confirmed against the current code: ```text build_copilot_upstream_url("https://api.githubcopilot.com", "/v1/messages") -> "https://api.githubcopilot.com/messages" # 404 ``` ## Fix Keep `/v1` for the messages endpoint; still strip it for the OpenAI paths: ```python if normalized_path.startswith("/v1/") and not normalized_path.startswith("/v1/messages"): normalized_path = normalized_path[3:] ``` Now `/v1/messages` (and `/v1/messages/batches`) route to `.../v1/messages`, while `/v1/chat/completions` -> `/chat/completions` and `/v1/responses` -> `/responses` are unchanged, on both the public and GHE Copilot hosts. Non-Copilot upstreams are untouched. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/copilot_auth.py`: exclude `/v1/messages` from the `/v1`-strip in `build_copilot_upstream_url`. - `tests/test_copilot_auth.py`: assert `/v1/messages` (+ batches, + GHE host) keep `/v1` while the OpenAI paths still strip it. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/copilot_auth.py tests/test_copilot_auth.py All checks passed! $ uvx mypy@1.20.2 --ignore-missing-imports headroom/copilot_auth.py Success: no issues found in 1 source file # copilot_auth is import-light, so I ran the real function in the project venv # (uv sync): /v1/messages -> .../v1/messages, /v1/chat/completions -> /chat/completions. ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, project venv (`uv sync --extra proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`. - Exact command / steps: called the real `build_copilot_upstream_url` before and after the change for `/v1/messages`, `/v1/messages/batches`, `/v1/chat/completions`, `/v1/responses`, `/v1/embeddings`, and a non-Copilot host. - Observed result: before, `/v1/messages` -> `.../messages` (the 404); after, `.../v1/messages`. Batches keep `/v1` too; the OpenAI paths still strip `/v1` (`/chat/completions`, `/responses`, `/embeddings`); `https://api.anthropic.com/v1/messages` is unchanged. Ran against the actual module. - Not tested: a live OpenCode -> Copilot Claude round trip; the added unit tests assert the URL construction directly. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes `copilot_auth` is a light module, so I verified the fix against the real function in the venv (output above) in addition to the unit tests. Scope is limited to the messages endpoint (the reported 404); every other Copilot path is byte-identical to before.
…omlabs-ai#2433) ## Description Adds an optional external provider for the information-preserving compaction of protected (excluded) tool output, mirroring the existing `proxy_extension` / `compressor` extension seams. Lets an out-of-tree extension supply its own reversible compaction for excluded tools without forking the router. Default behavior is unchanged. Closes # ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - New `headroom/transforms/lossless_provider.py`: `set_lossless_provider` / `get_lossless_provider`. Contract: `content -> (compacted, kind) | None`, where `compacted` must be byte-recoverable (or data-lossless for structured data), and the provider must be deterministic and per-block so the prefix cache stays byte-stable across turns. - `ContentRouter._lossless_compact_excluded` consults a registered provider first and is **authoritative** when one is set; it falls back to the built-in folds only if the provider raises. With no provider registered (the default) behavior is byte-for-byte identical to before. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text $ python -m pytest tests/test_lossless_excluded_compaction.py -q tests/test_lossless_excluded_compaction.py ........... [100%] 11 passed in 0.60s $ ruff check headroom/transforms/lossless_provider.py headroom/transforms/content_router.py All checks passed! $ mypy headroom/transforms/lossless_provider.py headroom/transforms/content_router.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: local, Python 3.12 venv, `ContentRouter(ContentRouterConfig())`. - Exact command / steps: (1) default — call `_lossless_compact_excluded(GREP)` with no provider; (2) register `set_lossless_provider(lambda c: ("<<folded>>","custom"))` and call again; (3) register a provider that raises. - Observed result: (1) built-in search-heading fold `("…","search")`; (2) returns `("<<folded>>","custom")` — provider is authoritative, built-in not run; provider returning `None` yields `None` (no built-in fallback); (3) provider exception → falls back to the built-in fold. Covered by the 3 new tests. - Not tested: the broad `tests/test_transforms/test_content_router.py` suite stalls locally on model downloads (HF/ONNX); CI runs it. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`
…headroomlabs-ai#2439) ## Description `PrometheusMetrics.record_request` appends one durable JSONL event per compressed request. That append is synchronous: `open` + `fcntl.flock` + `write`, plus a full-file rewrite once the ledger passes 1 MB. It runs on the event loop, inside `self._lock`. `export()` takes that same lock and holds it for the entire Prometheus serialization, so a slow ledger write stops `/metrics` cold. In a repro run of 200 compressed requests, `/metrics` completed zero scrapes and the event loop never yielded once across 6.4 seconds. The append now runs in a thread, outside the lock. `savings_ledger` already takes its own `flock` across processes, so the metrics lock was never what made the write safe. Both halves are one change. Awaiting inside the lock would hold it for the whole write rather than just the syscall, which is worse than what is on main today. The file already documents this hazard against itself. `record_stage_timings` (`prometheus_metrics.py:867-874`) picks a plain `threading.Lock` over `self._lock` specifically because "the async lock is also held by `export()` during Prometheus scrapes." The ledger append was the pattern that docstring warns about. No filed issue for this one. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Move the `savings_ledger.record_savings_event` call in `record_request` out of `async with self._lock` and run it through `asyncio.to_thread`. The call site keeps its keyword arguments verbatim; `to_thread` forwards `**kwargs`, so no `functools.partial` wrapper is needed. - Keep the `await`. Callers still see the event on disk when `record_request` returns, which `tests/test_savings_ledger_before_forwarded.py` asserts synchronously. - Add `tests/test_savings_ledger_offload.py`: lock scope, event-loop responsiveness, durability on return, and both arms of the `tokens_saved > 0 and not stateless` gate. `savings_ledger.py` is untouched. It stays synchronous so the MCP `headroom_compress` caller in `ccr/mcp_server.py:789` does not have to change. Sizing the executor is left alone on purpose. `asyncio.to_thread` uses the default pool, which is the documented tool for blocking I/O and already the idiom here (`helpers.py:1297`, `server.py:1694`, `:3557`, `:3613`, `:4244`). The compression pools are sized `max(1, os.cpu_count())` for CPU-bound work, and `PrometheusMetrics` holds no reference to `HeadroomProxy` anyway, so reaching them would mean a new constructor parameter. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_savings_ledger_offload.py tests/test_savings_ledger.py tests/test_savings_ledger_before_forwarded.py -q ======================== 26 passed, 1 warning in 4.08s ========================= $ ruff check . && ruff format --check headroom/proxy/prometheus_metrics.py tests/test_savings_ledger_offload.py All checks passed! 2 files already formatted $ mypy headroom/proxy/prometheus_metrics.py Success: no issues found in 1 source file ``` Broader sweep across the blast radius, 145 test files matching savings / metrics / outcome / stats / proxy / handler / server / ledger / cost / prometheus, each run under a per-file wall-clock watchdog: ```text 138 files pass, 1470 tests passed 7 non-green: HANG tests/test_agent_savings.py HANG tests/test_ccr_mcp_server.py HANG tests/test_netcost_gate.py HANG tests/test_proxy_compress_endpoint.py HANG tests/test_proxy_mode_benchmark.py HANG tests/test_read_maturation_handler_nobust.py FAIL tests/test_proxy_copilot_auth_hooks.py::test_openai_passthrough_applies_copilot_auth Same 7 files re-run with headroom/proxy/prometheus_metrics.py reverted to c400f90: identical set, identical failure. diff of the two non-green lists is empty. ``` The before and after non-green sets match exactly, so nothing here is a regression from this PR. See Additional Notes for the hang. ## Real Behavior Proof - Environment: macOS 15.4 (Darwin 25.4.0) arm64, Python 3.13.13, uv-managed venv, git worktree at `upstream/main` `c400f908`. - Exact command / steps: a standalone asyncio script, not the unit tests. It builds a real `PrometheusMetrics` (no injected tracker, so it self-constructs with `save_flush_every=PROXY_SAVINGS_FLUSH_EVERY` exactly as the proxy does) against a real on-disk ledger pre-seeded to 3.00 MB so `_maybe_compact`'s full-file rewrite actually fires. It then drives 200 `record_request` calls at concurrency 16 while a `/metrics` scraper calls `export()` every 20 ms and a canary coroutine ticks every 5 ms. Ran twice from the same script: once with `headroom/proxy/prometheus_metrics.py` reverted to `c400f908`, once with this change. Seeding the ledger past 1 MB is the part that matters. On a fresh ledger the write is microseconds, compaction never fires, and the run shows no delta at all. - Observed result: before, `/metrics` completed 0 scrapes and the canary ticked once in 6419 ms. After, 206 scrapes at p50 0.1 ms and max 0.2 ms, and 418 canary ticks with a 92.3 ms worst gap. Total wall clock barely moved, 6419 ms to 6473 ms, which is the expected result and not a null one: the same disk work still serializes on the ledger's own `flock`, now in a thread instead of on the loop. Unit-test view of the same behavior, with a 500 ms stub standing in for the write: before, `event loop stalled 0.506s during a 0.500s ledger write` and the competing lock holder waited `+0.502s`; after, both pass. - Not tested: Windows, where `savings_ledger` already skips locking because `fcntl` is unavailable. Multi-process contention on one ledger file, which this change does not alter. The residual 92.3 ms loop gap after the fix, which traces to `SavingsTracker._save_locked`'s `os.fsync` (`savings_tracker.py:1445`) firing every 25th request from inside the same lock, a separate path this PR leaves alone. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A, no user-visible surface. ## Additional Notes Docs checklist item is N/A. Nothing user-facing moves; `headroom savings` reads the same ledger, with the same contents, written from a thread. Two related in-lock costs on this same code path are deliberately out of scope, one logical change per commit: `_current_savings_tracker_totals()` at `:798` rebuilds `cost_tracker.stats()` per request, and `_resolve_litellm_model` in `savings_tracker.py` is uncached across roughly seven calls per request. Happy to follow up on either. `make ci-precheck` was not run end to end. The uv-managed worktree venv has no `pip`, so the `ci-precheck-python` hook's `pip install -e .` step fails on this machine for reasons unrelated to the change. Ran `pytest`, `ruff`, and `mypy` directly instead, output above. No Rust touched. One heads up worth passing on, since it is why the numbers above are a sweep and not a single full-suite line. Seven test files do not complete on this macOS box: six hang and one fails. The hangs park the main thread in `_dispatch_semaphore_wait_slow` with CPU time frozen and never recover, and `pytest-timeout --timeout-method=signal` cannot break them out, so the block is native, below the interpreter. `tests/test_adversarial_grid.py::test_grid_shape_and_schema` is the first one a full run reaches. All seven reproduce identically on unmodified `c400f908` with this change reverted, so they predate the PR. I ran the reverted comparison specifically to rule out a thread-before-fork interaction from the new `to_thread` call, which was the plausible way this change could have caused it. It did not. Happy to open a separate issue with the sample output if that is useful. --- ## Follow-up: a cancellation bug this change introduced Self-review turned up a second problem in this change, so the fix rides along here. Moving the append into a thread added the first suspension point in `record_request` that can tear state. The metrics lock at `prometheus_metrics.py:701` suspends too, but only under contention, and it sits ahead of every mutation, so a cancellation there recorded nothing at all. The new await is different. It sits after the Prometheus counters commit and before OTel and the funnel's effects 2/3/4, and it suspends on every compressed request. Four of the funnel's call sites are `finally:` blocks inside streaming async generators (`streaming.py:1611`, `:1859`, `:2069`, `openai.py:8614`). A client disconnect cancels that task. The cancellation lands on the new await, so Prometheus counts the request while the cost tracker, the request log, and the PERF line `headroom perf` reads never see it. `emit_request_outcome` has one try/except and it sits before `record_request`, so nothing catches this. `_record_request_outcome` now wraps the funnel in `asyncio.shield`. One line at a single choke point, covering all 28 call sites. The shield leaves the cancellation itself alone: the await still raises `CancelledError`, so generator teardown propagates as before. Only the bookkeeping survives. Real stack, uvicorn 0.40.0 + starlette 1.3.1, raw-socket disconnect mid-stream: | effect | before | after | |---|---|---| | Prometheus counters | committed | committed | | ledger write | ran | ran | | OTel | **skipped** | ran | | cost tracker | **skipped** | ran | | request log | **skipped** | ran | | PERF line | **skipped** | ran | | caller sees `CancelledError` | yes | yes | The new test fails on its parent commit with a `TimeoutError`. ## Test changes Dropped `test_event_loop_keeps_running_during_the_ledger_write`. It detected a strict subset of what the lock test already detects: | scenario | lock test | loop test | |---|---|---| | correct: outside lock + `to_thread` | PASS | PASS | | regress: INSIDE lock + `to_thread` | FAIL | **PASS** | | regress: outside lock + sync write | FAIL | FAIL | | pre-fix: INSIDE lock + sync write | FAIL | FAIL | Added a concurrency test in its place, which covers what the offload actually introduces: before the move every proxy ledger write ran on the one event-loop thread and was serialised for free, and now N in-flight requests append from N worker threads. One thing left open. That same intra-process concurrency reaches `_maybe_compact`, which rewrites the file in place. On POSIX the ledger's own `flock` serialises it. On Windows `_HAS_FCNTL` is false and all locking is skipped, so a single Windows proxy can now interleave writers where the loop thread used to serialise them. The cross-process form of that is pre-existing and called out at `savings_ledger.py:38`. Happy to take the intra-process guard here or in a follow-up. Two notes on the sweep above, now that the diff is three files. The blast radius re-run at this head is 136 of 145 files green, and the nine non-green are identical with and without the change. Two of them (`test_gemini_function_response_waste.py`, `test_openai_responses_context_compaction.py`) are not in the seven listed earlier; I re-ran both against a reverted `server.py` and they hang the same way on both sides.
…els (headroomlabs-ai#2441) (headroomlabs-ai#2445) ## Description `headroom wrap opencode` currently routes `headroom/*` models only to ordinary Anthropic or OpenAI backends. A user with a GitHub Copilot subscription cannot point those `headroom/*` requests at the Copilot seat while keeping Headroom compression and stats, even though Headroom already has the validated subscription resolver, the proxy seed path, and the OpenCode provider route needed to do it. This PR adds `--copilot-subscription` to the OpenCode wrap command. It reuses the existing Copilot subscription token resolver, passes the validated endpoint and token seed into the existing proxy startup path, rejects unsupported runtime modes, and treats any non-empty Copilot API token as a private session seed so token-only sessions do not reuse a shared proxy. The generated OpenCode provider still targets the local proxy, and subscription secrets stay out of OpenCode config, environment, and terminal output. Closes headroomlabs-ai#2441 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `headroom wrap opencode --copilot-subscription` in `headroom/cli/wrap.py`. - Reuse the existing validated Copilot subscription resolver through one small required-resolution helper shared with the dedicated Copilot wrapper. - Pass the resolved endpoint and token seed into `_ensure_proxy()` as `openai_api_url`, `copilot_api_token`, `copilot_refresh_oauth_token`, and `copilot_api_token_expires_at`. - Reject `--copilot-subscription` with `--no-proxy`, `--prepare-only`, and translated backends before proxy or OpenCode launch. - Validate subscription mode before snapshotting OpenCode config, so rejected invocations don't create stale backups. - Scrub inherited Copilot proxy seed variables from the OpenCode child environment. - Treat any non-empty Copilot API token as a private session seed so token-only sessions do not reuse shared or persistent proxies. - Add focused OpenCode and persistent-proxy coverage for seed handoff, guard failures, direct-token isolation, secret non-disclosure, and unchanged non-subscription behavior. - Leave `CHANGELOG.md` untouched because Headroom generates changelog entries from conventional commits. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_cli/test_wrap_opencode.py tests/test_cli/test_wrap_persistent.py tests/test_cli/test_wrap_copilot.py -q`, `106 passed in 136.65s`) - [x] Linting passes (`uv run ruff check headroom/cli/wrap.py tests/test_cli/test_wrap_opencode.py tests/test_cli/test_wrap_persistent.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text Targeted subscription tests pass: OpenCode `7 passed, 37 deselected in 0.36s`, persistent proxy `2 passed, 29 deselected in 0.34s`, and dedicated Copilot `11 passed, 20 deselected in 0.39s`. Coverage includes inherited resolver-input env scrubbing, HEADROOM_BACKEND rejection, no-backup-on-rejection, OpenCode-only scrub scoping, and private-proxy teardown on config-injection failure. The full focused command `uv run pytest tests/test_cli/test_wrap_opencode.py tests/test_cli/test_wrap_persistent.py tests/test_cli/test_wrap_copilot.py -q` passed with `106 passed in 136.65s`. Ruff check and format check pass. ``` ## Real Behavior Proof - Environment: Windows, `uv` development environment, local CLI tests, no live Copilot seat on this host - Exact command / steps: run the focused OpenCode and persistent-proxy tests with a mocked `CopilotSubscriptionTokenResolution`, then capture the proof rows for seed handoff, direct-token isolation, guards, and secret non-disclosure - Observed result: Targeted subscription and proxy-seed tests pass, including OpenCode-only resolver-input env scrubbing and private-proxy teardown on config-injection failure; the full focused command passed with `106 passed in 136.65s`; Ruff check and format check pass. - Not tested: live Copilot subscription seat run on this host ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - Feature approval comes from the open `enhancement` label on headroomlabs-ai#2441. - Keep the final live-backend claim behind manual owner proof. Local CLI tests can prove config, guard, secret, and proxy-seed behavior, but they cannot prove a real Copilot seat on this host. - `CHANGELOG.md` remains untouched because Headroom's release pipeline generates changelog entries from conventional commits.
… request (headroomlabs-ai#2450) ## Description The per-request JSONL feed (`--log-file`) collapsed all cache signal into a single `cache_hit: bool`, defined as `cache_read_tokens > 0 or from_response_cache`. A call that was billed cache-*creation* (write) with zero reads is therefore indistinguishable from a real cache-*read* hit. On Claude Code traffic where the proxy pays repeated cache writes, this hides the real economics from users (the issue's "cache_hit inverts the user's real economics" telemetry complaint). The provider-truth counters already ride on `RequestOutcome`, `cache_read_tokens`, `cache_write_tokens`, `uncached_input_tokens`, parsed from the upstream response usage on every path (`handlers/anthropic.py`, `handlers/streaming.py`, `backends/litellm.py`) but were dropped when the `RequestLog` entry was constructed in `emit_request_outcome`. This surfaces them per call. Refs headroomlabs-ai#2438 (Finding 1, telemetry sub-item). The core prompt-cache preservation regression (Finding 1) and Finding 3 are architectural and tracked separately. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Add `cache_read_tokens`, `cache_write_tokens`, `uncached_input_tokens` (optional, default `0`) to `RequestLog` (`headroom/proxy/models.py`). Optional so existing consumers and serialized logs stay backward compatible. - Populate the three fields at the single log-emit site in `emit_request_outcome` (`headroom/proxy/outcome.py`) from the values already on `RequestOutcome`. `cache_hit` is unchanged. - Add `tests/test_proxy_cache_telemetry.py`: drive `emit_request_outcome` through the real proxy funnel with logging enabled and assert the JSONL entry carries the write/uncached deltas even when `cache_hit` is False; plus a default-value backward-compat check. - Leave `CHANGELOG.md` untouched release-please generates it. ## Testing - [x] Unit tests pass (`python -m pytest tests/test_proxy_cache_telemetry.py -q`) - [x] Linting passes (`ruff check`, `ruff format --check` on the three changed files) - [x] Type checking passes (`mypy headroom/proxy/models.py headroom/proxy/outcome.py --ignore-missing-imports`) - [x] New tests added for new functionality ### Test Output ```text $ python -m pytest tests/test_proxy_cache_telemetry.py -q 2 passed, 1 warning in 8.51s $ ruff check headroom/proxy/models.py headroom/proxy/outcome.py tests/test_proxy_cache_telemetry.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13, local dev checkout on a branch off upstream/main - Exact command / steps: Built a `RequestOutcome` with `cache_read_tokens=0, cache_write_tokens=800, uncached_input_tokens=200` and ran it through `emit_request_outcome` against a real proxy app (`create_app`) with `log_requests=True` and a temp `log_file`, then read the JSONL back. - Observed result: The written entry carries `cache_read_tokens=0`, `cache_write_tokens=800`, `uncached_input_tokens=200` a cache-write-only call is now distinguishable from a cache-read hit in the log, where previously only `cache_hit` (False here) was recorded. - Not tested: A live Anthropic call end to end from this environment, the funnel is exercised with a synthetic outcome carrying real provider-usage values instead. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…on (headroomlabs-ai#2449) ## Description Under `--target-ratio 0.4` a session died mid-run with a fatal Anthropic 400: ``` messages.13.content.0.server_tool_use.input: Input should be an object ``` Root cause is not compression of the request: the request path passes structured blocks through byte-for-byte. It is **SSE stream reconstruction**. When the proxy rebuilds a full Anthropic message from the streamed response (non-stream retry, buffered, and CCR round-trip paths), the `content_block_stop` handler parsed the accumulated `_partial_json` into `input` only for blocks whose type was exactly `tool_use`. A `server_tool_use` block streams its input identically via `input_json_delta`, so its input was never reassembled: the block kept the empty start-event `input: {}` and leaked the internal `_partial_json` scratch key. That reconstructed block becomes assistant history, and on the next turn the client replays it, so Anthropic rejects `server_tool_use.input`. `--target-ratio` only makes the buffered/reconstructed path more likely; it does not itself rewrite the block. Refs headroomlabs-ai#2438 (Finding 2). Findings 1 (prompt-cache regression) and 3 (compression not engaging) are architectural and tracked separately. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/streaming.py` (`_parse_sse_to_response`): gate the `content_block_stop` `_partial_json` → `input` parse on the presence of `_partial_json`, not `type == "tool_use"`, so `server_tool_use` (and any future tool-ish block) is reassembled. Always strip the scratch key; `input` is always a parsed object (`{}` on malformed/empty JSON). - `headroom/ccr/response_handler.py` (`StreamingCCRHandler._reconstruct_anthropic_response`): same stop-handler fix, and relax the `input_json_delta` accumulator that was likewise gated on `type == "tool_use"` so server_tool_use partial JSON is accumulated at all. - Regression tests in `tests/test_sse_thinking_blocks.py` and `tests/test_ccr_response_handler_extra.py`: a `server_tool_use` whose input arrives via `input_json_delta` must reconstruct to the parsed object with no `_partial_json` leak. - Leave `CHANGELOG.md` untouched, release-please generates it. ## Testing - [x] Unit tests pass (`python -m pytest tests/test_sse_thinking_blocks.py tests/test_ccr_response_handler_extra.py -q`) - [x] Linting passes (`ruff check`, `ruff format --check` on the four changed files) - [x] Type checking passes (`mypy headroom/proxy/handlers/streaming.py headroom/ccr/response_handler.py --ignore-missing-imports`) - [x] New tests added for new functionality ### Test Output ```text $ python -m pytest tests/test_sse_thinking_blocks.py tests/test_ccr_response_handler_extra.py -q 26 passed in 3.36s $ ruff check <changed files> All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13, local dev checkout on a branch off upstream/main - Exact command / steps: Fed a synthetic Anthropic SSE stream with a `server_tool_use` block whose `input` arrives as `input_json_delta` partial JSON through both reconstructors (`_parse_sse_to_response`, `_reconstruct_anthropic_response`); then temporarily restored the `type == "tool_use"` guard and re-ran. - Observed result: With the fix, the reconstructed block has `input == {"query": ...}` and no `_partial_json` key. With the old guard the test fails, `input` stays `{}` and the scratch key leaks, reproducing the malformed block that Anthropic rejects on replay. - Not tested: End-to-end multi-turn `--target-ratio` session against the live Anthropic API from this environment, reproduced at the reconstruction seam instead; the reporter observed the 400 on real traffic. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
… (headroomlabs-ai#2455) ## Description PR headroomlabs-ai#2445 added the missing OpenCode subscription path, but the shared Copilot subscription resolver still lets Business and Enterprise payload hosts route through segmented `*.githubcopilot.com` domains and still drops an explicit `GITHUB_COPILOT_API_URL` pin on two resolution paths. This follow-up moves the final hosted-route decision back into the shared resolver, normalizes `api.business.githubcopilot.com` and `api.enterprise.githubcopilot.com` to the generic host by default, and makes the explicit pin win on token exchange, explicit API token, and Copilot-token candidate resolution. Both `headroom wrap copilot --subscription` and `headroom wrap opencode --copilot-subscription` inherit the same fix because they already consume the same `CopilotSubscriptionTokenResolution.api_url`. Refs headroomlabs-ai#2441. Attribution: headroomlabs-ai#2445 (comment) reported and narrowed the Business or Enterprise regression, and headroomlabs-ai#2445 (comment) scoped the shared-resolver follow-up that this change implements. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Centralize subscription hosted-route selection so explicit `GITHUB_COPILOT_API_URL` pins win on token exchange, explicit API token, and Copilot-token candidate resolution. - Normalize `api.business.githubcopilot.com` and `api.enterprise.githubcopilot.com` to `https://api.githubcopilot.com` by default, extending the existing individual-seat normalization. - Extend focused auth and wrapper tests so both subscription wrappers prove the corrected shared resolver output and the private-proxy isolation contract stays intact. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_copilot_auth.py -q`) - [x] Unit tests pass (`uv run pytest tests/test_cli/test_wrap_copilot.py -q`) - [x] Unit tests pass (`uv run pytest tests/test_cli/test_wrap_opencode.py -q`) - [x] Unit tests pass (`uv run pytest tests/test_cli/test_wrap_persistent.py -q`) - [x] Linting passes (`uv run ruff check .`) - [x] Formatting check passes (`uv run ruff format . --check`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_copilot_auth.py -q -> 84 passed uv run pytest tests/test_cli/test_wrap_copilot.py -q -> 31 passed uv run pytest tests/test_cli/test_wrap_opencode.py -q -> 44 passed in 143.46s uv run pytest tests/test_cli/test_wrap_persistent.py -q -> 31 passed uv run ruff check . -> All checks passed! uv run ruff format . --check -> 1331 files already formatted ``` ## Real Behavior Proof - Environment: Windows - Exact command / steps: Run the focused auth, Copilot wrapper, OpenCode wrapper, and persistent-proxy pytest files after implementing the shared resolver change, then ask lucasp1337 to rerun the Business or Enterprise `--copilot-subscription` scenario from PR headroomlabs-ai#2445 (comment) on a real seat. - Observed result: Focused auth and wrapper pytest runs passed locally, including the enterprise-host exchange reproduction row, explicit-pin precedence on all three producer paths, both subscription wrapper routes, and the private-proxy isolation regression. Live Business or Enterprise success stays behind reporter retest. - Not tested: live Business or Enterprise tenant run ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes No `CHANGELOG.md` edit is needed because Headroom generates release notes from conventional commits. Risk for maintainers: PR headroomlabs-ai#641 manually validated a Business seat against the GitHub-returned hosted domain in June on `gpt-5.4`, so generic-by-default could affect tenants that genuinely require a dedicated host. This follow-up keeps the documented escape hatch intact by making `GITHUB_COPILOT_API_URL` win on every path. Live-seat proof boundary: lucasp1337 offered to retest on a Business or Enterprise seat in PR headroomlabs-ai#2445 (comment). Keep any live success claim behind that rerun.
…nx" (headroomlabs-ai#2448) ## Description With `HEADROOM_KOMPRESS_BACKEND=onnx_coreml`, every Kompress compression call and the startup canary crash with `'_OnnxModel' object has no attribute 'parameters'`, so Kompress silently degrades to passthrough and `/health` reports `kompress: unhealthy, backend: null`. Root cause: `headroom/transforms/kompress_compressor.py` gated the ONNX-vs-PyTorch branch with an exact string match `backend == "onnx"`. But `_load_kompress_onnx` returns `onnx_coreml` (CoreML) or `onnx_cpu` — never the bare string `onnx`. So under `onnx_coreml` the code built PyTorch tensors and dispatched to a device via `next(model.parameters())`, which the `_OnnxModel` wrapper doesn't implement. This is the accelerated backend Apple Silicon users reach for, so the fast path is exactly the broken one. Fixes headroomlabs-ai#2442 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Change the four exact-match `backend == "onnx"` sites in `headroom/transforms/kompress_compressor.py` to `backend.startswith("onnx")`, matching the convention already used by `_model_device_type`: `_timed_canary`, `compress`, `compress_batch`, and the batch-parallelism guard in `_should_use_sequential_fallback`. - Update the guard comment ("ONNX CPU provider" → "ONNX EPs") since it now covers all ONNX execution providers. - Add regression tests exercising `_timed_canary` on `onnx_coreml` (must take the numpy path and never touch `.parameters()`) with a negative control proving the PyTorch branch still dispatches to a device. - Leave `CHANGELOG.md` untouched — release-please generates it from conventional commits. - Out of scope: the secondary `/health` under-reporting the issue flags as informational (deferred-preload warmup object never flips to `loaded`). ## Testing - [x] Unit tests pass (`python -m pytest tests/test_transforms/test_kompress_compressor.py::TestOnnxBackendPrefixGating -q`) - [x] Linting passes (`ruff check`, `ruff format --check` on the two changed files) - [x] Type checking passes (`mypy headroom/transforms/kompress_compressor.py --ignore-missing-imports`) - [x] New tests added for new functionality ### Test Output ```text $ python -m pytest tests/test_transforms/test_kompress_compressor.py::TestOnnxBackendPrefixGating -q collected 2 items tests\test_transforms\test_kompress_compressor.py .. [100%] 2 passed in 2.20s $ ruff check headroom/transforms/kompress_compressor.py tests/test_transforms/test_kompress_compressor.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13, local dev checkout on a branch off upstream/main (no Apple Silicon / CoreML hardware available) - Exact command / steps: Ran the new `TestOnnxBackendPrefixGating` regression; then temporarily reverted one site back to `backend == "onnx"` and re-ran to confirm the test discriminates. - Observed result: With the fix, `_timed_canary(model, tokenizer, "onnx_coreml")` returns a float and never touches `.parameters()`. Reverting one site makes the onnx_coreml test fail (it takes the `pt` tensor path and hits the paramless model), proving the test catches the exact bug. The issue reporter separately verified the fix on real Apple Silicon hardware (onnxruntime 1.27.0, CoreMLExecutionProvider): zero occurrences of the error afterward and compression completing on the CoreML session. - Not tested: End-to-end run on real CoreML hardware from this environment — reproduced via the unit-level device-dispatch seam instead; hardware confirmation is in the issue. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…e prefixes (headroomlabs-ai#2369) ## Description On an HTTP tool-output re-read, `cross_turn_dedup` folds a contiguous span that already appeared in an earlier block into a compact pointer, and when the line numbers shifted by a constant it carries the offset as a `delta` so the original bytes recover as `int(number) + delta`. The module states this renumber path is "strictly lossless" for UNPADDED numbers only. `_LINENO_RE = ^(\d+)(:|\t)(.*)$` does not enforce the "unpadded" restriction: `\d+` also matches a LEADING-ZERO prefix. A timestamped log row such as `08:00:01 ...` is read as line number `8`, not as data, so a re-read shifted by a constant (a later window of the same hourly log) folds under a uniform delta. Recovery then renders `str(int("08") + 1)` = `"9"`, not `"09"`: the round-trip is not byte-exact. This is a lossy (false-positive) fold in a module whose stated posture is to prefer false negatives (`CONTRIBUTING.md:129`, `cross_turn_dedup.py:45-50`). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/transforms/cross_turn_dedup.py`: restrict `_LINENO_RE` to `[1-9]\d*` so a leading-zero run stays non-numbered and can fold only on an EXACT match (delta 0), never under a lossy renumber. Real `grep -n` / `sed -n` / `rg -n` numbers never carry a leading zero, so the intended renumber-fold feature is unchanged. Added a comment stating why the character class is load-bearing. - `tests/test_cross_turn_dedup.py`: added a delta-aware reconstruction helper and three regression tests (the existing `_reconstruct` asserts delta is absent, so it never exercised the numbered path this bug lives on). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed Three named scenarios, one test each: 1. `test_zero_padded_prefix_not_folded_lossily`: a padded shifted re-read is left verbatim (`spans_folded == 0`). This fails on `main` (it folds under a delta). 2. `test_unpadded_renumber_still_folds_and_recovers_exactly`: an unpadded `grep -n` read renumbered by `+5` still folds and reconstructs byte-exact (feature guard). 3. `test_padded_content_exact_redisplay_still_folds`: the same padded rows re-displayed verbatim still fold with delta 0 (surgical-scope guard). ### Test Output ```text --- ruff check --- All checks passed! --- ruff format --check --- 2 files already formatted --- mypy --- Success: no issues found in 1 source file --- pytest: 3 new tests on the branch (fixed) --- 3 passed, 14 deselected --- pytest: revert regex to \d+ (simulate main): the regression test must FAIL --- 1 failed ``` ## Real Behavior Proof - Environment: clean `python:3.12-slim` Docker, `PYTHONPATH` at the source tree, core deps installed by name (tiktoken, pydantic, litellm, click, rich, opentelemetry-api, pyyaml, tomlkit), `ruff==0.15.17`, `mypy==1.20.2`. - Exact command / steps: import the module and print provenance, then run ruff, ruff format, mypy on the two changed files, then `pytest` the three new tests on the branch, then revert only the regex to `\d+` and re-run the regression test. - Observed result: module `cross_turn_dedup.py` (sha256 7ff204b65f40617d505895841aa1cfc1ee54bf63ffe6520198f0aa05a850aa8d), regex now `^([1-9]\d*)(:|\t)(.*)$`; ruff, ruff format, mypy all green; branch `3 passed`, reverted-regex main `1 failed`. Breakdown: - `module: /src/headroom/transforms/cross_turn_dedup.py` - `sha256: 7ff204b65f40617d505895841aa1cfc1ee54bf63ffe6520198f0aa05a850aa8d` - `regex : ^([1-9]\d*)(:|\t)(.*)$` - ruff, ruff format, mypy: all green (output above). - Branch: `3 passed`. Reverted-regex main: the regression test `1 failed`. - Not tested: the router-level and Rust-backed integration tests in this file (`test_apply_*`, `test_dedup_*`) need the compiled `headroom._core` extension, which is not built in this lightweight container; they are `ModuleNotFoundError` on both `main` and this branch here, so they were not exercised. The change is a pure-stdlib regex in a pure-stdlib function; the unit-level `dedup_blocks` tests above cover it directly. I also did not measure how often real-world tool output hits the leading-zero shifted shape; the argument is the module's own strictly-lossless contract, not observed field frequency. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (N/A: no doc surface) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Per `CONTRIBUTING.md` ("Bug or small fix -> Open a PR with repro + test"), this goes straight to a PR rather than an issue. One concern only; no dependency or generated-file changes.
Compression savings were reported gross, but every CCR retrieval re-injects
the original content (plus tool-call overhead) as real billed tokens that
offset them. Record retrieved-token volume at each retrieval site and make
savings net:
- Savings view (headroom savings, session summary, dashboard, /stats) debits
retrieved tokens + a fixed per-retrieval overhead; net is never clamped and
renders negative when retrievals outweigh compression.
- Cost view attributes retrieval cost without debiting it — the continuation
usage is already billed, so debiting again would double-count.
- Durable ledger gains kind ("compress"|"retrieve") and tokens_retrieved;
missing kind reads as compress (backward compatible). Both MCP-tool and
proxy in-process retrievals write retrieve events.
- Proxy CCR handler captures dropped first-response usage as overhead.
- Dashboard adds a net "CCR Retrievals" card; lifetime tile labelled gross.
Adds unit + payload tests; existing /stats keys unchanged (additive only).
…rable retrievals The CCR card was gated on session-scoped retrieval counts that reset on every proxy restart, so it vanished whenever the current process had no retrievals. Render it unconditionally, and surface the savings ledger's lifetime retrieval totals (retrievals / tokens_retrieved / net) via persistent_savings.lifetime so retrievals recorded by prior proxy processes remain visible after a restart.
PR governanceThis PR does not yet satisfy the required template fields:
Please update the PR body, or move the PR back to draft while it is still in progress. |
github-actions
Bot
added
the
status: needs author action
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes #
Type of Change
Changes Made
Testing
pytest)ruff check .)mypy headroom)Test Output
Real Behavior Proof
Review Readiness
Checklist
CHANGELOG.md— it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this)Screenshots (if applicable)
Add screenshots to help explain your changes.
Additional Notes