feat: compression extraction — Rust knob exposure, CCR hardening, traffic audits by chopratejas · Pull Request #818 · headroomlabs-ai/headroom

chopratejas · 2026-06-10T16:39:44Z

Description

A data-driven push for better compression savings without accuracy loss, in four parts: expose and tune the Rust compressor knobs, harden the CCR retrieval store, add traffic-audit tooling that sizes opportunities from real transcripts, and introduce read maturation — a new, live-validated mechanism that compresses Read outputs before they ever enter the provider prefix cache.

Type of Change

Bug fix (non-breaking change that fixes an issue)
New feature (non-breaking change that adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation update
Performance improvement
Code refactoring (no functional changes)

Changes Made

1. Rust compressor extraction

Expose lossless_min_savings_ratio end-to-end and lower the default 0.30 → 0.15 (lockstep across Rust, PyO3, and both Python config classes) so the lossless Table/CSV compaction path wins more often.
Expose the CompactConfig heuristics (core-field fraction, heterogeneity ratio, flatten cap, bucket bounds) through PyO3 + Python.
SearchCompressor grouped-by-file output (rg --heading style — path once per file instead of per match). Library default off; the proxy enables it in token mode.
Complete factor_out_constants: constant fields now emit once in a _constant_fields sentinel with slim rows (defensive per-item value match; default off).
ContentRouter accepts a SmartCrusher config override and the search-grouping knob.

2. CCR store hardening

Session-scale TTL: 300s → 1800s (CCRConfig, CompressionEntry, CompressionStore, Rust DEFAULT_TTL — lockstep).
SQLite is the default CCR backend (~/.headroom/ccr_store.db, WAL): survives proxy restarts and is shared across workers. HEADROOM_CCR_BACKEND=memory opts out.
Multi-worker safety: busy_timeout, and corruption detection narrowed so transient SQLITE_BUSY errors can never trigger database deletion.
Data-at-rest hygiene: chmod 600 on db + sidecars, expired rows swept at open.
Retrieval-miss messages are actionable (re-read the file / re-run the command).

3. Traffic audit tooling (measure before tuning)

headroom audit-reads: sizes Read opportunities from local Claude Code transcripts (read share, stale %, line-number overhead, context residency, cache-death windows).
--simulate-maturation: Mechanism B risk sizing (re-read rates, never-touched-again share, quiesce coverage, at-risk edits).
--codex: shell-read classifier for Codex transcripts (rtk-wrapper aware, workdir resolution).
Findings that shaped this PR (81 sessions): Reads are 67% of tool bytes; median Read lingers 118 turns (~13x lifetime cost); a prototyped repeat-Read dedup measured 0.1% and was removed rather than shipped as dead code.

4. Read maturation (Mechanism B) — experimental, default OFF

Activity-based: a fresh large Read is held out of the provider cache (trailing breakpoint relocated before it), stays verbatim while its file is active, and matures into a CCR-backed marker once the file is quiet for quiesce_turns (default 5; max_hold_turns bounds busy files).
Only the final compressed form ever enters the cache — no cached byte is ever mutated; matured markers replay byte-identically.
Wired into the Anthropic handler behind --read-maturation / HEADROOM_READ_MATURATION=1; session state rides on the prefix tracker; advisory (can never fail a request).
Live-validated against the Anthropic API: held content excluded from cache_creation; after maturation the prior cached prefix still served — the no-bust invariant holds end-to-end.

5. Rebase / CI fixups (this update)

Rebased onto latest main (was 28 commits behind): picks up ci: pass CODECOV_TOKEN to coverage uploads (#968), which is what was turning the 4 test shards red — the tests themselves passed (1528) but the post-test codecov upload exited non-zero on a protected branch.
Resolved the duplicate lossless_min_savings_ratio that two independent main/branch additions left in SmartCrusherConfig and the Rust-config kwarg (import-time SyntaxError + mypy no-redef).
Aligned CCR tests with the new defaults (SQLite backend, 1800s TTL) across test_ccr, test_adapter_hooks, test_compression_store, test_proxy_ccr, and the lossy row-drop bridge test.

Testing

Unit tests pass (pytest)
Linting passes (ruff check .)
Type checking passes (mypy headroom)
New tests added for new functionality
Manual testing performed

Test Output

$ python -m pytest tests/test_proxy_ccr.py tests/test_ccr.py tests/test_compression_store.py tests/test_adapter_hooks.py tests/test_ccr_row_drop_store_bridge.py -q
170 passed, 4 warnings in 42.49s

$ python -m pytest tests/test_audit_reads.py tests/test_audit_codex.py tests/test_read_maturation.py tests/test_transforms_content_router.py tests/test_smart_crusher_toin_attachment.py -q
83 passed

$ mypy headroom/
Success: no issues found in 365 source files

$ python -m compileall headroom/ -q
COMPILE-OK

# CI (run 27488990477, pre-rebase head): all 4 shards ran to completion —
#   "1528 passed, 120 skipped, 4922 deselected"
# The red shards were the codecov upload step, not test failures; fixed by
# the #968 rebase above.

Real Behavior Proof

Environment: macOS (darwin), Python 3.12 venv; branch feat/compression-extraction rebased onto origin/main (head 7cb0f43); GitHub Actions CI run 27488990477 for the test shards
Exact command / steps: rebased onto latest main (clean, 13 commits replayed, 0 conflicts); ran the pytest suites and mypy above locally; inspected CI shard logs to confirm the failure was the codecov upload, not the test phase
Observed result: 253 targeted tests pass locally; mypy clean on 365 files; CI test phase reports 1528 passed, 120 skipped; the only red step (codecov upload-coverage → "Token required because branch is protected") is resolved by the rebased-in ci: pass CODECOV_TOKEN to coverage uploads (fixes red test shards) #968 CODECOV_TOKEN fix
Not tested: the read-maturation live-API no-bust validation (tests/test_live/) was not re-run in this rebase pass (requires provider keys); it was validated when the feature first landed, and no maturation code changed in the rebase — only CCR-default test assertions and the duplicate-field resolution

Review Readiness

I have performed a self-review
This PR is ready for human review

Checklist

My code follows the project's style guidelines
I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
I have updated the CHANGELOG.md if applicable

Additional Notes

CHANGELOG is generated by release-please from the conventional commits, so the CHANGELOG box is intentionally left unchecked. "Manual testing performed" is unchecked deliberately — see Real Behavior Proof → Not tested for the exact boundary (the live-API maturation validation was not re-run in this rebase pass).

Follow-ups (tracked, not in this PR)

Mechanism B provider extensions: OpenAI-family wiring (no breakpoint hold — bounded near-tail bust) and the Codex runtime read-detector (the audit classifier is the prototype).
Pilot enablement playbook: run audit-reads --simulate-maturation on target traffic → pick quiesce_turns → enable via env → watch cache hit rate + read_maturation:N transform tags.

JerrettDavis · 2026-06-10T18:08:08Z

This has a lot of promising work in it, but it is too large to review while one of the main CI jobs is red. Please fix or explain the failing test (1) job and rerun the suite on the current branch.

Once CI is clean, I would treat this as a high-blast-radius review: Rust/PyO3 config exposure, the SQLite CCR backend default, and read maturation all need careful maintainer review before merge.

JerrettDavis

Reviewed the diff and failing GitHub job log. I did not run PR code locally. This head is not ready because est (3) fails in ests/test_proxy_ccr.py: est_retrieve_nonexistent_hash still expects CCR TTL: 300 seconds while the implementation now returns CCR TTL: 1800 seconds, and est_stats_empty_store asserts 1800 == 300. Please update the remaining CCR proxy endpoint expectations to the new session-scale default or keep the old default if that was not intended, then get the normal CI suite green.

…t-Read dedup Compressor extraction: - Expose lossless_min_savings_ratio to Python and lower the default 0.30 -> 0.15 (lockstep in Rust, PyO3, and both Python config classes) so the lossless Table/CSV compaction path wins more often - Expose CompactConfig heuristics (core_field_fraction, heterogeneous_core_ratio, max_flatten_inner_keys, min/max_buckets) end-to-end; SmartCrusher::new now builds the compaction stage from the crusher config instead of pinning CompactConfig::default() - SearchCompressor group_by_file output mode (rg --heading style): path emitted once per file instead of on every match line; enabled by the proxy in token mode, library default unchanged - Complete factor_out_constants: constants the analyzer extracts are now emitted once in a {"_constant_fields": ...} sentinel with slim rows; defensive per-item value match; still default-off - ContentRouter accepts a SmartCrusher config override and the search grouping knob CCR hardening: - Session-scale TTL: 300s -> 1800s across CCRConfig, CompressionEntry, CompressionStore, and Rust DEFAULT_TTL (an expired entry silently converts lossless-with-retrieval into lossy) - SQLite is the default CCR store backend (~/.headroom/ccr_store.db, WAL): survives proxy restarts and is shared across workers, closing the largest multi-worker retrieval gap; HEADROOM_CCR_BACKEND=memory opts out - Retrieval-miss messages are actionable (re-read the file / re-run the command) instead of reporting a stale 5-minute TTL Repeat-Read dedup (ReadState.DEDUP_REPEAT): - When a NEW Read is byte-identical to an earlier Read of the same file still in context, the NEW copy becomes a pointer marker; the earlier copy keeps the verbatim bytes. Cache-safe by construction: the new copy is at the conversation tail and was never cache-written, unlike compress_superseded which mutates the cached older copy - Both client formats supported (Anthropic tool_result blocks, OpenAI role=tool messages); stale wins over dedup; frozen prefix untouched; deterministic markers preserve the load-bearing 'Retrieve original: hash=' phrase Tests: 38 new unit tests (both wire formats, frozen-boundary safety, determinism, SQLite roundtrip/restart survival, config plumbing, grouped output), 4 live API tests under tests/test_live/ (skipped without keys) proving real Anthropic + OpenAI endpoints accept the transformed shapes and models answer correctly through dedup pointers. 887 Rust tests and 378 existing transform/store tests pass.

…nities from real transcripts Streams local Claude Code transcripts (read-only) and reports, per mechanism, the addressable Read bytes: identical repeats, subset containment, write-readback, stale reads, line-number scaffolding, file-class split, per-Read context residency, and cache-death windows. Run on a deployment's traffic BEFORE tuning compression defaults — opportunity sizes vary heavily by workload. On the development corpus (81 sessions): Reads are 67.1% of tool bytes, median context residency is 118 assistant turns per Read, stale reads are 14.5% of Read bytes, line-number scaffolding 8.9% — while identical repeats are just 0.1%.

…% measured impact audit-reads over 81 real sessions measured byte-identical repeat Reads at 3 occurrences / 0.1% of Read bytes: the mechanism is dead code on real traffic. Removed rather than left default-on; the implementation remains in this branch's history if a deployment's audit ever shows a material identical-repeat share. Live API tests keep the stale-marker and grouped-search coverage on both client formats (the OpenAI dedup test became a stale-marker test so Codex-format marker validation survives).

…efore cache entry Motivated by audit-reads: median Read context residency is 118 assistant turns, so a Read's lifetime cost is ~13x its size — and the only cache-safe moment to shrink it is before it is ever cache-written. Core state machine (ReadMaturationManager): a fresh large Read is held out of the provider cache for hold_requests requests (trailing message breakpoint relocated to just before it via relocate_cache_breakpoint), stays verbatim while the model acts on it, then matures into a CCR-backed marker. Only the final compressed form enters the cache; no cached byte is ever mutated. Matured markers replay byte-identically on every subsequent request. Safety: frozen-prefix content untouched; blocks carrying client cache_control untouched; read_lifecycle markers respected; both client formats supported; full originals stored under strategy 'read_maturation' with the standard re-read recovery contract. Flag-gated OFF (ReadMaturationConfig.enabled=False). Proxy handler wiring and pilot validation are the next increment; 16 unit tests cover the state machine, determinism, CCR retrieval, and breakpoint relocation.

…lator into audit-reads Policy change driven by simulation over real traffic (now reproducible via `headroom audit-reads --simulate-maturation`): - Touch gaps are fat-tailed (next-touch p50=4 turns, p90=81) — no fixed hold window covers the tail. Replaced hold_requests with an activity rule: a held Read matures once its FILE has been quiet for quiesce_turns (default 5, covering the median activity cluster), with max_hold_turns=25 bounding the hold cost for ever-busy files. - Empirical safety gain: at-risk edits (edit lands on a file whose read has matured) drop from 36% under fixed hold=1 to 11.7% at quiesce=5. - Recovery is the model's observed behavior, not a hope: 35.5% of real reads are re-reads of an already-read file, and 95% of those are partial-range reads made while the full text was still in context — the model already prefers going back to disk. - Holding is now derived from the conversation itself (per-file last-touch scan), so the decision survives proxy restarts; only matured markers are stateful. audit-reads gains --simulate-maturation (text + json), reporting re-read rates, never-touched-again share (60.7% of big reads — pure savings), quiesce-window coverage, and activity-based at-risk edits. 30 tests across the simulator and the rewritten decision machinery.

Codex reads files through shell commands (cat / sed -n / head / tail / rtk read), so the Claude Code audit cannot see them. This adds a shell command classifier (rtk-wrapper aware, workdir path resolution, range detection) and a Codex transcript audit. Findings on the development corpus (144 sessions, 50MB tool output): reads are 51.9% of output bytes; 66% are partial slices; 55% of reads target an already-read path; hottest file read 393 times. 78% of read outputs clear the 2KB maturation floor — the slice-grinder profile that read maturation targets. The classifier doubles as the read-detector adapter Codex needs for Mechanism B.

…lag-gated) Wiring, all behind ProxyConfig.read_maturation (default off; --read-maturation / HEADROOM_READ_MATURATION=1): - Runs after the compression pipeline (read_lifecycle markers respected) and before body assembly, so held-Read breakpoint relocation lands in the forwarded request. - Session state (matured markers) rides on the PrefixCacheTracker — typed slot, same session affinity and TTL cleanup as freeze state. - Maturation savings are counted into optimized_tokens/tokens_saved and tagged in transforms_applied as read_maturation:N. - Advisory by construction: wrapped so a maturation failure can never fail the request. The same call site serves Vertex Claude once the vertex-anthropic pipeline routing lands (shared handler path). OpenAI-family wiring (maturation without breakpoint hold — bounded near-tail bust on implicit caching) and the Codex shell-read detector adapter are the remaining provider extensions. Next gate before enabling anywhere: live validation that provider cache_read/cache_creation numbers behave as predicted across a hold -> mature -> re-freeze cycle.

… live no-bust validation Bug found while designing the live validation: Claude Code parks its cache breakpoint on the newest content block — which, right after a Read, is the Read's tool_result itself. The 'never touch cache_control blocks' guard therefore skipped holding exactly the most common case, letting the verbatim read get cache-written. Under this mechanism the proxy owns breakpoint placement, so the guard is removed: held reads keep verbatim content while relocate_cache_breakpoint() strips/moves their breakpoint. Adds the live validation gate (tests/test_live/test_live_maturation.py), PASSING against the real Anthropic API: - Request A (held): provider cache_creation excludes the read content — the hold works. - Request B (matured): provider cache_read covers >=90% of request A's cache_creation — the prefix survived the read's replacement, proving breakpoint relocation does not break prefix matching and nothing is ever busted. - Matured request's uncached input is marker-sized, not read-sized. The mechanism's central design claim is now empirically validated end-to-end.

Bug found by reasoning through multi-worker write contention: sqlite3.OperationalError ('database is locked' / SQLITE_BUSY) is a DatabaseError subclass, so the corruption handler treated a routine lock collision as corruption and DELETED the database — destroying every stored original while sibling workers still held handles to the unlinked inode (split-brain). Fixes: - PRAGMA busy_timeout=5000: writers wait for the lock instead of failing (writes are tiny; contention resolves in milliseconds). - Corruption detection narrowed to explicit messages ('malformed', 'not a database'); all other DatabaseErrors are logged as transient and the operation degrades to a miss — data is never destroyed over a transient error. Data-at-rest hygiene (originals can contain sensitive tool output, e.g. .env reads): - chmod 600 on the db and its -wal/-shm sidecars. - Expired rows swept at every open, so a quiet store no longer holds expired originals on disk indefinitely (opportunistic purge only ran on writes). HEADROOM_CCR_BACKEND=memory remains the no-disk opt-out. 5 new tests: busy-does-not-delete, corruption-does-reset, busy_timeout configured, purge-on-open, file privacy.

…rop path - _create_default_ccr_backend now returns SQLiteBackend when no env is set (persistent default); test redirects HEADROOM_CCR_SQLITE_PATH to tmp_path so it never touches the real store. - CCRConfig.store_ttl_seconds default moved to session-scale 1800. - row-drop bridge test: make the per-item blob unique so the cheaper lossless Table/CSV path (lossless_min_savings_ratio 0.30 -> 0.15) cannot win — the test exists to exercise the LOSSY row-drop path and its Rust -> Python CCR store bridge.

Main independently exposed lossless_min_savings_ratio (at 0.30) while this branch lowered it to 0.15 with a getattr fallback; the rebase auto-merge kept both copies of the dataclass field and the Rust-config kwarg without a conflict marker (SyntaxError at import, caught by mypy no-redef). Keep the branch's 0.15 default and defensive getattr form.

…efault The CCR store default TTL moved 300 -> 1800 (session-scale); two assertions in test_proxy_ccr.py still expected the old 300s value in the miss-detail string and the stats endpoint. test_compression_store and test_ccr were already updated; this was the last straggler.

github-actions · 2026-06-15T06:41:19Z

PR governance

This PR follows the template and is marked ready for human review.

Copilot

Pull request overview

This PR expands Headroom’s compression system with new tunable Rust-backed knobs, strengthens CCR persistence/safety (SQLite default + longer TTL), adds offline traffic-audit tooling, and introduces an experimental “read maturation” mechanism (Mechanism B) that keeps fresh Reads out of the provider prefix cache until they quiesce, then replaces them with CCR-backed markers.

Changes:

Exposes additional Rust SmartCrusher and SearchCompressor configuration knobs end-to-end (Rust ⇄ PyO3 ⇄ Python) and updates defaults (notably lossless_min_savings_ratio).
Hardens CCR storage with a session-scale TTL default (1800s) and a new SQLite backend as the default global/proxy store, with multi-worker and corruption-handling safeguards.
Adds new audit tooling (headroom audit-reads, Codex mode, and maturation simulation) plus unit/live tests for maturation, lifecycle markers, and grouped search output.

Reviewed changes

Copilot reviewed 43 out of 44 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/test_read_maturation.py	Unit tests for Mechanism B decision logic, replay determinism, and breakpoint relocation.
tests/test_proxy_ccr.py	Updates CCR TTL expectations to 1800s.
tests/test_live/test_live_openai.py	Live OpenAI validation for lifecycle marker shapes in `role="tool"` messages.
tests/test_live/test_live_maturation.py	Live Anthropic validation of Mechanism B “no-bust” cache invariant.
tests/test_live/test_live_anthropic.py	Live Anthropic validation for lifecycle markers and grouped search format readability.
tests/test_live/conftest.py	Loads provider keys from repo-root `.env` for live tests (when present).
tests/test_live/init.py	Initializes live test package.
tests/test_compressor_config_exposure.py	Verifies new Rust knob exposure + lockstep defaults across Python/PyO3/Rust.
tests/test_compression_store.py	Updates default TTL assertions to 1800s.
tests/test_ccr.py	Updates CCRConfig default TTL assertion to 1800s.
tests/test_ccr_sqlite_backend.py	New tests for SQLite CCR backend behavior, safety, and defaults.
tests/test_ccr_row_drop_store_bridge.py	Adjusts fixture to reliably exercise the lossy row-drop CCR bridge.
tests/test_audit_reads.py	Tests for Claude transcript read-opportunity audit + CLI + maturation simulation.
tests/test_audit_codex.py	Tests for Codex transcript audit and command classifier + CLI.
tests/test_adapter_hooks.py	Updates default CCR backend expectation to SQLite.
headroom/transforms/smart_crusher.py	Lowers `lossless_min_savings_ratio` default and plumbs new compaction heuristic knobs to Rust.
headroom/transforms/search_compressor.py	Adds `group_by_file` knob and forwards it to Rust config.
headroom/transforms/read_maturation.py	Implements Mechanism B manager + breakpoint relocation helper.
headroom/transforms/read_lifecycle.py	Improves marker text; simplifies counts init; documents removed repeat-Read dedup prototype.
headroom/transforms/content_router.py	Adds SmartCrusher config override and search grouping plumbing.
headroom/proxy/server.py	Enables grouped search output in token mode; adds read maturation env/config wiring; refines CCR retrieve behavior.
headroom/proxy/models.py	Adds `read_maturation` flag to ProxyConfig (default off).
headroom/proxy/handlers/anthropic.py	Wires Mechanism B into Anthropic handler behind `read_maturation` flag; relocates breakpoints on hold.
headroom/config.py	Adds `ReadMaturationConfig`; updates CCR TTL default; exposes new SmartCrusher compaction defaults.
headroom/cli/proxy.py	Adds `--read-maturation` CLI flag.
headroom/cli/main.py	Registers new audit command group.
headroom/cli/audit.py	Implements `headroom audit-reads` CLI (Claude + Codex modes, JSON/text, simulation).
headroom/cli/init.py	Imports the new audit module for command registration.
headroom/ccr/mcp_server.py	Makes CCR miss hints more actionable (re-read file / re-run command).
headroom/cache/prefix_tracker.py	Adds session-scoped `read_maturation_manager` slot for handler wiring.
headroom/cache/compression_store.py	Sets TTL default to 1800s; adds `CCR_MISS_MESSAGE`; defaults global backend creation to SQLite (with opt-out).
headroom/cache/backends/sqlite.py	New SQLite backend with WAL, busy timeout, hygiene sweep, and corruption handling.
headroom/cache/backends/init.py	Exports SQLite backend and updates backend module documentation.
headroom/audit/reads.py	Offline Claude transcript audit for Read opportunity sizing and residency/gap metrics.
headroom/audit/maturation.py	Offline maturation simulation over transcripts to size Mechanism B risk/coverage.
headroom/audit/codex.py	Offline Codex transcript audit + shell command read/search classification.
headroom/audit/init.py	Exposes audit APIs/reports for importers.
crates/headroom-py/src/lib.rs	Exposes new Rust config fields via PyO3; updates defaults.
crates/headroom-parity/src/lib.rs	Adjusts parity comparator config defaults handling for new SmartCrusher fields.
crates/headroom-core/src/transforms/smart_crusher/crusher.rs	Plumbs compaction heuristics; adds constant-field factoring behavior + tests.
crates/headroom-core/src/transforms/smart_crusher/config.rs	Adds new SmartCrusher config fields and updates defaults (0.15 savings ratio).
crates/headroom-core/src/transforms/smart_crusher/compaction/mod.rs	Adds `csv_schema(config)` constructor to honor explicit compaction configs.
crates/headroom-core/src/transforms/search_compressor.rs	Adds grouped-by-file output mode and propagates config.
crates/headroom-core/src/ccr/mod.rs	Updates Rust CCR default TTL to 1800s.

Mechanism B hardcoded ReadMaturationConfig(enabled=True), so only the on/off flag was plumbed — the pilot playbook's "pick quiesce_turns" was impossible. Expose quiesce_turns / max_hold_turns / min_size_bytes through ProxyConfig, HEADROOM_READ_MATURATION_* env vars, and --read-maturation-* CLI flags; the Anthropic handler now builds ReadMaturationConfig from them. Add a multi-turn handler-integration test (mocked upstream, in-memory CCR store for determinism) covering the no-bust invariant — verbatim Read never cache-written before maturation, held->matured ordering — and that quiesce_turns is honored end-to-end.

Test failure (test (4)): test_run_server_uses_import_string_for_multiple_workers cleaned up HEADROOM_PROXY_CONFIG_JSON via monkeypatch.delenv, but run_server sets it through raw os.environ — delenv recorded the JSON value and re-restored it on teardown, leaking it into later tests. _proxy_config_from_env then short-circuited on the leaked JSON and ignored HEADROOM_* vars, failing test_read_maturation_knobs_from_env under shard ordering. - Leaker now pops the var in a finally (no monkeypatch re-restore). - read_maturation test clears the var so it exercises the env-parse path. Copilot review: - server.py: drop redundant store.exists()/get_entry_status re-check in the search branch (get_entry_status(clean_expired=True) already gated it) — saves a backend read + lock per search-retrieve. - backends/__init__.py: docstring/example corrected — CompressionStore() defaults to in-memory; SQLite is the get_compression_store()/env default. - sqlite.py: _entry_from_json docstring corrected — missing required fields raise rather than defaulting. - compression_store.py: CCR_MISS_MESSAGE no longer hardcodes 30 min; notes it is the default and configurable via HEADROOM_CCR_TTL_SECONDS.

…action

Read maturation now lives in its own PR (#1068). This PR keeps the other compression knobs + the SQLite CCR store + audit-reads (traffic audits), which is the simulation that sizes the maturation mechanism. Removed from here: - config.ReadMaturationConfig - ProxyConfig read_maturation* fields - PrefixCacheTracker.read_maturation_manager - anthropic.py handler maturation block - CLI --read-maturation* flags/params/wiring - _proxy_config_from_env env wiring (both server paths) - transforms/read_maturation.py + its 3 test files audit-reads (headroom/audit/maturation.py, codex.py) stays — it is the offline simulation, not the live transform.

chopratejas · 2026-06-17T03:09:48Z

Read maturation (Mechanism B) has been split out of this PR into #1068. This PR now keeps the Rust knob exposure, CCR hardening, and traffic audits (audit-reads) — the latter is the offline simulation that sizes the maturation mechanism, not the live transform.

🤖 I have created a release *beep* *boop* --- <details><summary>0.27.0</summary> ## [0.27.0](v0.26.0...v0.27.0) (2026-06-22) ### Features * **cli:** add headroom doctor setup diagnostics ([#926](#926)) ([e45cf4e](e45cf4e)) * **cli:** add headroom update command and release banner ([#1088](#1088)) ([26be2c3](26be2c3)) * compression extraction — Rust knob exposure, CCR hardening, traffic audits ([#818](#818)) ([b7be381](b7be381)) * measure and surface token throughput (tokens/sec) through the proxy ([#983](#983)) ([0d89c67](0d89c67)) * output-token reduction — verbosity shaper, per-user learning, counterfactual savings ([#965](#965)) ([a99dc61](a99dc61)) * **policy:** decay P_alive from idle time near cache TTL ([#856](#856) P3b) ([#1028](#1028)) ([fe4f9ee](fe4f9ee)) * **providers:** add Cortex Code (Snowflake CoCo) as a supported agent ([#1190](#1190)) ([d9d0bf4](d9d0bf4)) * **proxy:** cc-switch reconciler — keep Headroom in the request path alongside cc-switch ([#1030](#1030)) ([e8fc8a0](e8fc8a0)) * **proxy:** hot-reload live env knobs so a reused proxy picks them up without a restart ([#1090](#1090)) ([6904d47](6904d47)) * **proxy:** make COMPRESSION_TIMEOUT_SECONDS configurable via env ([#946](#946)) ([#991](#991)) ([addebdb](addebdb)) * **transforms:** tabular + spreadsheet (.xlsx/.xls) compression ([#1128](#1128)) ([d789a7c](d789a7c)) * **vertex:** turnkey Claude Code + Vertex compression (+ fixes from the Vertex review) ([#1113](#1113)) ([0e05915](0e05915)) ### Bug Fixes * **ccr:** accept 12-char SmartCrusher hashes in tool injection ([#1095](#1095)) ([#1141](#1141)) ([9f7f3ad](9f7f3ad)) * **ccr:** return stored content when headroom_retrieve query matches nothing ([#1213](#1213)) ([#1236](#1236)) ([08fb845](08fb845)) * **content-router:** honor target_ratio in compression cache + add proxy --target-ratio flag ([#1108](#1108)) ([8894ee0](8894ee0)) * **dashboard:** light-mode backgrounds + aligned savings tables ([#1064](#1064)) ([5eae32b](5eae32b)) * **deps:** make litellm optional on Python 3.14 ([#956](#956)) ([#993](#993)) ([b2f04e4](b2f04e4)) * **e2e:** align Codex wrap e2e with global-only RTK guidance ([#1240](#1240)) ([#1254](#1254)) ([bc12ace](bc12ace)) * **init:** set ENABLE_TOOL_SEARCH=true so Claude Code keeps deferring tools ([#746](#746)) ([#995](#995)) ([500ec2b](500ec2b)) * **kompress:** never block the request path on the cold-cache model download ([#1161](#1161)) ([3fc2a78](3fc2a78)) * **memory:** use ONNX embedder for `wrap --memory` sync ([#1092](#1092)) ([#1262](#1262)) ([4f9feda](4f9feda)) * **openclaw:** wrap plugin export as {register} object for OpenClaw 2026.x compatibility ([#1218](#1218)) ([2e6c442](2e6c442)) * **providers:** update DeepSeek V3 context limit from 128K to 1M ([#1038](#1038)) ([#1137](#1137)) ([bcabc5c](bcabc5c)) * **proxy:** allow disabling periodic TOIN stats logging ([#1265](#1265)) ([b5f63d8](b5f63d8)) * **proxy:** honor HEADROOM_EXCLUDE_TOOLS for Codex /v1/responses tool outputs ([#940](#940)) ([#1053](#1053)) ([f03e77b](f03e77b)) * **proxy:** preserve byte-faithful Anthropic tool forwarding ([#1222](#1222)) ([1f18d59](1f18d59)) * **proxy:** route Codex OAuth image requests ([#1215](#1215)) ([381d771](381d771)) * **proxy:** scope CORS to loopback + gate operator/content endpoints ([#1226](#1226)) ([bd55a42](bd55a42)) * **proxy:** stamp X-Client: codex on Responses endpoint for unidentified callers ([#1036](#1036)) ([b0cd032](b0cd032)) * **proxy:** treat NODE_EXTRA_CA_CERTS as additive, not replacement ([#998](#998)) ([#1031](#1031)) ([c987283](c987283)) * **telemetry:** switch anonymous telemetry to opt-in (off by default) ([#1223](#1223)) ([b998697](b998697)) * **tokenizers:** bound tiktoken vocab load so a stalled download cannot hang requests ([#956](#956)) ([#994](#994)) ([7e86baf](7e86baf)) * **unwrap:** remove ANTHROPIC_BASE_URL + ENABLE_TOOL_SEARCH and init hooks on unwrap ([#992](#992)) ([5b84691](5b84691)) * **wrap:** keep Codex RTK guidance global ([#1240](#1240)) ([7c26a54](7c26a54)) * **wrap:** percent-encode non-ASCII cwd names in X-Headroom-Project header ([#1071](#1071)) ([9f712cc](9f712cc)) * **wrap:** write env.ANTHROPIC_BASE_URL to settings.json so daemon-spawned conversations inherit proxy ([#951](#951)) ([#1078](#1078)) ([a554c3a](a554c3a)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

…anism B) (#1068) ## What Splits **read maturation (Mechanism B)** out of #818 into its own PR, so #818 can stay focused on the other compression knobs + the SQLite CCR store. Audit-reads (traffic audits) stays in #818. Read maturation holds fresh large `Read` outputs **out of the provider prefix cache** while their file is still active, keeps them verbatim the whole time the model is working with them, and **matures** them into a CCR-backed marker once the file has been quiet for `quiesce_turns`. Only the final compressed form ever enters the cache, so **no cached byte is ever mutated — there is nothing to bust.** Activity-based rather than a fixed hold window: the `audit-reads` simulation showed next-touch gaps are fat-tailed (p50 = 4 turns, p90 = 81), so no fixed window covers the tail while a quiesce rule covers the activity cluster and lets the tail self-heal via partial-range re-reads. **Default OFF** — experimental, flag/env gated, validated in pilots first. ## Changes - `config.ReadMaturationConfig` (`enabled=False`, `quiesce_turns=5`, `max_hold_turns=25`, `min_size_bytes=2048`) - `ProxyConfig` fields mirroring the above - `ReadMaturationManager` transform + `relocate_cache_breakpoint` (`headroom/transforms/read_maturation.py`) - Session-scoped manager rides on `PrefixCacheTracker` — shares the session's cache affinity and TTL cleanup - Handler wiring in `anthropic.py`: runs **after** compression (so `read_lifecycle` markers are respected) and **before** body assembly; advisory — never fails the request - CLI flags + env vars (`--read-maturation*` / `HEADROOM_READ_MATURATION*`) - `_proxy_config_from_env` wiring for both the multi-worker and CLI server paths ## Bug fix included The `--read-maturation` flag was missing `envvar="HEADROOM_READ_MATURATION"`, so the env var was silently ignored on the CLI path (only the multi-worker `_proxy_config_from_env` path read it). Fixed here. ## Tests - `tests/test_read_maturation.py` — unit - `tests/test_read_maturation_handler_nobust.py` — handler never busts cache - `tests/test_live/test_live_maturation.py` — live harness `25 passed` locally; ruff check/format clean; mypy clean (only pre-existing `annotation-unchecked` notes).

…ffic audits (headroomlabs-ai#818) ## Description A data-driven push for better compression savings without accuracy loss, in four parts: expose and tune the Rust compressor knobs, harden the CCR retrieval store, add traffic-audit tooling that sizes opportunities from real transcripts, and introduce **read maturation** — a new, live-validated mechanism that compresses Read outputs *before* they ever enter the provider prefix cache. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made ### 1. Rust compressor extraction - Expose `lossless_min_savings_ratio` end-to-end and lower the default 0.30 → 0.15 (lockstep across Rust, PyO3, and both Python config classes) so the lossless Table/CSV compaction path wins more often. - Expose the `CompactConfig` heuristics (core-field fraction, heterogeneity ratio, flatten cap, bucket bounds) through PyO3 + Python. - `SearchCompressor` grouped-by-file output (`rg --heading` style — path once per file instead of per match). Library default off; the proxy enables it in token mode. - Complete `factor_out_constants`: constant fields now emit once in a `_constant_fields` sentinel with slim rows (defensive per-item value match; default off). - `ContentRouter` accepts a SmartCrusher config override and the search-grouping knob. ### 2. CCR store hardening - Session-scale TTL: 300s → 1800s (CCRConfig, CompressionEntry, CompressionStore, Rust `DEFAULT_TTL` — lockstep). - **SQLite is the default CCR backend** (`~/.headroom/ccr_store.db`, WAL): survives proxy restarts and is shared across workers. `HEADROOM_CCR_BACKEND=memory` opts out. - Multi-worker safety: `busy_timeout`, and corruption detection narrowed so transient `SQLITE_BUSY` errors can never trigger database deletion. - Data-at-rest hygiene: `chmod 600` on db + sidecars, expired rows swept at open. - Retrieval-miss messages are actionable (re-read the file / re-run the command). ### 3. Traffic audit tooling (measure before tuning) - `headroom audit-reads`: sizes Read opportunities from local Claude Code transcripts (read share, stale %, line-number overhead, context residency, cache-death windows). - `--simulate-maturation`: Mechanism B risk sizing (re-read rates, never-touched-again share, quiesce coverage, at-risk edits). - `--codex`: shell-read classifier for Codex transcripts (rtk-wrapper aware, workdir resolution). - Findings that shaped this PR (81 sessions): Reads are 67% of tool bytes; median Read lingers 118 turns (~13x lifetime cost); a prototyped repeat-Read dedup measured 0.1% and was **removed** rather than shipped as dead code. ### 4. Read maturation (Mechanism B) — experimental, default OFF - Activity-based: a fresh large Read is held **out** of the provider cache (trailing breakpoint relocated before it), stays verbatim while its file is active, and matures into a CCR-backed marker once the file is quiet for `quiesce_turns` (default 5; `max_hold_turns` bounds busy files). - Only the final compressed form ever enters the cache — **no cached byte is ever mutated**; matured markers replay byte-identically. - Wired into the Anthropic handler behind `--read-maturation` / `HEADROOM_READ_MATURATION=1`; session state rides on the prefix tracker; advisory (can never fail a request). - Live-validated against the Anthropic API: held content excluded from cache_creation; after maturation the prior cached prefix still served — the no-bust invariant holds end-to-end. ### 5. Rebase / CI fixups (this update) - Rebased onto latest `main` (was 28 commits behind): picks up `ci: pass CODECOV_TOKEN to coverage uploads (headroomlabs-ai#968)`, which is what was turning the 4 test shards red — the tests themselves passed (1528) but the post-test codecov upload exited non-zero on a protected branch. - Resolved the duplicate `lossless_min_savings_ratio` that two independent main/branch additions left in `SmartCrusherConfig` and the Rust-config kwarg (import-time `SyntaxError` + mypy `no-redef`). - Aligned CCR tests with the new defaults (SQLite backend, 1800s TTL) across `test_ccr`, `test_adapter_hooks`, `test_compression_store`, `test_proxy_ccr`, and the lossy row-drop bridge test. ## Testing  - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_proxy_ccr.py tests/test_ccr.py tests/test_compression_store.py tests/test_adapter_hooks.py tests/test_ccr_row_drop_store_bridge.py -q 170 passed, 4 warnings in 42.49s $ python -m pytest tests/test_audit_reads.py tests/test_audit_codex.py tests/test_read_maturation.py tests/test_transforms_content_router.py tests/test_smart_crusher_toin_attachment.py -q 83 passed $ mypy headroom/ Success: no issues found in 365 source files $ python -m compileall headroom/ -q COMPILE-OK # CI (run 27488990477, pre-rebase head): all 4 shards ran to completion — # "1528 passed, 120 skipped, 4922 deselected" # The red shards were the codecov upload step, not test failures; fixed by # the headroomlabs-ai#968 rebase above. ``` ## Real Behavior Proof - Environment: macOS (darwin), Python 3.12 venv; branch `feat/compression-extraction` rebased onto `origin/main` (head 7cb0f43); GitHub Actions CI run 27488990477 for the test shards - Exact command / steps: rebased onto latest main (clean, 13 commits replayed, 0 conflicts); ran the pytest suites and mypy above locally; inspected CI shard logs to confirm the failure was the codecov upload, not the test phase - Observed result: 253 targeted tests pass locally; mypy clean on 365 files; CI test phase reports `1528 passed, 120 skipped`; the only red step (codecov `upload-coverage` → "Token required because branch is protected") is resolved by the rebased-in headroomlabs-ai#968 CODECOV_TOKEN fix - Not tested: the read-maturation live-API no-bust validation (`tests/test_live/`) was not re-run in this rebase pass (requires provider keys); it was validated when the feature first landed, and no maturation code changed in the rebase — only CCR-default test assertions and the duplicate-field resolution ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes CHANGELOG is generated by release-please from the conventional commits, so the CHANGELOG box is intentionally left unchecked. "Manual testing performed" is unchecked deliberately — see `Real Behavior Proof` → `Not tested` for the exact boundary (the live-API maturation validation was not re-run in this rebase pass). ### Follow-ups (tracked, not in this PR) - Mechanism B provider extensions: OpenAI-family wiring (no breakpoint hold — bounded near-tail bust) and the Codex runtime read-detector (the audit classifier is the prototype). - Pilot enablement playbook: run `audit-reads --simulate-maturation` on target traffic → pick `quiesce_turns` → enable via env → watch cache hit rate + `read_maturation:N` transform tags.

🤖 I have created a release *beep* *boop* --- <details><summary>0.27.0</summary> ## [0.27.0](headroomlabs-ai/headroom@v0.26.0...v0.27.0) (2026-06-22) ### Features * **cli:** add headroom doctor setup diagnostics ([headroomlabs-ai#926](headroomlabs-ai#926)) ([e45cf4e](headroomlabs-ai@e45cf4e)) * **cli:** add headroom update command and release banner ([headroomlabs-ai#1088](headroomlabs-ai#1088)) ([26be2c3](headroomlabs-ai@26be2c3)) * compression extraction — Rust knob exposure, CCR hardening, traffic audits ([headroomlabs-ai#818](headroomlabs-ai#818)) ([b7be381](headroomlabs-ai@b7be381)) * measure and surface token throughput (tokens/sec) through the proxy ([headroomlabs-ai#983](headroomlabs-ai#983)) ([0d89c67](headroomlabs-ai@0d89c67)) * output-token reduction — verbosity shaper, per-user learning, counterfactual savings ([headroomlabs-ai#965](headroomlabs-ai#965)) ([a99dc61](headroomlabs-ai@a99dc61)) * **policy:** decay P_alive from idle time near cache TTL ([headroomlabs-ai#856](headroomlabs-ai#856) P3b) ([headroomlabs-ai#1028](headroomlabs-ai#1028)) ([fe4f9ee](headroomlabs-ai@fe4f9ee)) * **providers:** add Cortex Code (Snowflake CoCo) as a supported agent ([headroomlabs-ai#1190](headroomlabs-ai#1190)) ([d9d0bf4](headroomlabs-ai@d9d0bf4)) * **proxy:** cc-switch reconciler — keep Headroom in the request path alongside cc-switch ([headroomlabs-ai#1030](headroomlabs-ai#1030)) ([e8fc8a0](headroomlabs-ai@e8fc8a0)) * **proxy:** hot-reload live env knobs so a reused proxy picks them up without a restart ([headroomlabs-ai#1090](headroomlabs-ai#1090)) ([6904d47](headroomlabs-ai@6904d47)) * **proxy:** make COMPRESSION_TIMEOUT_SECONDS configurable via env ([headroomlabs-ai#946](headroomlabs-ai#946)) ([headroomlabs-ai#991](headroomlabs-ai#991)) ([addebdb](headroomlabs-ai@addebdb)) * **transforms:** tabular + spreadsheet (.xlsx/.xls) compression ([headroomlabs-ai#1128](headroomlabs-ai#1128)) ([d789a7c](headroomlabs-ai@d789a7c)) * **vertex:** turnkey Claude Code + Vertex compression (+ fixes from the Vertex review) ([headroomlabs-ai#1113](headroomlabs-ai#1113)) ([0e05915](headroomlabs-ai@0e05915)) ### Bug Fixes * **ccr:** accept 12-char SmartCrusher hashes in tool injection ([headroomlabs-ai#1095](headroomlabs-ai#1095)) ([headroomlabs-ai#1141](headroomlabs-ai#1141)) ([9f7f3ad](headroomlabs-ai@9f7f3ad)) * **ccr:** return stored content when headroom_retrieve query matches nothing ([headroomlabs-ai#1213](headroomlabs-ai#1213)) ([headroomlabs-ai#1236](headroomlabs-ai#1236)) ([08fb845](headroomlabs-ai@08fb845)) * **content-router:** honor target_ratio in compression cache + add proxy --target-ratio flag ([headroomlabs-ai#1108](headroomlabs-ai#1108)) ([8894ee0](headroomlabs-ai@8894ee0)) * **dashboard:** light-mode backgrounds + aligned savings tables ([headroomlabs-ai#1064](headroomlabs-ai#1064)) ([5eae32b](headroomlabs-ai@5eae32b)) * **deps:** make litellm optional on Python 3.14 ([headroomlabs-ai#956](headroomlabs-ai#956)) ([headroomlabs-ai#993](headroomlabs-ai#993)) ([b2f04e4](headroomlabs-ai@b2f04e4)) * **e2e:** align Codex wrap e2e with global-only RTK guidance ([headroomlabs-ai#1240](headroomlabs-ai#1240)) ([headroomlabs-ai#1254](headroomlabs-ai#1254)) ([bc12ace](headroomlabs-ai@bc12ace)) * **init:** set ENABLE_TOOL_SEARCH=true so Claude Code keeps deferring tools ([headroomlabs-ai#746](headroomlabs-ai#746)) ([headroomlabs-ai#995](headroomlabs-ai#995)) ([500ec2b](headroomlabs-ai@500ec2b)) * **kompress:** never block the request path on the cold-cache model download ([headroomlabs-ai#1161](headroomlabs-ai#1161)) ([3fc2a78](headroomlabs-ai@3fc2a78)) * **memory:** use ONNX embedder for `wrap --memory` sync ([headroomlabs-ai#1092](headroomlabs-ai#1092)) ([headroomlabs-ai#1262](headroomlabs-ai#1262)) ([4f9feda](headroomlabs-ai@4f9feda)) * **openclaw:** wrap plugin export as {register} object for OpenClaw 2026.x compatibility ([headroomlabs-ai#1218](headroomlabs-ai#1218)) ([2e6c442](headroomlabs-ai@2e6c442)) * **providers:** update DeepSeek V3 context limit from 128K to 1M ([headroomlabs-ai#1038](headroomlabs-ai#1038)) ([headroomlabs-ai#1137](headroomlabs-ai#1137)) ([bcabc5c](headroomlabs-ai@bcabc5c)) * **proxy:** allow disabling periodic TOIN stats logging ([headroomlabs-ai#1265](headroomlabs-ai#1265)) ([b5f63d8](headroomlabs-ai@b5f63d8)) * **proxy:** honor HEADROOM_EXCLUDE_TOOLS for Codex /v1/responses tool outputs ([headroomlabs-ai#940](headroomlabs-ai#940)) ([headroomlabs-ai#1053](headroomlabs-ai#1053)) ([f03e77b](headroomlabs-ai@f03e77b)) * **proxy:** preserve byte-faithful Anthropic tool forwarding ([headroomlabs-ai#1222](headroomlabs-ai#1222)) ([1f18d59](headroomlabs-ai@1f18d59)) * **proxy:** route Codex OAuth image requests ([headroomlabs-ai#1215](headroomlabs-ai#1215)) ([381d771](headroomlabs-ai@381d771)) * **proxy:** scope CORS to loopback + gate operator/content endpoints ([headroomlabs-ai#1226](headroomlabs-ai#1226)) ([bd55a42](headroomlabs-ai@bd55a42)) * **proxy:** stamp X-Client: codex on Responses endpoint for unidentified callers ([headroomlabs-ai#1036](headroomlabs-ai#1036)) ([b0cd032](headroomlabs-ai@b0cd032)) * **proxy:** treat NODE_EXTRA_CA_CERTS as additive, not replacement ([headroomlabs-ai#998](headroomlabs-ai#998)) ([headroomlabs-ai#1031](headroomlabs-ai#1031)) ([c987283](headroomlabs-ai@c987283)) * **telemetry:** switch anonymous telemetry to opt-in (off by default) ([headroomlabs-ai#1223](headroomlabs-ai#1223)) ([b998697](headroomlabs-ai@b998697)) * **tokenizers:** bound tiktoken vocab load so a stalled download cannot hang requests ([headroomlabs-ai#956](headroomlabs-ai#956)) ([headroomlabs-ai#994](headroomlabs-ai#994)) ([7e86baf](headroomlabs-ai@7e86baf)) * **unwrap:** remove ANTHROPIC_BASE_URL + ENABLE_TOOL_SEARCH and init hooks on unwrap ([headroomlabs-ai#992](headroomlabs-ai#992)) ([5b84691](headroomlabs-ai@5b84691)) * **wrap:** keep Codex RTK guidance global ([headroomlabs-ai#1240](headroomlabs-ai#1240)) ([7c26a54](headroomlabs-ai@7c26a54)) * **wrap:** percent-encode non-ASCII cwd names in X-Headroom-Project header ([headroomlabs-ai#1071](headroomlabs-ai#1071)) ([9f712cc](headroomlabs-ai@9f712cc)) * **wrap:** write env.ANTHROPIC_BASE_URL to settings.json so daemon-spawned conversations inherit proxy ([headroomlabs-ai#951](headroomlabs-ai#951)) ([headroomlabs-ai#1078](headroomlabs-ai#1078)) ([a554c3a](headroomlabs-ai@a554c3a)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

…anism B) (headroomlabs-ai#1068) ## What Splits **read maturation (Mechanism B)** out of headroomlabs-ai#818 into its own PR, so headroomlabs-ai#818 can stay focused on the other compression knobs + the SQLite CCR store. Audit-reads (traffic audits) stays in headroomlabs-ai#818. Read maturation holds fresh large `Read` outputs **out of the provider prefix cache** while their file is still active, keeps them verbatim the whole time the model is working with them, and **matures** them into a CCR-backed marker once the file has been quiet for `quiesce_turns`. Only the final compressed form ever enters the cache, so **no cached byte is ever mutated — there is nothing to bust.** Activity-based rather than a fixed hold window: the `audit-reads` simulation showed next-touch gaps are fat-tailed (p50 = 4 turns, p90 = 81), so no fixed window covers the tail while a quiesce rule covers the activity cluster and lets the tail self-heal via partial-range re-reads. **Default OFF** — experimental, flag/env gated, validated in pilots first. ## Changes - `config.ReadMaturationConfig` (`enabled=False`, `quiesce_turns=5`, `max_hold_turns=25`, `min_size_bytes=2048`) - `ProxyConfig` fields mirroring the above - `ReadMaturationManager` transform + `relocate_cache_breakpoint` (`headroom/transforms/read_maturation.py`) - Session-scoped manager rides on `PrefixCacheTracker` — shares the session's cache affinity and TTL cleanup - Handler wiring in `anthropic.py`: runs **after** compression (so `read_lifecycle` markers are respected) and **before** body assembly; advisory — never fails the request - CLI flags + env vars (`--read-maturation*` / `HEADROOM_READ_MATURATION*`) - `_proxy_config_from_env` wiring for both the multi-worker and CLI server paths ## Bug fix included The `--read-maturation` flag was missing `envvar="HEADROOM_READ_MATURATION"`, so the env var was silently ignored on the CLI path (only the multi-worker `_proxy_config_from_env` path read it). Fixed here. ## Tests - `tests/test_read_maturation.py` — unit - `tests/test_read_maturation_handler_nobust.py` — handler never busts cache - `tests/test_live/test_live_maturation.py` — live harness `25 passed` locally; ruff check/format clean; mypy clean (only pre-existing `annotation-unchecked` notes).

…ffic audits (headroomlabs-ai#818) ## Description A data-driven push for better compression savings without accuracy loss, in four parts: expose and tune the Rust compressor knobs, harden the CCR retrieval store, add traffic-audit tooling that sizes opportunities from real transcripts, and introduce **read maturation** — a new, live-validated mechanism that compresses Read outputs *before* they ever enter the provider prefix cache. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made ### 1. Rust compressor extraction - Expose `lossless_min_savings_ratio` end-to-end and lower the default 0.30 → 0.15 (lockstep across Rust, PyO3, and both Python config classes) so the lossless Table/CSV compaction path wins more often. - Expose the `CompactConfig` heuristics (core-field fraction, heterogeneity ratio, flatten cap, bucket bounds) through PyO3 + Python. - `SearchCompressor` grouped-by-file output (`rg --heading` style — path once per file instead of per match). Library default off; the proxy enables it in token mode. - Complete `factor_out_constants`: constant fields now emit once in a `_constant_fields` sentinel with slim rows (defensive per-item value match; default off). - `ContentRouter` accepts a SmartCrusher config override and the search-grouping knob. ### 2. CCR store hardening - Session-scale TTL: 300s → 1800s (CCRConfig, CompressionEntry, CompressionStore, Rust `DEFAULT_TTL` — lockstep). - **SQLite is the default CCR backend** (`~/.headroom/ccr_store.db`, WAL): survives proxy restarts and is shared across workers. `HEADROOM_CCR_BACKEND=memory` opts out. - Multi-worker safety: `busy_timeout`, and corruption detection narrowed so transient `SQLITE_BUSY` errors can never trigger database deletion. - Data-at-rest hygiene: `chmod 600` on db + sidecars, expired rows swept at open. - Retrieval-miss messages are actionable (re-read the file / re-run the command). ### 3. Traffic audit tooling (measure before tuning) - `headroom audit-reads`: sizes Read opportunities from local Claude Code transcripts (read share, stale %, line-number overhead, context residency, cache-death windows). - `--simulate-maturation`: Mechanism B risk sizing (re-read rates, never-touched-again share, quiesce coverage, at-risk edits). - `--codex`: shell-read classifier for Codex transcripts (rtk-wrapper aware, workdir resolution). - Findings that shaped this PR (81 sessions): Reads are 67% of tool bytes; median Read lingers 118 turns (~13x lifetime cost); a prototyped repeat-Read dedup measured 0.1% and was **removed** rather than shipped as dead code. ### 4. Read maturation (Mechanism B) — experimental, default OFF - Activity-based: a fresh large Read is held **out** of the provider cache (trailing breakpoint relocated before it), stays verbatim while its file is active, and matures into a CCR-backed marker once the file is quiet for `quiesce_turns` (default 5; `max_hold_turns` bounds busy files). - Only the final compressed form ever enters the cache — **no cached byte is ever mutated**; matured markers replay byte-identically. - Wired into the Anthropic handler behind `--read-maturation` / `HEADROOM_READ_MATURATION=1`; session state rides on the prefix tracker; advisory (can never fail a request). - Live-validated against the Anthropic API: held content excluded from cache_creation; after maturation the prior cached prefix still served — the no-bust invariant holds end-to-end. ### 5. Rebase / CI fixups (this update) - Rebased onto latest `main` (was 28 commits behind): picks up `ci: pass CODECOV_TOKEN to coverage uploads (headroomlabs-ai#968)`, which is what was turning the 4 test shards red — the tests themselves passed (1528) but the post-test codecov upload exited non-zero on a protected branch. - Resolved the duplicate `lossless_min_savings_ratio` that two independent main/branch additions left in `SmartCrusherConfig` and the Rust-config kwarg (import-time `SyntaxError` + mypy `no-redef`). - Aligned CCR tests with the new defaults (SQLite backend, 1800s TTL) across `test_ccr`, `test_adapter_hooks`, `test_compression_store`, `test_proxy_ccr`, and the lossy row-drop bridge test. ## Testing  - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_proxy_ccr.py tests/test_ccr.py tests/test_compression_store.py tests/test_adapter_hooks.py tests/test_ccr_row_drop_store_bridge.py -q 170 passed, 4 warnings in 42.49s $ python -m pytest tests/test_audit_reads.py tests/test_audit_codex.py tests/test_read_maturation.py tests/test_transforms_content_router.py tests/test_smart_crusher_toin_attachment.py -q 83 passed $ mypy headroom/ Success: no issues found in 365 source files $ python -m compileall headroom/ -q COMPILE-OK # CI (run 27488990477, pre-rebase head): all 4 shards ran to completion — # "1528 passed, 120 skipped, 4922 deselected" # The red shards were the codecov upload step, not test failures; fixed by # the headroomlabs-ai#968 rebase above. ``` ## Real Behavior Proof - Environment: macOS (darwin), Python 3.12 venv; branch `feat/compression-extraction` rebased onto `origin/main` (head 7cb0f43); GitHub Actions CI run 27488990477 for the test shards - Exact command / steps: rebased onto latest main (clean, 13 commits replayed, 0 conflicts); ran the pytest suites and mypy above locally; inspected CI shard logs to confirm the failure was the codecov upload, not the test phase - Observed result: 253 targeted tests pass locally; mypy clean on 365 files; CI test phase reports `1528 passed, 120 skipped`; the only red step (codecov `upload-coverage` → "Token required because branch is protected") is resolved by the rebased-in headroomlabs-ai#968 CODECOV_TOKEN fix - Not tested: the read-maturation live-API no-bust validation (`tests/test_live/`) was not re-run in this rebase pass (requires provider keys); it was validated when the feature first landed, and no maturation code changed in the rebase — only CCR-default test assertions and the duplicate-field resolution ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes CHANGELOG is generated by release-please from the conventional commits, so the CHANGELOG box is intentionally left unchecked. "Manual testing performed" is unchecked deliberately — see `Real Behavior Proof` → `Not tested` for the exact boundary (the live-API maturation validation was not re-run in this rebase pass). ### Follow-ups (tracked, not in this PR) - Mechanism B provider extensions: OpenAI-family wiring (no breakpoint hold — bounded near-tail bust) and the Codex runtime read-detector (the audit classifier is the prototype). - Pilot enablement playbook: run `audit-reads --simulate-maturation` on target traffic → pick `quiesce_turns` → enable via env → watch cache hit rate + `read_maturation:N` transform tags.

github-actions Bot added status: has conflicts Pull request has merge conflicts with the base branch status: ci failing Required or reported CI checks are failing labels Jun 11, 2026

chopratejas force-pushed the feat/compression-extraction branch from 772b36b to 573137a Compare June 12, 2026 06:42

github-actions Bot added status: ci failing Required or reported CI checks are failing and removed status: has conflicts Pull request has merge conflicts with the base branch status: ci failing Required or reported CI checks are failing labels Jun 12, 2026

JerrettDavis requested changes Jun 12, 2026

View reviewed changes

github-actions Bot removed the status: ci failing Required or reported CI checks are failing label Jun 14, 2026

chopratejas added 13 commits June 14, 2026 23:20

style: ruff format fixups in proxy handler and server

39dfe1a

chopratejas force-pushed the feat/compression-extraction branch from fbfd7a8 to 7cb0f43 Compare June 15, 2026 06:33

JerrettDavis requested a review from Copilot June 15, 2026 19:35

Copilot started reviewing on behalf of JerrettDavis June 15, 2026 19:36 View session

Copilot AI reviewed Jun 15, 2026

View reviewed changes

Comment thread headroom/proxy/server.py Outdated

Comment thread headroom/cache/backends/__init__.py Outdated

Comment thread headroom/cache/backends/__init__.py Outdated

Comment thread headroom/cache/backends/sqlite.py Outdated

Comment thread headroom/cache/compression_store.py

github-actions Bot added status: ready for review Pull request body is complete and the author marked it ready for human review and removed status: ready for review Pull request body is complete and the author marked it ready for human review labels Jun 16, 2026

chopratejas added 2 commits June 16, 2026 13:51

Merge remote-tracking branch 'origin/main' into feat/compression-extr…

a3d4932

…action

github-actions Bot added status: ready for review Pull request body is complete and the author marked it ready for human review and removed status: ci failing Required or reported CI checks are failing labels Jun 16, 2026

JerrettDavis approved these changes Jun 16, 2026

View reviewed changes

chopratejas mentioned this pull request Jun 17, 2026

feat(read-maturation): activity-based hold-back Read maturation (Mechanism B) #1068

Merged

chopratejas changed the title ~~feat: compression extraction — Rust knob exposure, CCR hardening, read maturation (Mechanism B), traffic audits~~ feat: compression extraction — Rust knob exposure, CCR hardening, traffic audits Jun 17, 2026

chopratejas merged commit b7be381 into main Jun 17, 2026
55 checks passed

zj0001 mentioned this pull request Jun 17, 2026

[BUG] lossless:table compaction emits CCR markers that never reach the store — /v1/retrieve 404s #1083

Closed

github-actions Bot mentioned this pull request Jun 22, 2026

chore: release main #1274

Merged

Navigation Menu

Uh oh!

feat: compression extraction — Rust knob exposure, CCR hardening, traffic audits - #818

feat: compression extraction — Rust knob exposure, CCR hardening, traffic audits#818
chopratejas merged 17 commits into
mainfrom
feat/compression-extraction

chopratejas commented Jun 10, 2026 •

edited

Loading

Uh oh!

JerrettDavis commented Jun 10, 2026

Uh oh!

JerrettDavis left a comment

Uh oh!

github-actions Bot commented Jun 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chopratejas commented Jun 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

chopratejas commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of Change

Changes Made

1. Rust compressor extraction

2. CCR store hardening

3. Traffic audit tooling (measure before tuning)

4. Read maturation (Mechanism B) — experimental, default OFF

5. Rebase / CI fixups (this update)

Testing

Test Output

Real Behavior Proof

Review Readiness

Checklist

Additional Notes

Follow-ups (tracked, not in this PR)

Uh oh!

JerrettDavis commented Jun 10, 2026

Uh oh!

JerrettDavis left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 15, 2026

PR governance

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chopratejas commented Jun 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chopratejas commented Jun 10, 2026 •

edited

Loading