feat: output-token reduction — verbosity shaper, per-user learning, counterfactual savings

[chopratejas]

Description

Adds the first levers that reduce the tokens the model writes back (output), complementing Headroom's existing input compression. Output costs 5× input on Opus-class models and is full of waste (ceremony, restated code, deep "thinking" on routine steps). Two phases in one self-contained PR off main: the request-side output shaper, then per-user verbosity learning plus an honest counterfactual savings estimator and dashboard surfacing.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Performance improvement
• Code refactoring (no functional changes)

Changes Made

• Output shaper (output_shaper.py, opt-in HEADROOM_OUTPUT_SHAPER=1): cache-safe verbosity steering appended to the system-prompt tail (5 levels); effort routing that lowers output_config.effort on mechanical tool-result continuations; legacy thinking.budget_tokens clamp. Never injects effort where absent, never toggles thinking.type.
• headroom learn --verbosity: mines Claude Code transcripts for behavioral signals (interrupts, length-adaptive fast-skips, echo ratio), recommends a verbosity level (heuristic + optional --llm-judge), and seeds the savings baseline.
• Counterfactual estimator (output_savings.py): per-stratum synthetic-control (estimated) + A/B holdout (measured) with a propagated 95% CI; conversation-stable arm assignment for A/B validity and prefix-cache safety.
• AIMD verbosity controller (verbosity_controller.py): additive-increase / fast-back-off state machine; live signal emission gated off by default.
• Wiring + surfaces: shaper resolves the learned level; recording rides the existing transforms_applied channel through the outcome funnel (no RequestOutcome changes); headroom output-savings CLI; dashboard "Output Tokens Saved" card.
• Docs: simple-words user guide + design doc with the counterfactual methodology.

Testing

• Unit tests pass (pytest)
• Linting passes (ruff check .)
• Type checking passes (mypy headroom)
• New tests added for new functionality
• Manual testing performed

Test Output

$ pytest tests/test_output_savings.py tests/test_output_savings_cli.py \
        tests/test_verbosity_learn.py tests/test_verbosity_controller.py \
        tests/test_output_shaper.py -q
94 passed in 0.54s

$ pytest tests/test_request_outcome.py tests/test_handler_outcome_tag_invariant.py \
        tests/test_proxy_dashboard_stats_cache.py -q
44 passed

$ ruff format --check .
831 files already formatted

$ mypy headroom --ignore-missing-imports
Success: no issues found in 361 source files

Real Behavior Proof

• Environment: macOS, Python 3.12 (.venv), anthropic 0.76, live API model claude-opus-4-8.
• Exact command / steps: HEADROOM_OUTPUT_SHAPER=1; headroom learn --verbosity --apply (seeds level + baseline); python scripts/eval_output_shaper.py A (live before/after); simulate holdout traffic then headroom output-savings.
• Observed result: code-review ask — baseline 1,750 output tokens → L2 1,354 (−22.7%) → L3 599 (−65.8%), same bugs found. learn --verbosity on 24 real sessions → 11% interrupt / 26% fast-skip → L3 (high confidence). Measured A/B path → 31.7% reduction (95% CI 27.7%–35.7%). 94 new tests + 44 existing outcome/dashboard tests green; ruff + mypy clean.
• Not tested: live streaming-path recording exercised only via unit tests (the transforms_applied funnel is shared across paths); runtime AIMD signal emission is gated off by default and not exercised live.

Review Readiness

• I have performed a self-review
• This PR is ready for human review

Checklist

• My code follows the project's style guidelines
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have updated the CHANGELOG.md if applicable

Additional Notes

Output savings are counterfactual (we never observe what the model would have written), so the estimator separates estimated (vs a learned baseline) from measured (A/B holdout via HEADROOM_OUTPUT_HOLDOUT) and always reports a confidence band — never a single made-up number. CHANGELOG left unchecked (release-please manages it). Runtime AIMD self-tuning is intentionally a TODO (controller built/tested; live signal emission gated behind HEADROOM_VERBOSITY_AUTOTUNE).

[github-actions]

PR governance

This PR follows the template and is marked ready for human review.

[Copilot]

Pull request overview

Adds output-side token reduction to Headroom’s proxy (verbosity steering + effort routing), plus a per-user verbosity learning flow and a counterfactual/measured savings estimator surfaced via CLI and dashboard.

Changes:

• Introduces an opt-in request rewriter (output_shaper) that appends cache-safe verbosity steering to the system-prompt tail and routes effort down on mechanical tool-result continuations.
• Adds verbosity learning (headroom learn --verbosity) that mines Claude Code transcripts for behavioral signals and seeds a per-stratum output-token baseline.
• Adds an output-savings ledger/estimator with holdout arm assignment, CLI reporting, and dashboard stats surfacing.

Reviewed changes

Copilot reviewed 20 out of 21 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/test_verbosity_learn.py	Unit tests for transcript signal extraction + heuristic recommendation.
tests/test_verbosity_controller.py	Unit tests for AIMD verbosity controller state machine + persistence.
tests/test_output_shaper.py	Unit tests for turn classification, steering idempotency, and effort routing.
tests/test_output_savings.py	Unit tests for stratification, arm assignment, baseline/holdout estimation, and echo ratio.
tests/test_output_savings_cli.py	Smoke tests for CLI output + outcome→ledger wiring via labels.
scripts/eval_output_shaper.py	Live eval script comparing baseline vs shaped request output tokens.
README.md	Documents the new “Output token reduction” feature and basic usage.
headroom/proxy/verbosity_controller.py	Adds AIMD controller + state load/save helpers.
headroom/proxy/server.py	Surfaces output reduction stats in /stats payload for dashboard consumption.
headroom/proxy/output_shaper.py	Implements request-side verbosity steering and effort routing.
headroom/proxy/output_savings.py	Implements ledger, stratification, deterministic holdout assignment, and estimators.
headroom/proxy/outcome.py	Records output-token observations into the savings ledger (best-effort).
headroom/proxy/handlers/anthropic.py	Wires shaper + holdout assignment + stratum labels into Anthropic request path.
headroom/learn/verbosity.py	Implements transcript parsing, signal extraction, baseline building, and recommendation logic.
headroom/dashboard/templates/dashboard.html	Adds “Output Tokens Saved” hero card.
headroom/cli/output_savings.py	Adds headroom output-savings command.
headroom/cli/main.py	Registers the new CLI command module.
headroom/cli/learn.py	Adds --verbosity / --llm-judge mode and persistence/baseline seeding.
docs/proposals/output-token-reduction.md	Design doc including methodology/constraints and measurement approach.
docs/output-token-reduction-guide.md	End-user guide for enabling, learning, and measuring output reduction.
.gitignore	Ensures the new eval script is not ignored under scripts/*.