Skip to content

Commit 27ddde1

Browse files
fix(transforms/code): coerce language aliases instead of raising (#1975)
## Description `CodeAwareCompressor.compress()` picks the language for AST-based compression like this (`headroom/transforms/code_compressor.py`): ```python if language: detected_lang = CodeLanguage(language.lower()) # <-- raises on anything not an exact enum value confidence = 1.0 elif self.config.language_hint: detected_lang = CodeLanguage(self.config.language_hint.lower()) confidence = 1.0 else: detected_lang, confidence = detect_language(code) ``` `CodeLanguage` only accepts `python`/`javascript`/`typescript`/`go`/`rust`/`java`/`c`/`cpp`/`perl`. The very common markdown fence tags and hints — `js`, `ts`, `py`, `jsx`, `tsx`, `node`, `rs`, `c++` — are **not** enum values, so `CodeLanguage("js")` raises `ValueError`. That construction is *above* the method's own `try/except`, so: - **Direct callers** — `CodeAwareCompressor().compress(code, language="js")` and the module-level `compress_code(code, language="js")` — crash with an uncaught `ValueError`. - **In the router (mixed content):** `split_into_sections` extracts the raw fence tag (`_CODE_FENCE_PATTERN` captures `\w*`, e.g. `js`) into `ContentSection.language`, and that string is passed straight into `compress(...)`. The `ValueError` is swallowed by the outer `try/except` in the strategy dispatch, so a ` ```js ` / ` ```ts ` / ` ```py ` block silently **skips code-aware compression** even when `enable_code_aware=True`, falling back to the generic path. So the three most common web/scripting languages, written with their usual fence tags, never get the structure-aware compressor. Closes: no issue filed — found while auditing the code-compression language path. ## Fix Add a `coerce_language()` helper that maps common aliases/fence tags to the canonical `CodeLanguage` and returns `CodeLanguage.UNKNOWN` (never raises) for anything unrecognized. `compress()` now coerces the hint and, when the result is `UNKNOWN`, falls back to content-based `detect_language(code)` instead of constructing the enum directly: ```python if language: detected_lang = coerce_language(language) if detected_lang == CodeLanguage.UNKNOWN: detected_lang, confidence = detect_language(code) else: confidence = 1.0 ``` ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/transforms/code_compressor.py`: add `_LANGUAGE_ALIASES` and `coerce_language()`; use them in `compress()` for both the `language` argument and `config.language_hint`, with a content-detection fallback on `UNKNOWN`. - `tests/test_code_compressor_language_alias.py`: cover alias mapping, canonical passthrough, case/whitespace handling, unknown-returns-UNKNOWN (no `ValueError`), and that `compress(language="js")` no longer raises. ## Testing - [x] New regression tests added (`tests/test_code_compressor_language_alias.py`) - [x] Linting passes (`ruff check`) and formatting is clean (`ruff format --check`) - [ ] Full `pytest` deferred to CI (local-OOM reason below). ```text $ uv run ruff check headroom/transforms/code_compressor.py tests/test_code_compressor_language_alias.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.10, headroom from this branch. Importing `headroom` pulls in the torch/transformers stack and a full `pytest` gets OOM-killed on this box, so I verified the coercion logic with a dependency-free script (replicating the enum + helper) and left the full pytest to CI. - Exact command / steps: ran the common aliases and the canonical values through both the old `CodeLanguage(value.lower())` construction and the new `coerce_language()`. - Observed result: the old construction raises `ValueError` on every alias (the crash / silent-skip); the new helper maps them and never raises: ```text OK alias 'js': old raised ValueError -> new maps to javascript OK alias 'ts': old raised ValueError -> new maps to typescript OK alias 'py': old raised ValueError -> new maps to python OK alias 'jsx': old raised ValueError -> new maps to javascript OK alias 'node': old raised ValueError -> new maps to javascript OK canonical values pass through OK case-insensitive + trimmed OK unknown -> UNKNOWN (no ValueError) LANGUAGE COERCION VERIFIED ``` - Not tested: running a full mixed-content document with ` ```js ` fences through a booted compression pipeline (needs the heavy stack). The unit tests exercise the coercion directly and the `compress(language="js")` entry point. Full local `pytest` deferred to CI (OOM, per above). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes — ran lint + a standalone logic check; full pytest deferred to CI (local OOM, disclosed above) - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - No new dependencies; a small lookup table plus a helper and a call-site change. - @JerrettDavis tagging you — this one silently disables code-aware compression for the most common fence tags (`js`/`ts`/`py`), so it may be worth a look when you have a moment.
1 parent 69fd218 commit 27ddde1

3 files changed

Lines changed: 128 additions & 5 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
4141
* **install:** stop leaking a file descriptor on every `headroom install start`. `start_detached_agent()` opened the agent log file and handed it to `subprocess.Popen` but never closed the parent's copy, so each call leaked one fd (and pinned the log file open against rotation). The parent now closes its copy in a `try/finally` once the child has inherited it — the close also runs if `Popen` raises ([#1554](https://github.com/headroomlabs-ai/headroom/issues/1554)).
4242
* **memory/sync:** stop the Codex AGENTS.md sync adapter from erasing previously-synced memories on every export. `sync_export` hands each adapter only the *delta* (memories the agent lacks), but `CodexAdapter.write_memories` rebuilt its whole managed section from just that delta — so each sync overwrote the section with only the new items, thrashing the file between disjoint subsets and never accumulating. It now merges the delta into the facts already present (deduped), matching the additive contract the ClaudeCode adapter already follows.
4343
* **memory/sync:** stop the Claude Code sync adapter from clobbering distinct memories that share a first line. `write_memories` derived each file name from the first line of the content only (`headroom_{slug}.md`), so two different DB memories whose first lines slugify identically wrote to the same file and the second silently overwrote the first — and because the loser never landed on disk, the next sync re-exported it, ping-ponging the pair forever. When the slug is already taken by a *different* memory (distinct `headroom_id`) the file name is now disambiguated with a content-hash suffix; an update to the same memory still rewrites its slug file in place, so existing file names are unchanged.
44+
* **transforms/code:** stop raising `ValueError` on common language hints and fence tags. `CodeAwareCompressor.compress()` built the language with `CodeLanguage(language.lower())`, which only accepts the exact enum values (`python`/`javascript`/`typescript`/…). A markdown ` ```js ` / ` ```ts ` / ` ```py ` fence tag (or any caller passing an alias) raised `ValueError` — crashing direct callers, and inside the content router the error was swallowed so those blocks silently skipped code-aware compression. A new `coerce_language` helper maps the common aliases to their canonical language and returns `UNKNOWN` (never raises) for unrecognized tags, falling back to content-based detection.
4445
* **cli/proxy:** honor `HEADROOM_MIN_TOKENS=0` / `HEADROOM_MAX_ITEMS=0`. The Click `proxy` command built these with `_get_env_int_optional(name) or 500`/`or 50`, so an explicit `0` — a legitimate value (`min_tokens_to_crush=0` means "crush every item") — was treated as falsy and silently replaced with the default. The `headroom proxy` argparse path already preserved `0` via `_get_env_int`, so the two entry points disagreed. The Click path now uses the same None-checking helper.
4546
* **proxy:** strip the inbound `Content-Encoding`/`Transfer-Encoding` request headers on the Anthropic `/v1/messages` and OpenAI `/v1/chat/completions` paths before forwarding upstream. `read_request_json_with_bytes` already decompresses the inbound body (zstd/gzip/deflate/br), so the bytes forwarded upstream are plain JSON — but these two handlers left the original `content-encoding` header in place, so a client (or an edge proxy like a Cloudflare Worker) that sent a compressed body got its request rejected with upstream HTTP 400 because the provider tried to decompress already-decoded JSON. The `/v1/responses` handler already carried this fix (#1542); it is now applied to the messages and chat paths too.
4647
* **models:** fix the model registry's prefix fallback silently returning the wrong context window. `ModelRegistry.get` accepted any registered name as a `str.startswith` prefix and returned the *first* match, so `gpt-4-32k-0613` resolved to `gpt-4` (8192) instead of `gpt-4-32k` (32768), and unregistered ids like `gpt-4.1`/`gpt-4.5` inherited `gpt-4`'s 8192-token window — making the proxy think a nearly-empty context was almost full and compress far too aggressively. The fallback now requires the registered name to end at a version boundary in the query (so `gpt-4.1` no longer matches `gpt-4`) and picks the longest qualifying name (so `gpt-4-32k-0613``gpt-4-32k`).

headroom/transforms/code_compressor.py

Lines changed: 57 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -219,6 +219,49 @@ class CodeLanguage(Enum):
219219
UNKNOWN = "unknown"
220220

221221

222+
# Common language hints and markdown fence tags that are not the canonical
223+
# ``CodeLanguage`` value. Mapping them here keeps ``` ```js ``` / ``` ```ts ```
224+
# / ``` ```py ``` fenced blocks (and callers that pass an alias) on the
225+
# code-aware path instead of raising ValueError.
226+
_LANGUAGE_ALIASES: dict[str, CodeLanguage] = {
227+
"js": CodeLanguage.JAVASCRIPT,
228+
"jsx": CodeLanguage.JAVASCRIPT,
229+
"mjs": CodeLanguage.JAVASCRIPT,
230+
"cjs": CodeLanguage.JAVASCRIPT,
231+
"node": CodeLanguage.JAVASCRIPT,
232+
"ts": CodeLanguage.TYPESCRIPT,
233+
"tsx": CodeLanguage.TYPESCRIPT,
234+
"py": CodeLanguage.PYTHON,
235+
"python3": CodeLanguage.PYTHON,
236+
"golang": CodeLanguage.GO,
237+
"rs": CodeLanguage.RUST,
238+
"c++": CodeLanguage.CPP,
239+
"cxx": CodeLanguage.CPP,
240+
"cc": CodeLanguage.CPP,
241+
"hpp": CodeLanguage.CPP,
242+
"pl": CodeLanguage.PERL,
243+
}
244+
245+
246+
def coerce_language(value: str) -> CodeLanguage:
247+
"""Map a language hint or markdown fence tag to a ``CodeLanguage``.
248+
249+
Accepts the canonical enum values and common aliases/fence tags
250+
(``js``/``ts``/``py``/...). Unknown strings return ``CodeLanguage.UNKNOWN``
251+
instead of raising ``ValueError`` from ``CodeLanguage(value)``, so an
252+
unrecognized fence tag falls back to content-based detection rather than
253+
crashing the caller (or, inside the router, silently skipping code-aware
254+
compression because the ValueError is swallowed).
255+
"""
256+
key = (value or "").strip().lower()
257+
if not key:
258+
return CodeLanguage.UNKNOWN
259+
try:
260+
return CodeLanguage(key)
261+
except ValueError:
262+
return _LANGUAGE_ALIASES.get(key, CodeLanguage.UNKNOWN)
263+
264+
222265
class DocstringMode(Enum):
223266
"""How to handle docstrings."""
224267

@@ -1015,13 +1058,22 @@ def compress(
10151058
syntax_valid=True,
10161059
)
10171060

1018-
# Detect or use specified language
1061+
# Detect or use specified language. An explicit hint or fence tag may be
1062+
# an alias (js/ts/py/...) or something we don't recognize — coerce it
1063+
# instead of constructing CodeLanguage() directly (which raises), and
1064+
# fall back to content detection when the hint is unknown.
10191065
if language:
1020-
detected_lang = CodeLanguage(language.lower())
1021-
confidence = 1.0
1066+
detected_lang = coerce_language(language)
1067+
if detected_lang == CodeLanguage.UNKNOWN:
1068+
detected_lang, confidence = detect_language(code)
1069+
else:
1070+
confidence = 1.0
10221071
elif self.config.language_hint:
1023-
detected_lang = CodeLanguage(self.config.language_hint.lower())
1024-
confidence = 1.0
1072+
detected_lang = coerce_language(self.config.language_hint)
1073+
if detected_lang == CodeLanguage.UNKNOWN:
1074+
detected_lang, confidence = detect_language(code)
1075+
else:
1076+
confidence = 1.0
10251077
else:
10261078
detected_lang, confidence = detect_language(code)
10271079

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
"""Regression tests for language-hint / fence-tag coercion in code_compressor.
2+
3+
`CodeAwareCompressor.compress(code, language=...)` used to build the language
4+
with `CodeLanguage(language.lower())`, which raises `ValueError` for anything
5+
that is not an exact enum value. Common markdown fence tags and hints — `js`,
6+
`ts`, `py` — are not enum values, so:
7+
8+
* direct callers (`compress(code, language="js")`) crashed, and
9+
* inside the router the ValueError was swallowed, so ` ```js ` / ` ```ts ` /
10+
` ```py ` fenced blocks silently skipped code-aware compression.
11+
12+
`coerce_language` maps aliases to the canonical language and returns UNKNOWN
13+
(never raises) for unrecognized tags, letting the caller fall back to
14+
content-based detection.
15+
"""
16+
17+
import pytest
18+
19+
from headroom.transforms.code_compressor import CodeLanguage, coerce_language
20+
21+
22+
@pytest.mark.parametrize(
23+
"alias,expected",
24+
[
25+
("js", CodeLanguage.JAVASCRIPT),
26+
("jsx", CodeLanguage.JAVASCRIPT),
27+
("node", CodeLanguage.JAVASCRIPT),
28+
("ts", CodeLanguage.TYPESCRIPT),
29+
("tsx", CodeLanguage.TYPESCRIPT),
30+
("py", CodeLanguage.PYTHON),
31+
("python3", CodeLanguage.PYTHON),
32+
("golang", CodeLanguage.GO),
33+
("rs", CodeLanguage.RUST),
34+
("c++", CodeLanguage.CPP),
35+
],
36+
)
37+
def test_coerce_language_maps_common_aliases(alias, expected):
38+
assert coerce_language(alias) == expected
39+
40+
41+
@pytest.mark.parametrize(
42+
"canonical",
43+
["python", "javascript", "typescript", "go", "rust", "java", "c", "cpp", "perl"],
44+
)
45+
def test_coerce_language_accepts_canonical_values(canonical):
46+
assert coerce_language(canonical) == CodeLanguage(canonical)
47+
48+
49+
def test_coerce_language_is_case_insensitive_and_trims():
50+
assert coerce_language(" JS ") == CodeLanguage.JAVASCRIPT
51+
assert coerce_language("Python") == CodeLanguage.PYTHON
52+
53+
54+
@pytest.mark.parametrize("value", ["", " ", "not-a-language", "brainfuck", "yaml"])
55+
def test_coerce_language_unknown_returns_unknown_not_valueerror(value):
56+
# The whole point: never raise, so an unrecognized fence tag can fall back
57+
# to content detection instead of crashing / being swallowed.
58+
assert coerce_language(value) == CodeLanguage.UNKNOWN
59+
60+
61+
def test_compress_with_alias_language_does_not_raise():
62+
"""The direct API path must not raise on a common alias."""
63+
from headroom.transforms.code_compressor import CodeAwareCompressor
64+
65+
code = "function add(a, b) {\n return a + b;\n}\n"
66+
compressor = CodeAwareCompressor()
67+
# Before the fix this raised ValueError: 'js' is not a valid CodeLanguage.
68+
result = compressor.compress(code, language="js")
69+
assert result is not None
70+
assert result.compressed is not None

0 commit comments

Comments
 (0)