Skip to content

Commit 6d5516d

Browse files
ParideboyclaudeJerrettDavis
authored
feat(code): add PHP support to CodeAwareCompressor (headroomlabs-ai#2423)
## Description Adds PHP to `CodeAwareCompressor`, fixing headroomlabs-ai#201. PHP was already *detected* as code (Magika labels in `headroom/compression/detector.py` include `php`, and the Rust `magika_detector.rs` lists it too) but there was no PHP `LangConfig`, so PHP content silently passed through uncompressed. This wires PHP through the tree-sitter compression path following the C# pattern (the most recently added, fully functional language — deliberately not the quarantined Perl path). A secondary detection bug is fixed along the way: PHP's `$variables` match Perl's prefilter regex, and the existing Perl-dominance guard in `detect_language` returned `UNKNOWN` for PHP files. An explicit `<?php` open tag — which no Perl source contains — now drops Perl from the candidate set before that guard runs. ## Type of Change - [ ] Bug fix - [x] New feature - [ ] Documentation update - [ ] Refactor - [ ] Other ## Changes Made - `headroom/transforms/code_compressor.py`: `CodeLanguage.PHP` + `phtml`/`php5`/`php7`/`php8` aliases; PHP `LangConfig` built from the actual tree-sitter-php grammar (node names verified by parsing samples): `namespace_use_declaration` imports, `function_definition`/`method_declaration` functions, `class_declaration`/`interface_declaration`/`trait_declaration` classes, `enum_declaration` types, `declaration_list` class bodies, `compound_statement` function bodies. `namespace_definition` maps to `package_node` so statement-scoped `namespace App;` hoists ahead of the `use` imports (required PHP ordering); the rare block-scoped `namespace A { }` form takes the same path and is preserved verbatim — valid output, just no compression inside the block. PHP prefilter regexes added; supported-languages error message updated; `<?php`-tag Perl disambiguation in `detect_language`. - `headroom/transforms/content_detector.py`: `php` entry in `_CODE_PATTERNS` so raw PHP classifies as `SOURCE_CODE` and reaches the code-aware route. - `tests/test_transforms/test_code_compressor.py`: new `TestPhpSupport` mirroring `TestCSharpSupport` — signatures preserved / bodies elided, `<?php` → `namespace` → `use` → declarations ordering, auto-detection despite the Perl sigil overlap, alias coercion, malformed passthrough. - `tests/test_code_compressor_language_alias.py`: `php` in the canonical list, `phtml` in the alias table. - `docs/content/docs/code-compression.mdx`: PHP added to the Tier 2 supported-languages row. No new dependency: `tree-sitter-language-pack` (the existing `[code]` extra) already ships the PHP grammar. No Rust changes needed. ## Testing - [x] New unit tests added and passing - [x] Full affected test suites pass locally **Test Output** ``` $ python -m pytest tests/test_transforms/test_code_compressor.py tests/test_code_compressor_language_alias.py -q ============================= 120 passed in 7.81s ============================= $ python -m pytest tests/test_transforms/ -q 3 failed, 443 passed # the 3 failures (kompress ONNX thread caps, kompress size gate, # text_crusher unicode parity) reproduce identically on a clean # upstream/main checkout in this environment — pre-existing local # ONNX runtime quirks, unrelated to this change $ ruff check . (0.15.17, CI-pinned) → All checks passed! | ruff format --check → clean $ mypy headroom/transforms/code_compressor.py headroom/transforms/content_detector.py --ignore-missing-imports Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13, tree-sitter + tree-sitter-language-pack (<1.0) installed, branch `feat/201-php-code-compression` off `upstream/main`. - Exact command / steps: parsed PHP samples (namespaced class w/ methods, block-scoped namespace, mixed HTML+PHP) with `tree_sitter_language_pack.get_parser('php')` to verify every node name used in the config; then ran `CodeAwareCompressor().compress(php_code, language="php")` and `compress(php_code)` (auto-detection) on a 48-line realistic service class. - Observed result: explicit and auto-detected paths both return `language=CodeLanguage.PHP`, `compression_ratio=0.64`, `syntax_valid=True`; method bodies elided to `// [N lines omitted]` while `<?php`, `namespace`, `use` lines, class header, and all signatures are preserved verbatim in the original order. Before the detection fix, auto-detection returned `UNKNOWN` (Perl prefilter dominance) — reproduced and then verified fixed. - Not tested: exotic PHP shapes (heredoc-heavy code, attributes `#[...]` on methods, interleaved multi-`<?php ?>` HTML templates beyond the basic mixed case); these fall back to verbatim preservation via the uncaptured-node pass or malformed-passthrough, both of which are covered by tests for the simple cases. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
1 parent b7a79ac commit 6d5516d

6 files changed

Lines changed: 169 additions & 6 deletions

File tree

docs/content/docs/code-compression.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ Naive truncation breaks code. Cutting a function in half leaves invalid syntax t
1818
| Tier | Languages | Support Level |
1919
|---|---|---|
2020
| Tier 1 | Python, JavaScript, TypeScript | Full AST analysis |
21-
| Tier 2 | Go, Rust, Java, C, C++ | Function body compression |
21+
| Tier 2 | Go, Rust, Java, C, C++, C#, PHP | Function body compression |
2222

2323
## What Gets Preserved vs Compressed
2424

headroom/transforms/code_compressor.py

Lines changed: 40 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -167,7 +167,7 @@ def _get_parser(language: str) -> Any:
167167
except Exception as e:
168168
raise ValueError(
169169
f"Language '{language}' is not supported by tree-sitter. "
170-
f"Supported: python, javascript, typescript, go, rust, java, c, cpp, csharp. "
170+
f"Supported: python, javascript, typescript, go, rust, java, c, cpp, csharp, php. "
171171
f"Error: {e}"
172172
) from e
173173

@@ -223,6 +223,7 @@ class CodeLanguage(Enum):
223223
CPP = "cpp"
224224
PERL = "perl"
225225
CSHARP = "csharp"
226+
PHP = "php"
226227
UNKNOWN = "unknown"
227228

228229

@@ -247,6 +248,10 @@ class CodeLanguage(Enum):
247248
"cc": CodeLanguage.CPP,
248249
"hpp": CodeLanguage.CPP,
249250
"pl": CodeLanguage.PERL,
251+
"phtml": CodeLanguage.PHP,
252+
"php5": CodeLanguage.PHP,
253+
"php7": CodeLanguage.PHP,
254+
"php8": CodeLanguage.PHP,
250255
}
251256

252257

@@ -461,6 +466,23 @@ class LangConfig:
461466
container_node_types=frozenset({"namespace_declaration"}),
462467
opaque_node_types=frozenset({"preproc_if"}),
463468
),
469+
CodeLanguage.PHP: LangConfig(
470+
import_nodes=frozenset({"namespace_use_declaration"}),
471+
function_nodes=frozenset({"function_definition", "method_declaration"}),
472+
class_nodes=frozenset({"class_declaration", "interface_declaration", "trait_declaration"}),
473+
type_nodes=frozenset({"enum_declaration"}),
474+
body_node_types=frozenset({"compound_statement"}),
475+
decorator_node=None,
476+
comment_prefix="//",
477+
uses_colon_after_signature=False,
478+
# Statement-scoped `namespace App;` hoists to the top of the output
479+
# (before the use declarations); the rarer block-scoped
480+
# `namespace A { ... }` form takes the same path and is preserved
481+
# verbatim — valid output, no compression inside the block.
482+
package_node="namespace_definition",
483+
detection_hints=("<?php", "function ", "namespace ", "->", "$this"),
484+
class_body_node_types=frozenset({"declaration_list"}),
485+
),
464486
}
465487

466488

@@ -666,6 +688,16 @@ def summary(self) -> str:
666688
),
667689
re.compile(r"\bget;\s*set;", re.MULTILINE),
668690
],
691+
CodeLanguage.PHP: [
692+
re.compile(r"<\?php\b"),
693+
re.compile(r"^\s*namespace\s+[\w\\]+\s*;", re.MULTILINE),
694+
re.compile(r"^\s*use\s+[\w\\]+(\s+as\s+\w+)?\s*;", re.MULTILINE),
695+
re.compile(
696+
r"^\s*(public|private|protected|static|abstract|final)?\s*function\s+\w+\s*\(",
697+
re.MULTILINE,
698+
),
699+
re.compile(r"\$this->|->\w+\s*\(", re.MULTILINE),
700+
],
669701
}
670702

671703

@@ -720,6 +752,13 @@ def detect_language(code: str) -> tuple[CodeLanguage, float]:
720752
if candidates[CodeLanguage.CPP] >= 2:
721753
candidates[CodeLanguage.C] = 0
722754

755+
# Disambiguation: PHP's sigil variables ($x) overlap Perl's prefilter.
756+
# An explicit `<?php` open tag is unambiguous — no Perl source contains
757+
# it, so drop Perl from the candidates before the Perl-dominance guard
758+
# below returns UNKNOWN for what is actually PHP.
759+
if CodeLanguage.PHP in candidates and "<?php" in sample:
760+
candidates.pop(CodeLanguage.PERL, None)
761+
723762
perl_score = candidates.get(CodeLanguage.PERL, 0)
724763
if perl_score > 0:
725764
best_non_perl = max(

headroom/transforms/content_detector.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,13 @@ class DetectionResult:
111111
),
112112
re.compile(r"^.*\b(get|set|init);"), # auto-property accessors
113113
],
114+
"php": [
115+
re.compile(r"<\?php\b"),
116+
re.compile(r"^\s*namespace\s+[\w\\]+\s*;"),
117+
re.compile(r"^\s*use\s+[\w\\]+(\s+as\s+\w+)?\s*;"),
118+
re.compile(r"^\s*(public|private|protected|static|abstract|final)?\s*function\s+\w+\s*\("),
119+
re.compile(r"\$this->"),
120+
],
114121
}
115122

116123
# Structured-config (YAML/TOML/INI) patterns. TOML and INI share the

tests/test_code_compressor_language_alias.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@
3232
("golang", CodeLanguage.GO),
3333
("rs", CodeLanguage.RUST),
3434
("c++", CodeLanguage.CPP),
35+
("phtml", CodeLanguage.PHP),
3536
],
3637
)
3738
def test_coerce_language_maps_common_aliases(alias, expected):
@@ -40,7 +41,7 @@ def test_coerce_language_maps_common_aliases(alias, expected):
4041

4142
@pytest.mark.parametrize(
4243
"canonical",
43-
["python", "javascript", "typescript", "go", "rust", "java", "c", "cpp", "perl"],
44+
["python", "javascript", "typescript", "go", "rust", "java", "c", "cpp", "perl", "php"],
4445
)
4546
def test_coerce_language_accepts_canonical_values(canonical):
4647
assert coerce_language(canonical) == CodeLanguage(canonical)

tests/test_proxy_savings_history.py

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1227,9 +1227,11 @@ def test_savings_tracker_batches_saves_and_matches_immediate(tmp_path):
12271227
batched.record_request(**events[2]) # buffered again
12281228
batched.flush() # tail persisted
12291229

1230-
assert json.loads(batched_path.read_text(encoding="utf-8")) == json.loads(
1231-
immediate_path.read_text(encoding="utf-8")
1232-
)
1230+
batched_payload = json.loads(batched_path.read_text(encoding="utf-8"))
1231+
immediate_payload = json.loads(immediate_path.read_text(encoding="utf-8"))
1232+
for payload in (batched_payload, immediate_payload):
1233+
payload["lifetime_metrics"]["persistence"].pop("last_saved_at", None)
1234+
assert batched_payload == immediate_payload
12331235

12341236

12351237
def test_failed_save_retries_on_next_record_not_after_full_window(tmp_path, monkeypatch):

tests/test_transforms/test_code_compressor.py

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
CodeCompressorConfig,
2222
CodeLanguage,
2323
DocstringMode,
24+
coerce_language,
2425
detect_language,
2526
is_tree_sitter_available,
2627
is_tree_sitter_loaded,
@@ -2096,3 +2097,116 @@ def test_detect_language_identifies_csharp(self):
20962097
lang, confidence = detect_language(code)
20972098
assert lang == CodeLanguage.CSHARP
20982099
assert confidence > 0.0
2100+
2101+
2102+
@pytest.mark.skipif(not TREE_SITTER_INSTALLED, reason="tree-sitter grammar pack not installed")
2103+
class TestPhpSupport:
2104+
"""PHP (``php`` grammar) parity with C#: signatures preserved verbatim,
2105+
function/method bodies compressed, ``<?php`` tag and ``namespace``/``use``
2106+
header order preserved, Perl sigil-overlap disambiguated in detection,
2107+
malformed input passed through.
2108+
"""
2109+
2110+
def _compressor(self):
2111+
return CodeAwareCompressor(
2112+
CodeCompressorConfig(
2113+
min_tokens_for_compression=1,
2114+
max_body_lines=1,
2115+
enable_ccr=False,
2116+
)
2117+
)
2118+
2119+
def test_class_methods_compress_signatures_preserved(self):
2120+
code = (
2121+
"<?php\n"
2122+
"namespace App\\Service;\n"
2123+
"\n"
2124+
"use App\\Model\\User;\n"
2125+
"\n"
2126+
"final class UserService {\n"
2127+
" private $logger;\n"
2128+
"\n"
2129+
" public function process(User $u): bool {\n"
2130+
" $name = strtolower(trim($u->getName()));\n"
2131+
" $tags = [];\n"
2132+
" foreach ($u->getTags() as $tag) {\n"
2133+
" $tags[] = $tag->normalize();\n"
2134+
" }\n"
2135+
" $this->logger->info($name);\n"
2136+
" return true;\n"
2137+
" }\n"
2138+
"}\n"
2139+
)
2140+
result = self._compressor().compress(code, language="php")
2141+
2142+
assert result.language == CodeLanguage.PHP
2143+
assert result.syntax_valid is True
2144+
assert result.compression_ratio < 1.0
2145+
# signature + class header preserved verbatim
2146+
assert "final class UserService" in result.compressed
2147+
assert "public function process(User $u): bool" in result.compressed
2148+
# method body actually compressed
2149+
assert "lines omitted" in result.compressed
2150+
assert "$tag->normalize()" not in result.compressed
2151+
# the class is emitted exactly once
2152+
assert result.compressed.count("class UserService") == 1
2153+
2154+
def test_php_tag_and_namespace_precede_uses_and_types(self):
2155+
"""``<?php`` must stay first and ``namespace X;`` must precede the
2156+
``use`` imports and type declarations — any other order is not valid
2157+
PHP."""
2158+
code = (
2159+
"<?php\n"
2160+
"namespace App\\Tools;\n"
2161+
"\n"
2162+
"use App\\Model\\Item;\n"
2163+
"\n"
2164+
"function helper(int $x): int {\n"
2165+
" $acc = 0;\n"
2166+
" for ($i = 0; $i < $x; $i++) {\n"
2167+
" $acc += $i;\n"
2168+
" $acc -= 1;\n"
2169+
" }\n"
2170+
" return $acc;\n"
2171+
"}\n"
2172+
)
2173+
result = self._compressor().compress(code, language="php")
2174+
2175+
assert result.language == CodeLanguage.PHP
2176+
assert result.syntax_valid is True
2177+
assert result.compression_ratio < 1.0
2178+
compressed = result.compressed
2179+
assert compressed.lstrip().startswith("<?php")
2180+
assert compressed.index("<?php") < compressed.index("namespace App\\Tools;")
2181+
assert compressed.index("namespace App\\Tools;") < compressed.index("use App\\Model\\Item;")
2182+
assert compressed.index("use App\\Model\\Item;") < compressed.index("function helper")
2183+
assert "lines omitted" in compressed
2184+
2185+
def test_detect_language_identifies_php(self):
2186+
"""Auto-detection recognizes PHP despite the Perl sigil overlap
2187+
(``$var`` matches Perl's prefilter; the ``<?php`` tag disambiguates)."""
2188+
code = (
2189+
"<?php\n"
2190+
"namespace Acme;\n"
2191+
"\n"
2192+
"use Acme\\Widget;\n"
2193+
"\n"
2194+
"class Svc {\n"
2195+
" public function add(int $a, int $b): int {\n"
2196+
" $sum = $a + $b;\n"
2197+
" return $sum;\n"
2198+
" }\n"
2199+
"}\n"
2200+
)
2201+
lang, confidence = detect_language(code)
2202+
assert lang == CodeLanguage.PHP
2203+
assert confidence > 0.0
2204+
2205+
def test_phtml_alias_coerces_to_php(self):
2206+
assert coerce_language("phtml") == CodeLanguage.PHP
2207+
assert coerce_language("php8") == CodeLanguage.PHP
2208+
2209+
def test_malformed_php_passes_through_unchanged(self):
2210+
code = "<?php\nclass Broken {\n public function oops( {\n"
2211+
result = self._compressor().compress(code, language="php")
2212+
assert result.compressed == code

0 commit comments

Comments
 (0)