Skip to content

TECHNIQUES.md

Latest commit

 

History

History
133 lines (101 loc) · 5.94 KB

TECHNIQUES.md

File metadata and controls

133 lines (101 loc) · 5.94 KB
Raw

Compression Techniques — Design Document

What We're Proving

Code written for human reading carries massive overhead that serves neither the machine executing it nor an LLM reasoning about it. We demonstrate 5 levels of compression, each preserving 100% functionality.

The Core Insight

Human code has 4 types of waste:

Type Example % of Click Machine value
Ceremonial Docstrings, Sphinx markup, version notes 23% Zero
Structural Blank lines, import repetition across files 19% Zero
Syntactic Verbose conditionals, redundant names ~10% Near-zero
Architectural 16 files × 16 import blocks, scattered related code ~5% Negative (harder to reason about)

Total non-logic overhead: ~47% of lines, ~40% of tokens.

Level Definitions

L1: Syntactic Clean

Rule: Remove everything that isn't executed.

Techniques:

  • Strip all docstrings (""" blocks inside functions/classes/modules)
  • Strip all comments (# lines)
  • Collapse multiple blank lines to single separators
  • Remove trailing whitespace
  • Keep all code, all imports, all type hints exactly as-is

What stays: Every executable line, every import, every type annotation. What goes: Docstrings, comments, excessive blank lines. Verification: AST comparison (identical minus Expr(Constant(str)) nodes).

L2: Logic Compression

Rule: Same behavior, fewer tokens.

Techniques:

  • Merge single-use helper functions inline
  • Simplify conditional chains (if/elif → dict lookup where applicable)
  • Use comprehensions instead of loop-append patterns
  • Deduplicate repeated patterns (e.g., Click has many "process params" loops)
  • Remove platform-specific code that's dead on the target platform (but we keep _winconsole.py as it IS part of Click — note its size)
  • Merge trivial one-liner methods into callers where called once
  • Use walrus operator (:=) where it eliminates duplicate calls
  • Combine related assignments

What stays: All public API, all behavior, all edge cases. What goes: Verbose patterns that can be expressed more densely. Verification: Full Click test suite must pass.

L3: Shard Architecture

Rule: Related logic lives together. Imports amortized. Machine-first headers.

Techniques:

  • Merge 16 files → 4-6 shards by logical domain:
    • _shard_core.py — Context, Command, Group, Parameter, Option, Argument
    • _shard_types.py — ParamType hierarchy, type conversion
    • _shard_ui.py — terminal UI, progress bars, pager, prompt, style
    • _shard_completion.py — shell completion (Bash/Zsh/Fish)
    • _shard_util.py — helpers, compat, testing, formatting
  • Each shard gets # AX: TAG | SUM: one-line summary header
  • Shared imports declared once per shard (saves ~200 tokens per eliminated file)
  • __init__.py with lazy loading for public API compatibility
  • Logically cohesive: related classes/functions stay together regardless of LOC

What stays: All public API, all behavior, all internal structure. What goes: File boundaries that scattered related code, redundant imports. Verification: Full Click test suite + import compatibility.

L4: Navigable (Machine-Indexed)

Rule: Replace human docs with machine-queryable metadata.

Techniques:

  • # SYM: headers mapping abbreviated names to semantics
  • # CALLS: fn1, fn2 / # CALLED_BY: fn3 dependency annotations
  • Contract signatures: # CONTRACT: (str, int) -> Command | raises UsageError
  • Section markers: # --- PARAMETER PROCESSING --- for rapid navigation
  • This ADDS tokens vs L3, but replaces the NEED for docstrings entirely

What stays: Everything from L3 + navigation metadata. What goes: Nothing removed vs L3 (this level adds navigability). Verification: Full Click test suite + metadata accuracy.

Why This Matters

For LLMs (Context Window)

  • 200K context ≈ 150K usable tokens
  • Original Click: 82K tokens = uses 55% of context for ONE library
  • L3 compressed: ~30K tokens = uses 20% of context
  • An LLM can now hold Click + 2 other libraries simultaneously

For Humans (if they care)

  • Docstrings served Sphinx docs generation. In 2026, LLMs generate docs on demand.
  • Comments explained "why" — but code that needs explaining is code that needs rewriting.
  • The shard system groups related code, making navigation EASIER not harder.

For Cost

  • 82K → 30K tokens = 63% fewer input tokens per API call
  • At $3/MTok (Claude), that's $0.16 saved per full-library read
  • Over 1000 reads in a research mission: $160 saved on ONE library

What's New vs Previous Atlas Work

The M2M benchmark tested DATA format compression (human prose vs structured M2M). This benchmark introduces CODE compression — fundamentally different:

  1. Architectural compression — merging files eliminates import overhead
  2. Logic compression — rewriting algorithms, not just reformatting
  3. Navigation metadata — Cortex-style annotations replace docstrings
  4. Verification via test suite — not Q&A scoring but real execution
  5. Public reproducibility — anyone can pip install click, run tests, verify

Potential Weaknesses to Address

  1. "You just removed docs" — L1 alone IS just removing docs. L2-L4 go far beyond. Counter: We measure tokens, not just LOC. L2 logic changes compress pure code too.

  2. "Click tests might not cover everything" — True. Counter: We run the official test suite. If Click trusts it, so can we.

  3. "Compressed code is unreadable" — For humans, yes (L2+). That's the point. Counter: L4 adds machine navigation. LLMs read L3 BETTER than L0 (proven in M2M benchmark).

  4. "This only works for Click" — Single project, might not generalize. Counter: We plan Flask and Bottle as follow-ups. Different architectures, same techniques.

  5. "Import amortization is trivial" — Each file saves ~10-20 tokens of imports. Counter: 16 files × 15 tokens = 240 tokens. Not huge, but compounds with other savings.