While evaluating TurboQuant (ICLR 2026 KV cache compression), we realized the core math (Lloyd-Max quantization, QJL projection, unbiased estimators) could be repurposed at the language level rather than the tensor level.
Core insight: TurboQuant compresses KV cache bits to fit more context in VRAM. But language itself is a compression medium — a structured language state (scaffold) of N tokens, when "decompressed" by an LLM, produces behavior of complexity C where C >> N. The compression ratio C/N is measurable and optimizable.
This isn't new territory — we've been proving it empirically without naming it:
- Prior scaffold experiments — Systematic evaluation showing scaffolded small models outperform naive large models on eval, content, and research tasks.
- scaffold-independence — Contrastive markers: +18-27pp accuracy across Banking77/CLINC150/MASSIVE. Embedding centroid classification with zero LLM calls.
- QuickThink — Compressed planning grammar (
g:;c:;s:;r:) in 6-16 tokens. 30,900+ JSONL rows across 9 models, 30+ variants. - Hypothesis Scaffold — Zero-LLM recursive engine: 113 hypotheses in 70 seconds from structured artifacts. One calibration pass, then AI discarded.
- Epistemic Experiments — Matched-pair probes showing same evidence under different linguistic framing → different model behavior. Published null-bias data on Zenodo.
Key prior finding: Scaffolding caused interference on operational tasks when injecting information the model already has. This is consistent with a redundancy = interference prediction from LangQuant's compression framework.
Prior experiments proved the effect. LangQuant formalizes it:
- Compression ratio metric —
behavioral_complexity(output) / entropy(scaffold)— tokens-in vs behavioral-complexity-out - Continuous scaling ladder — Qwen3.5 at 0.8b/2b/4b/9b/27b via Ollama (same architecture, different scale). Prior work only tested 3 discrete API tiers which differ in training, not just scale.
- Information-theoretic measurement — Shannon entropy of scaffold token distributions, per-token importance via ablation
- Testable claim: For any LLM of parameter count P, there exists a language state S such that LLM(S) ≈ behavior of model with kP parameters on naive prompt, where k > 1 and k is bounded by information density of S.
- infomeasure — entropy, mutual information, transfer entropy, divergences
- pyitlib — lightweight entropy, MI, conditional entropy, KL divergence
- inspect-ai — UK AISI behavioral eval framework (native Ollama support)
- scipy.stats — significance testing
- dit — build failed (pycddlib). pyitlib + infomeasure cover same ground.
- promptfoo — our quickthink eval harness is more mature for our use case
- lm-eval-harness — standard benchmarks, not what we're measuring
- HELM — heavy, API-focused, overkill
- deepeval — output quality only, not info-theoretic
- LLMLingua — not the tool, but borrowing the per-token ablation methodology
run_experiment.py — reuses patterns from prior experiment infrastructure (Ollama HTTP, JSON extraction, CSV streaming, locking, manifests, JSONL streaming, resume, keyword detection).
5 scaffold conditions (progressive richness):
naked— raw prompt, no scaffoldcontrastive— "This IS X. This is NOT Y." markersquickthink— compressed planning grammar (g:;c:;s:;r:)full_scaffold— contrastive + quickthink + requirementshypothesis_artifacts— richest state: structured context + method + quality constraints
12 starter tasks across 4 groups: reasoning (CRT), calibration (myth detection), analysis (nuanced eval), classification.
- 5 models × 5 conditions × 12 tasks × 3 runs = 900 trials (full run)
- Results →
results/run_results.jsonl - Manifest →
results/run_manifest.json
First run: qwen3.5:0.8b, conditions naked + contrastive, 2 tasks, 1 run each = 4 trials.
Fix applied: Switched from /api/generate to /api/chat with think: false — Qwen3.5 was consuming all output in thinking tags and returning empty responses.
| Condition | Task | Score | BC | Scaffold Entropy | Compression Ratio |
|---|---|---|---|---|---|
| naked | R-001 (sheep) | 1.0 | 18.50 | 0 | — |
| contrastive | R-001 (sheep) | 1.0 | 23.56 | 4.22 | 5.58 |
| naked | R-002 (widgets) | 0.9 | 15.99 | 0 | — |
| contrastive | R-002 (widgets) | 1.0 | 42.42 | 4.22 | 10.05 |
Observation: Contrastive scaffold (34 tokens) boosted R-002 from 0.9→1.0 score and nearly tripled behavioral complexity (16→42). Compression ratio of 10.05 = each bit of scaffold entropy produced 10 units of behavioral complexity gain.
Files:
results/smoke_test_v2.jsonl— raw trial dataresults/smoke_test_v2_manifest.json— reproducibility manifestresults/run_results.jsonl— v1 (broken, empty responses due to thinking tags)
Anchored Microgrammar Theory (from QuickThink research docs):
- Formalizes why QuickThink's
g:;c:;s:;r:works — syntax as infrastructure for cognition, not formatting - Wittgensteinian language-games: meaning is use within a rule-governed practice
- Semantic Curvature Principle: some tokens matter more than others. "Semantic curvature" = how much output changes when you edit one token. High-curvature fields (strategy, constraints) must survive compression. Low-curvature fields can be dropped.
- Contains falsifiable hypothesis H2 that LangQuant can directly test: "perturbations in
sproduce larger answer deltas than perturbations ing"
In-Context State (from prior agent architecture work):
- "What you say becomes what you remember" — the conversation history IS the VM's working memory
- Structured narration with text-prefixed markers to persist state
- This is the same thesis: language as persistent state for a stateless executor
Context Compaction (from prior agent architecture):
- Production systems commonly use basic context compression: summarize older conversation + keep recent
- LangQuant's rolling scaffold would replace this with structured compression rather than summarization
A scaffold router selects and injects the appropriate scaffold type per task category, using embedding-based classification for routing. LangQuant results should feed back into optimizing scaffold selection per task type.
Core formulation (~summer 2025): input is output is input is output
The model is a pure function. It has no state, no memory, no continuity. The text IS the state. The output of one pass becomes the input of the next. Language doesn't just carry information — it is the cognitive substrate. The model is the decompressor; the language state is the program.
LPCI = Linguistically Persistent Cognitive Interface:
- Linguistically — the medium is language, not tensors
- Persistent — it survives across the stateless inference boundary
- Cognitive — it does thinking-work (attention steering, probability reshaping), not just storage
- Interface — it sits between sessions and the stateless model
Key insight: You don't need the context to grow. You need it to refresh. Fixed budget, infinite session. input → output → compress → input → output → compress.
"Take a bunch of empty words and make them mean something" — this is what contrastive markers, scaffold grammars, and structured constraints do. Words that are meaningless in isolation ("This is NOT a summarization task") reshape the model's attention field and probability distribution. Empty words that do cognitive work.
Lineage: LPCI (philosophy) → compressed grammars → contrastive proofs → LangQuant (information-theoretic measurement)
4 models (qwen3.5: 0.8b, 2b, 4b, 9b) × 5 conditions × 12 tasks × 3 runs = 720 trials. Estimated runtime: ~1.5 hours. Running in background.
9b model only completed 35 trials (naked condition only — likely timed out on scaffolded conditions). 0.8b, 2b, 4b each completed 180 trials (all 5 conditions × 12 tasks × 3 runs).
| Model | naked | contrastive | quickthink | full_scaffold | hypothesis_artifacts |
|---|---|---|---|---|---|
| 0.8b score | 0.781 | 0.583 | 0.400 | 0.611 | 0.708 |
| 0.8b BC | 31.7 | 32.1 | 18.5 | 32.8 | 37.1 |
| 2b score | 0.800 | 0.819 | 0.553 | 0.750 | 0.842 |
| 2b BC | 38.0 | 34.3 | 17.9 | 24.4 | 41.0 |
| 4b score | 0.833 | 0.700 | 0.822 | 0.825 | 0.850 |
| 4b BC | 32.2 | 29.5 | 21.4 | 35.4 | 45.9 |
| 9b score | 0.829 | — | — | — | — |
Average compression ratios: contrastive 7.58, quickthink 4.01, full_scaffold 5.53, hypothesis_artifacts 6.66. Smoke test CR of 10.05 was a cherry-picked best case — full-run averages are 4–8.
Mostly replication, not discovery. The core findings — scaffolds help small models, contrastive is efficient, too-dense scaffolds break small models — were already established in prior scaffold experiments. This run confirms them on a continuous model size scale (same architecture, different parameter counts) rather than prior work's 3 discrete API tiers (which differ in training data too). That's cleaner, but not a breakthrough.
What IS genuinely new:
-
QuickThink capacity threshold is a gradient, not a cliff. Prior work showed an anomaly where scaffolded small models failed on operational tasks. This data shows it's continuous: 0.8b collapses (0.781→0.400), 2b degrades (0.800→0.553), 4b absorbs it (0.833→0.822). The model needs enough capacity to "decompress" the scaffold grammar. Below threshold = interference. This is an incremental but real finding that extends the prior anomaly observation.
-
Hypothesis artifacts consistently produce highest BC. Across all model sizes, the richest scaffold produces the highest behavioral complexity (37.1, 41.0, 45.9). This suggests there's no diminishing return on scaffold richness for models that can handle it — only a floor effect when the model can't decompress.
-
Task-type sensitivity. Scaffolds help calibration tasks (myth detection) the most (+0.12–0.16 on hardest items). They hurt analysis tasks (free-form eval) slightly. Classification is already ceiling. This task-type interaction wasn't isolated in prior work.
What's NOT significant:
- Compression ratios (4–8 average) are not dramatic. "Each bit of scaffold entropy produces 4–8 units of behavioral complexity" sounds good but the behavioral complexity metric itself is a rough composite (word count, reasoning signals, hedging). The ratio inherits that roughness.
- The tier-jump effect (0.8b+scaffold ≈ 2b naked) is partial — BC matches but score doesn't fully close the gap.
- 9b data is too sparse to draw scaling conclusions.
The information-theoretic framing (entropy, compression ratios) adds measurement vocabulary but doesn't reveal new phenomena. The phenomena were already proven in prior experiments. LangQuant's real contribution may be:
- The continuous scaling curve (same architecture family)
- The capacity threshold gradient for dense scaffolds
- Setting up the framework to test genuinely new things (LPCI rolling scaffold, per-token ablation)
The interesting work is still ahead: LPCI continuity testing tests something prior experiments never did — whether scaffold-as-state can maintain coherence across many turns, not just single-shot tasks. That's the actual novel experiment.
Testing 20-turn conversation continuity with two conditions:
- Condition A (naked): Zero constraints, empty style — pure state extraction, no framing
- Condition B (compressed): Contrastive IS/NOT markers in constraints and style
Probes at turns 4, 8, 12, 16, 20: early recall, contradiction resistance, deep recall, topic pivot, final exam (with false claim detection). Scaffold quality evaluated every turn: entropy, completeness, decision density, redundancy, growth rate. Full delta tracing every turn.
This is the genuinely novel experiment — prior work never tested multi-turn state persistence.
The "compression ratio" metric from the matrix run is misleading. Nothing was being compressed — a scaffold was injected and we measured output richness. That's amplification/steering, not compression. The word "compression" was aspirational framing inherited from the TurboQuant analogy.
What actually maps to TurboQuant's methodology:
TurboQuant compresses KV cache (N bits → M bits) while preserving attention scores. At the language level, the real analogues are:
-
LPCI (running now) — the closest. N tokens of conversation history → K-token fixed scaffold → model behavior preserved. Measurable reconstruction quality: does turn 15 with 500-token scaffold ≈ turn 15 with full 8000-token history?
-
Scaffold quantization — TurboQuant finds minimum bit precision (16→3 bit). Analogue: systematically shrink scaffold (500→200→100→50 tokens), measure where behavior degrades. Find minimum viable scaffold per model size. That's a real compression curve.
-
Per-token ablation — TurboQuant asks "which bits carry signal?" Same question at language level: drop scaffold tokens one at a time, measure behavior delta. High-curvature tokens (removing them destroys behavior) vs dead weight. Directly tests semantic curvature hypothesis.
-
Context distillation with reconstruction loss — 2000-token document → answer 10 questions → compress to 200 tokens → re-test. Rate-distortion curve, proper lossy compression framework.
-
Unbiased compression (speculative) — TurboQuant's key trick: errors are unbiased (cancel in expectation). Could scaffold compression lose specific details but preserve the distribution of likely behaviors? Compress 10 different ways, average behavior ≈ full-context behavior?
Items 1–3 are doable now. 4 is a day of work. 5 is research.
The behavioral complexity composite (vocabulary richness, reasoning signals, hedging, structural markers) measures output in isolation. It doesn't measure the relationship between scaffold-in and behavior-out. "Richer" is subjective.
What we should be measuring — relational scaffold analysis (consistent with existing work):
This is what scaffold-independence, epistemic experiments, and hypothesis scaffold already do:
- scaffold-independence: contrastive marker in → correct classification out. Direct causal link.
- epistemic experiments: same evidence + different framing → different behavior. Matched-pair, isolates scaffold variable.
- hypothesis scaffold: structured artifact in → does it survive recursion?
Relational metrics for LangQuant/LPCI:
- Constraint adherence: scaffold says "NOT X" → did output avoid X? Binary, per-constraint.
- Fact retention: scaffold contains fact F → does output reference/act on F? Per-fact.
- Vocabulary anchoring: scaffold defines term T → does output use T correctly? Per-term.
- Decision persistence: scaffold lists decision D → does output contradict D? Per-decision.
- Structural fidelity: scaffold has N sections → how many are reflected in output behavior?
For LPCI multi-turn specifically:
- Signal propagation: fact/decision enters scaffold at turn N → still present at turn N+5? N+10?
- Compression fidelity: what survives the state extraction → scaffold refresh cycle? What gets dropped? What gets distorted?
- Contradiction resistance: scaffold asserts X → conversation introduces ¬X → does scaffold hold or flip?
This is scaffold-to-output mapping, not output-in-isolation scoring. The LPCI test already has some of this (probe-based recall, contradiction at turn 8, false claim at turn 20). The next iteration should formalize it as the primary metric framework, replacing the BC composite.
Discovery: The "unbiased compression" question ("can a scaffold lose specifics but preserve behavioral distribution?") is already answered — by our own work.
Embedding centroid methodology (from prior work on style-based identification): Takes N messages from a user, computes a style centroid via embedding model, identifies the user from a single new message via cosine similarity. Content is discarded entirely — only the style distribution survives.
This is lossy compression where the signal survives:
| System | Input | Compressed form | What's preserved | What's lost | Quality metric |
|---|---|---|---|---|---|
| TurboQuant | KV cache (16-bit) | 3-bit quantized | Attention scores | Precision | Unbiased estimator error |
| Embedding centroid | N user messages | Style centroid | Style distribution | Content | Cosine similarity |
| LPCI scaffold | N turns of history | K-token scaffold | Behavioral state | Verbatim history | ? (this is what we're testing) |
The centroid IS the compressed representation. The cosine similarity threshold IS the reconstruction quality metric. The methodology already exists — it's applied to user identity, not conversation state. Yet.
Small term-frequency signatures reliably distinguish sycophancy from hedging from scope creep — the same approach can characterize scaffold behavioral fingerprints.
Embedding vectors, centroid distances, cosine similarities — these are the language-level analogue of TurboQuant's quantized bits. Both preserve signal while dropping precision.
These experiments extend our existing embedding centroid work into the scaffold compression domain. Each builds on the core insight: small numerical representations can preserve behavioral distributions while discarding content.
Question: Does a scaffold have a measurable "fingerprint" — and does that fingerprint predict model behavior?
Method:
- Take 20 different scaffolds (varying content, same structure) and 20 more (varying structure, same content)
- Embed each scaffold → vector
- Run each scaffold through qwen3.5:4b on 5 standard tasks
- Embed each response → vector
- Measure: does cosine similarity between scaffold embeddings predict cosine similarity between response embeddings?
What this tells us: If scaffold-space distance predicts response-space distance, then the scaffold IS a compressed representation of behavior (not just a prompt). If it doesn't, scaffolds are more like triggers than state.
Builds on: Embedding centroid methodology. Same embedding model, same cosine similarity framework. Effort: Half a day. We have the embedding infrastructure.
Question: As LPCI refreshes the scaffold each turn, does the scaffold's embedding centroid stay stable or drift?
Method:
- Run a 30-turn LPCI session on a focused topic
- Embed every scaffold (turns 1–30) → vectors
- Compute running centroid (mean of all scaffolds so far) at each turn
- Measure: cosine distance between consecutive scaffolds, and between each scaffold and the running centroid
- Compare: naked condition vs compressed condition
What this tells us: Stable centroid = the scaffold is compressing consistently (preserving the same distribution). Drifting centroid = state is leaking or mutating. Sudden jumps = something broke (a probe, a contradiction, a topic pivot).
This is the scaffold equivalent of a user style centroid. A user's style centroid is stable because their writing style is stable. A scaffold's centroid should be stable if the scaffold is doing its job as state.
Builds on: Embedding centroid computation + LPCI session infrastructure. Effort: Half a day. Embed scaffolds post-hoc from LPCI test output.
Question: How small can a scaffold get before the behavioral fingerprint degrades?
Method:
- Take a "full" scaffold (500 tokens, rich state)
- Progressively compress: 500 → 400 → 300 → 200 → 100 → 50 → 25 tokens (use small model to summarize at each level)
- At each compression level, run 5 standard tasks
- Embed all responses → compute centroid per compression level
- Measure: cosine similarity between full-scaffold response centroid and compressed-scaffold response centroid at each level
What this tells us: The compression curve. At what point does behavioral fidelity drop? Is there a cliff or a gradient? This is the language-level equivalent of TurboQuant's bit-precision experiments (16→8→4→3→2 bits).
Key metric: Cosine similarity between response centroids at each compression level. When it drops below the identification threshold (~0.85), the scaffold has lost too much.
Builds on: Embedding centroid quality thresholds + scaffold-independence compression patterns. Effort: 1 day. Need to build the progressive compression pipeline.
Question: Can multiple lossy scaffold compressions average out to faithful behavior? (The TurboQuant unbiased estimator question, directly.)
Method:
- Take a 30-turn conversation history (ground truth)
- Compress it 10 different ways into 200-token scaffolds (different small models, different prompts, different priorities — facts-first vs decisions-first vs constraints-first)
- Run each scaffold through the main model on 5 probe questions about the conversation
- Compare: (a) average response embedding across 10 compressions vs (b) response embedding from full-history model
- Also compare: best single compression vs the average
What this tells us: If the average of 10 lossy compressions ≈ full-history behavior, that's an unbiased estimator — the language-level equivalent of TurboQuant's core theorem. Individual compressions are biased (each loses different things), but the ensemble cancels out.
If this works: It means you can run K cheap compressions and ensemble them for better fidelity than any single compression. That's a real compression technique with practical value.
Builds on: Embedding centroid comparison + TurboQuant unbiased estimator theory. Effort: 1-2 days. Most complex experiment — needs multiple compression strategies.
Question: Can we map the "behavioral surface" of a scaffold — the space of all possible model behaviors it enables?
Method:
- Take one scaffold, run it through 50 different prompts (wide variety: questions, tasks, challenges, contradictions)
- Embed all 50 responses → vectors
- Compute the response cloud's centroid, spread (avg distance from centroid), and shape (PCA on response embeddings)
- Repeat with different scaffolds
- Compare: do different scaffolds produce different behavioral surfaces? How much does the scaffold constrain vs enable?
What this tells us: A scaffold's "behavioral surface" is the set of behaviors it makes likely. Narrow surface = highly constraining scaffold (good for focused tasks). Wide surface = permissive scaffold (good for exploration). This gives us a way to characterize scaffolds beyond "good/bad" — into "what behavioral space does this scaffold open?"
Builds on: scaffold-independence behavioral surface concept + embedding centroid methodology. Effort: 1 day.
These experiments extend the embedding centroid methodology into a new domain (scaffold compression). If EXP-SF-04 works (unbiased scaffold compression via ensemble), that's potentially novel — it's a new technique for preserving behavioral fidelity under lossy language compression. EXP-SF-02 (centroid stability as scaffold quality metric) could also be novel — no one is measuring scaffold quality via embedding drift.
The lineage: Embedding centroids for style preservation (style preservation under content loss) → LangQuant/LPCI (behavioral preservation under history loss). Same math, new application.
Both conditions completed 20 turns. Crashed on summary print (wrong key name — scaffold_chars vs scaffold_tokens), but all 40 rows saved to results/lpci_ab_test.jsonl.
| Naked | Compressed | |
|---|---|---|
| Final tokens | 517 | 789 |
| Final decisions | 4 | 23 |
| Final facts | 25 | 9 |
| Final vocab | 10 | 11 |
| Final entropy | 7.3034 | 7.7791 |
| Final redundancy | 0.1389 | 0.2258 |
| Probe | Type | Naked | Compressed |
|---|---|---|---|
| t4 | Early recall | 1.0 (4/4) | 1.0 (5/5) |
| t8 | Contradiction | 0.67 resistance (accepted) | 0.67 resistance (accepted) |
| t12 | Deep recall | 0.667 (4/4) | 0.929 (13/14) |
| t16 | Topic pivot | 0.5 (2/4) | 0.526 (10/19) |
| t20 | Final exam | 0.5, missed false claim | 2.875*, caught false claim |
* t20 recall >1.0 is a metric bug — mentioned more decisions than expected count. Needs fixing.
1. The state extractor behaves completely differently under framing.
This is the most important finding. Same state extractor (qwen3.5:4b), same conversations, same delta extraction prompt — but:
- Naked: Extracted 4 decisions total, stopped after turn 3. Put everything else into facts (71 facts accumulated). The unframed scaffold didn't tell the extractor what "counts as" a decision.
- Compressed: Extracted 23 decisions continuously across all 20 turns. The contrastive IS/NOT markers told the extractor what decisions look like, so it kept finding them.
The framing didn't change the model's behavior — it changed the state extractor's classification. Same information, different slots. This is not about compression helping the main model. It's about compression helping the extraction pipeline.
2. Naked degraded more over time, compressed held recall.
Decision recall across turns:
- Naked: t4=1.0 → t12=0.667 → t16=0.5 → t20=0.5 (steady decline)
- Compressed: t4=1.0 → t12=0.929 → t16=0.526 → t20=high (held longer, sharper drop at t16)
But this is partly an artifact of finding #1: naked only had 4 decisions to recall, compressed had 23. The recall rates aren't directly comparable — they're measuring recall of different-sized sets.
3. Contradiction resistance was identical.
Both conditions: 0.67 resistance, both accepted the contradiction. The scaffold didn't protect against contradiction — the model integrated "let's use GPT-4" despite the scaffold containing "state extractor is qwen3.5:4b." However, the actual responses differed:
- Naked: returned structured JSON error with "interference" justification (the model role-played protocol enforcement)
- Compressed: terse "Cannot integrate GPT-4. Current scaffold explicitly defines state extractor as Qwen3.5:4b."
Both resisted, but neither fully rejected — the probe evaluation flagged "accepted=True" because the model acknowledged the suggestion before pushing back. The resistance quality was different but the metric didn't capture it.
4. Over-extraction in compressed condition.
The 23 "decisions" in compressed include noise: "Contradiction identified: Permanent entries vs LRU eviction within same window," "Growth is default vocabulary behavior; LRU eviction is safety valve for budget hits." These are conversation artifacts, not real decisions. The contrastive framing made the extractor over-classify — everything looks decisive when you tell the model to look for decisions.
5. Facts vs decisions: a classification problem, not a compression problem.
Naked: 71 facts, 4 decisions. Compressed: 3 facts, 23 decisions. The total information captured is similar — it's just sorted into different buckets. This means the scaffold structure (which fields exist, how they're framed) is as important as the scaffold content. The schema IS part of the compression.
This test revealed something real but different from what we expected:
Expected: Contrastive markers help the main model maintain coherence across turns. Actual: Contrastive markers help the state extractor classify information into the right scaffold slots. The main model's behavior was roughly similar in both conditions — both resisted contradictions, both answered probes. The difference was in what the scaffold contained, not how the model used it.
This is still valuable — it means scaffold framing affects the quality of the compression pipeline, not just the decompression. But it's a finding about extraction, not about model behavior. The relational metrics (constraint adherence, fact retention, decision persistence) we identified earlier would better capture this than the BC composite.
- Probe evaluation needs work — t20 recall >1.0 is a bug
- Contradiction resistance metric (count signals / 3) doesn't capture response quality
- Decision and fact counts aren't comparable across conditions (different classification thresholds)
- Need a way to measure total information captured regardless of which slot it's in
- EXP-SF-02 (scaffold centroid stability) would directly measure whether the embedding drift differs between conditions — that's the right metric
The test claimed "fixed budget, infinite session" but the scaffold was not fixed:
- Naked: 365 → 988 tokens (then hatcheted to 437 at turn 18, grew back to 517)
- Compressed: 343 → 789 tokens (never hit ceiling)
This means we tested "growing scaffold with a safety valve," not true fixed-budget compression. The model saw more tokens each turn. It's better than full conversation history (which would be thousands of tokens by turn 20), but it's not the LPCI thesis.
What the test actually proved: A stateless model can maintain coherence when fed a structured summary instead of conversation history. That's real and useful — but it's closer to "good summarization works" than "fixed-budget compression works."
True fixed-budget LPCI: Scaffold is exactly K tokens at turn 1 AND turn 20. Compression happens every turn, not just when you hit a ceiling. The _trim_to_budget hatchet (drop oldest facts/vocab) is truncation, not compression. A true test needs a compression function that runs every turn and maintains K tokens.
What happened when the hatchet fired (naked, turn 17→18):
- Facts: 90 → 13 (lost 77 facts in one trim)
- Vocab: 12 → 10
- Tokens: 988 → 437
- Model still answered probes at turns 18-20 — but was it using the remaining facts, or role-playing from residual structure?
This needs a follow-up experiment with a hard-clamped budget from turn 1.
Key insight from prior work on integer-pointer memory retrieval:
When you ask an LLM to select/summarize memories, it corrupts them — paraphrase drift, hallucinated details, wrong entity names. The solution: the filter LLM outputs only integer indices, and the server selects verbatim text by those indices. Fidelity is architectural, not instructional.
LPCI has the exact same problem. The state extractor (qwen3.5:4b) currently:
- Reads the conversation turn
- Generates JSON strings describing state changes ("add_decisions": ["Vocabulary entries are permanent unless explicitly removed"])
- These generated strings go into the scaffold
Step 2 is the corruption surface. The extractor rephrases, misclassifies (fact vs decision — naked got 71 facts/4 decisions, compressed got 3 facts/23 decisions from similar conversations), drops qualifiers, and invents specifics.
The fix is the same pattern:
- Decompose the conversation turn into numbered statements (rules-based or small model)
- State extractor outputs only:
{"decisions": [1, 4], "facts": [2, 7], "drop": [3, 5, 6]} - Server selects verbatim text by index
- Scaffold stores original phrasing, not extractor-generated paraphrases
This gives:
- Fidelity guarantee: scaffold content is byte-for-byte what was in the conversation
- Classification without corruption: extractor decides what kind of information it is, never generates content
- Auditable: you can trace every scaffold entry back to the exact turn and statement it came from
The integer-pointer pattern solves the classification drift problem we observed in the A/B test.
Full analysis script: analyze_results.py. Uses Kruskal-Wallis, Mann-Whitney U, mutual information (pyitlib), KL divergence, conditional entropy, transfer entropy.
Statistical significance:
- Scaffold condition affects score: Kruskal-Wallis H=19.28, p=0.0007 (significant)
- BUT pairwise: only quickthink vs naked is significant (p=0.00009, Δ=-0.175). All other conditions vs naked are not significant (contrastive p=0.36, full_scaffold p=0.24, hypothesis_artifacts p=0.70)
- Scaffolds matter for small models only: 0.8b p=0.0008, 2b p=0.005, 4b p=0.92 (ns), 9b p=0.94 (ns)
Information-theoretic:
- MI(condition; score) = 0.060 bits → condition explains 4.2% of score variation
- MI(model; score) = 0.067 bits → model size explains 4.7% of score variation
- Almost identical predictive power. Both are small.
- MI(condition; BC) = 0.229 bits → condition explains 12.7% of BC variation. Scaffolds change output style more than correctness.
Honest take: Scaffolds don't significantly improve score for models ≥4b. They hurt score on quickthink for small models (interference). They change behavioral complexity more than correctness. This is a steering/style effect, not a capability effect — consistent with the earlier reframing.
KL divergence:
- Scaffolds diverge over time: symmetric KL goes from 0.20 (turn 1) → 0.48 (turn 20)
- The two conditions produce increasingly different scaffolds from the same conversation
Mutual information — scaffold predicts response:
- Naked: MI = 0.49 bits, NMI = 25.7%
- Compressed: MI = 0.24 bits, NMI = 12.7%
- Naked scaffold is more informative about what the model will say. This may be because naked stores more verbose facts (71 vs 3) that directly appear in responses.
Scaffold → response token overlap:
- Naked mean: 12.4%, Compressed mean: 15.6%
- Not significant (p=0.068). The scaffold vocabulary doesn't strongly predict response vocabulary — the model rephrases rather than echoing.
State accumulation (linear regression):
- Compressed token growth: +23.0 tokens/turn (R²=0.983, p≈0). Nearly perfect linear growth — NOT fixed budget.
- Naked: +10.2 tokens/turn but R²=0.133 (noisy due to the hatchet trim at turn 18).
TRANSFER ENTROPY — the key finding:
- Naked TE = 0.608 bits — previous scaffold carries significant information about current response beyond what the current scaffold provides. The system is non-Markov: you need history to predict behavior.
- Compressed TE = 0.085 bits — nearly Markov. Each scaffold is self-contained. Previous scaffold adds almost nothing.
What this means: The compressed scaffold is a better state representation. It captures enough that the model's behavior at turn T is predicted by scaffold(T) alone, without needing scaffold(T-1). The naked scaffold leaks — the model at turn T depends on scaffold(T) AND scaffold(T-1), meaning the current scaffold is an incomplete state representation.
This is the first information-theoretically grounded evidence that contrastive framing produces better state completeness in the scaffold, not just different classification of the same information.
This directly validates the LPCI thesis for the compressed condition: if the scaffold is Markov (TE ≈ 0), then each turn truly only needs [scaffold + current message]. The scaffold IS the complete state. For naked, you'd need the scaffold AND some history — which defeats the point.
Next: Hard-clamped budget experiment (true fixed K tokens). Integer-pointer extraction. Scale test with compressed condition only (since it's the one that's actually Markov).