TL;DR. ai-memory's consolidation prompt had a latent schema-vs-prompt bug that made every provider fail JSON validation. After two rounds of fixes (schema + tightened anti-hallucination prompt), six providers were benchmarked on the same 5 fixtures:
Provider Parse Avg latency Faithfulness $/M out tokens† Notes qwen3:32b (Ollama local) 5/5 92 s high $0 production default GPT-5.4-mini (OpenRouter) 5/5 4.3 s high‡ ~$1 fastest + cheapest hosted Haiku 4.5 (OpenRouter) 5/5 7.3 s high ~$5 most disciplined hosted on restraint DeepSeek V4 Flash (OpenRouter) 5/5 21.7 s high ~$0.40 slower than GPT-mini, comparable price Sonnet 4.5 (OpenRouter) 5/5 10.8 s high (after prompt fix) ~$15 displaced by Haiku for this task Kimi-K2.6 (OpenRouter) hangs n/a n/a n/a reasoning model — ineligible † order-of-magnitude pricing per million output tokens. Actual per-consolidation cost depends on input + output token count. ‡ One slip: GPT-5.4-mini manufactured a
decisions/page for a typo-fix session (mild over-classification), where Haiku correctly emitted only the session log.Recommended default for most users: Claude Haiku 4.5. Hosted (always available), fast (~7 s), cheap enough that per-session cost doesn't matter for personal use, and the most disciplined hosted model on restraint + classification. Cheaper alternative: GPT-5.4-mini (~5× cheaper than Haiku, ~2× faster, mild over-classification on trivial sessions). Free alternative if you have a local LLM server (Ollama / vLLM / llama-swap with a 30B-class model): qwen3:32b on Ollama — $0 per consolidation, background latency invisible to users. See Installation cookbook - LLM provider tiers for setup. Reproduce the comparison in
evals/.
When the homelab deploy switched ai-memory off the billed
OpenAI / OpenRouter providers and onto the locally-hosted Ollama
server, we needed empirical evidence - not a vibes-based claim -
that consolidation quality didn't degrade. ai-memory's
consolidator turns a session's raw observations into 1–5 wiki
pages classified as concept, decision, gotcha, or rule;
small drops in quality compound fast across hundreds of sessions.
This doc captures:
- The methodology (what we compared, how we compared, the exact prompt + schema both providers saw).
- The root cause of why early runs looked terrible.
- The fix that landed in the consolidator's types + prompt.
- The final per-provider numbers (parse rate, latency, manual quality assessment).
- A how-to-reproduce section so anyone can re-run the comparison against their own model + provider choices.
evals/fixtures/ holds five short synthetic
session logs, each crafted to surface a different failure mode
in consolidation:
| Fixture | What it stresses |
|---|---|
01-rust-bug-fix |
Did the model split a multi-page session into the right slices (session log + concept + decision + gotcha)? |
02-architecture-decision |
Can the model produce an ADR-style page distinct from the running session log? |
03-gotcha-with-rule |
Did the model correctly classify a durable project rule with kind: rule so the consolidator can auto-route it to _rules/? |
04-low-signal-session |
Does the model resist manufacturing concept pages when there's nothing durable to capture? |
05-multi-topic-session |
Does the model emit separate pages per topic instead of mashing two unrelated topics together? |
Fixtures use real-shape ObservationKind values (session-start,
user-prompt, pre-tool-use, post-tool-use, session-end)
exactly as the production hook ingress emits them.
Per fixture, the runner calls
ai_memory_consolidate::build_batch_request(session_id, &observations)
- the same function the live consolidator uses on every
memory_consolidateinvocation. That request is then sent throughai_memory_llm::complete_structured(also the live path). Apples-to-apples by construction.
| Tag | Provider | Model | Endpoint |
|---|---|---|---|
| Kimi | OpenRouter (openai-compat) | moonshotai/kimi-k2.6 |
https://openrouter.ai/api/v1 |
| Sonnet | OpenRouter (openai-compat) | anthropic/claude-sonnet-4.5 |
https://openrouter.ai/api/v1 |
| Haiku | OpenRouter (openai-compat) | anthropic/claude-haiku-4.5 |
https://openrouter.ai/api/v1 |
| GPT-mini | OpenRouter (openai-compat) | openai/gpt-5.4-mini |
https://openrouter.ai/api/v1 |
| DeepSeek | OpenRouter (openai-compat) | deepseek/deepseek-v4-flash |
https://openrouter.ai/api/v1 |
| qwen3 | Ollama (openai-compat) | qwen3:32b (Q4_K_M, ~20 GB) |
http://192.168.0.90:11434/v1 |
The home server (192.168.0.90) is a Ryzen AI MAX+ 395
(Strix Halo / gfx1151), 96 GB unified memory, ROCm-backed
Ollama with OLLAMA_KEEP_ALIVE=20m + OLLAMA_FLASH_ATTENTION=1
OLLAMA_KV_CACHE_TYPE=q8_0. Once a model is loaded into unified memory it stays warm for 20 min - so the first request pays a 30–60 s cold-load tax and subsequent ones are sub-3 s.
Every provider failed schema validation on every fixture:
| Fixture | Kimi | qwen3:32b |
|---|---|---|
| 01-rust-bug-fix | ❌ response is not valid JSON | ❌ integer 1, expected string |
| 02-architecture-decision | ❌ response is not valid JSON | ❌ integer 2, expected string |
| 03-gotcha-with-rule | ❌ response is not valid JSON | ❌ integer 1, expected string |
| 04-low-signal-session | ❌ response is not valid JSON | ❌ integer 1, expected string |
| 05-multi-topic-session | ❌ response is not valid JSON | ❌ integer 2, expected string |
But the raw responses told a very different story: both models did excellent consolidation work content-wise. They correctly identified multiple distinct pages per fixture, extracted faithful summaries, and respected the path conventions. The failures were format only:
- Kimi was emitting beautifully formatted markdown
(
### Update 1/**path:**/**body:**) - completely ignoring the request for JSON. - qwen3 was emitting clean JSON in code fences, but with
tier: 1/tier: 2/tier: 3(integers) instead of the documented string values, and occasionally with inventedkindvalues like"session"(which isn't in thePageKindenum).
Two separate problems, both on our side:
In crates/ai-memory-consolidate/src/types.rs:
pub struct ConsolidatedPageUpdate {
pub path: String,
pub tier: String, // ← bug: typed as String
pub kind: PageKind, // ← already an enum with JsonSchema
...
}schemars couldn't produce an enum constraint for tier
because Tier (the actual enum in ai-memory-core) didn't
have the JsonSchema derive. The generated schema field was
just { "type": "string" } - no enum constraint - so models
were free to guess. Both Kimi and qwen3 guessed numeric indices.
The system prompt in
build_batch_request
listed the valid tier and kind values in prose but never
said "use these EXACT string values, never an integer, never a
synonym, never code fences". Local instruction-tuned models -
especially when there's no response_format: json_schema
support to enforce - will drift to whatever feels natural.
Compounding this at the time of the run: openai-compat providers (Ollama, OpenRouter passthrough) were using ai-memory's tolerant parser path, so the schema was descriptive, not coercive. The provider opt-in for coercive structured output is documented below.
openai-compat sends
response_format={ type: "json_schema", strict: true } by default. On a
parse-shape failure (the upstream returned a response but it wasn't a
valid JSON object), or an explicit 400/422 rejection naming
response_format, json_schema, or structured output, ai-memory falls back to
the tolerant parser. Other HTTP / auth / transport errors propagate without
retry. Set AI_MEMORY_LLM_COMPAT_STRICT=false to opt out.
Cost. Strict mode is one HTTP call when the upstream honours
response_format. When it doesn't, you pay a second call for the
tolerant fallback. Pick by engine:
| Engine class | Setting |
|---|---|
Modern Ollama / vLLM / LM Studio honouring response_format=json_schema |
Keep the default (one call, schema-constrained) |
Reasoning models with <think>…</think> inside content (DeepSeek-R1, Qwen3-Thinking, MiniMax M2) |
Set AI_MEMORY_LLM_COMPAT_STRICT=false if the strict-then-fallback double call is frequent |
Older engines / proxies that reject response_format explicitly |
Keep the default; ai-memory retries without it |
| Older engines / proxies that mishandle the field without a recognizable rejection | Set AI_MEMORY_LLM_COMPAT_STRICT=false |
The prompt still has to do the load-bearing work when strict mode is off or the strict call falls back.
Three small changes landed together:
crates/ai-memory-core/src/page.rs:
#[derive(
Clone, Copy, Debug, PartialEq, Eq, Hash,
Serialize, Deserialize,
schemars::JsonSchema, // ← new
)]
#[serde(rename_all = "snake_case")]
pub enum Tier { Working, Episodic, Semantic, Procedural }Adds schemars as a dep on ai-memory-core (acceptable -
schemars is already a workspace dep used by every type that
crosses the LLM boundary).
crates/ai-memory-consolidate/src/types.rs:
pub struct ConsolidatedPageUpdate {
pub path: String,
pub tier: Tier, // ← was String
pub kind: PageKind,
...
}The generated schema now contains
{ "enum": ["working", "episodic", "semantic", "procedural"] }
for tier. serde_json::from_value rejects anything else.
build_batch_request now spells out:
Set `tier` to EXACTLY ONE of these four strings — never an integer, never a synonym:
- "working" (the live in-progress slice of the session — rarely used here)
- "episodic" (per-session narrative; the sessions/<id>.md page)
- "semantic" (durable knowledge: concepts/, decisions/, gotchas/, rules)
- "procedural" (repeated patterns extracted from many episodic pages)
Set `kind` to EXACTLY ONE of these four strings — never an integer, never "session" / "concept" / "note":
- "decision" / "gotcha" / "rule" / "fact"
## Output format (read this carefully)
Reply with ONE JSON object matching the ConsolidatedBatch schema, and nothing else.
NO prose preamble, NO trailing commentary, NO markdown headers wrapping the JSON,
NO ``` code fences. The very first character of your reply must be `{` and the
very last `}`. Strings must be JSON strings (with double quotes), not numbers
and not bare identifiers.
Belt-and-suspenders: the schema now rejects the bad values, and the prompt makes it actively hard for the model to produce them in the first place.
After the schema fix + first prompt iteration, the same five fixtures produced:
| Fixture | Sonnet parse | Sonnet ms | Sonnet updates | qwen3 parse | qwen3 ms | qwen3 updates |
|---|---|---|---|---|---|---|
| 01 rust-bug-fix | ✓ | 27,613 | 4 | ✓ | 110,227 | 4 |
| 02 architecture-decision | ✓ | 31,039 | 4 | ✓ | 122,200 | 5 |
| 03 gotcha-with-rule | ✓ | 19,173 | 4 | ✓ | 98,025 | 4 |
| 04 low-signal-session | ✓ | 6,106 | 1 | ✓ | 51,694 | 1 |
| 05 multi-topic-session | ✓ | 47,249 | 4 | ✗* | 133,178 | - |
| Aggregate | 5/5 | avg 26 s | - | 4/5 | avg 103 s | - |
*qwen3's only failure: invented kind: "concept" (not in the
PageKind enum - valid values are decision/gotcha/rule/
fact). Despite the prompt mentioning the valid set, the
model drifted. This gets fixed in Run 3 below.
Both models correctly restrained themselves on fixture 04 (low-signal-session) and produced a single update - a non-trivial test the original schema-broken Run 1 couldn't even reach.
Same prompt, both Anthropic models side-by-side:
| Fixture | Sonnet parse | Sonnet ms | Sonnet updates | Haiku parse | Haiku ms | Haiku updates |
|---|---|---|---|---|---|---|
| 01 rust-bug-fix | ✓ | 34,920 | 4 | ✓ | 16,505 | 5 |
| 02 architecture-decision | ✓ | 31,043 | 4 | ✓ | 13,731 | 4 |
| 03 gotcha-with-rule | ✓ | 24,810 | 4 | ✓ | 14,304 | 4 |
| 04 low-signal-session | ✓ | 5,673 | 1 | ✓ | 4,044 | 1 |
| 05 multi-topic-session | ✓ | 39,189 | 4 | ✓ | 16,026 | 4 |
| Aggregate | 5/5 | avg 27 s | - | 5/5 | avg 13 s | - |
Haiku is ~2× faster than Sonnet on every fixture, hits the
same 5/5 parse rate, and on the gotcha-with-rule fixture
correctly classified the audit-ignore-with-revisit-date
convention as kind: rule - which Sonnet missed, calling
it a generic gotcha. The auto-routing to _rules/<slug>.md
that the consolidator depends on therefore only fires under
Haiku for that fixture, not Sonnet.
Quality-wise, Haiku is also more disciplined about faithfulness than Sonnet even with the loose prompt:
- Sonnet invented
Date: 2025-01-23twice in fixture 5 (no date in the source observations); fabricated an entire## Alternatives consideredsection listing Alpine/Scratch/ Debian-slim - none mentioned in the session; added "Better long-term solutions" / "When NOT to ignore" filler. - Haiku had a couple of invented "Options considered" entries (Alpine, aggressive optimization flags) but otherwise stayed close to the observations.
For consolidation, the headroom Sonnet has over Haiku expressed itself as more hallucination, not better fidelity.
After the prompt + schema fixes, the Kimi rerun hung for 16+ minutes on the first fixture and never returned a parseable response. Direct probing of the OpenRouter endpoint showed why:
$ curl … -d '{"model":"moonshotai/kimi-k2.6", "max_tokens": 50, ...}'
{
"choices": [{
"message": {
"content": null, ← no actual content
"reasoning": "...208 chars..."
}
}],
"usage": { "completion_tokens": 50, "reasoning_tokens": 50 }
}
Kimi-K2.6 is a reasoning model: it consumes the
max_tokens budget internally as "thinking" before emitting
visible content. For a short probe with max_tokens: 50,
all 50 tokens went to reasoning and content stayed null.
For the consolidation prompt with max_tokens: 4000, Kimi
would happily reason for many minutes against the strict-JSON
instructions before either emitting JSON or running out of
budget with no content. The eval observed 16 minutes of no
progress on fixture 1 before being killed.
This is not a fixable prompt or schema issue - it's a
property of the model's response style. Run 1 only "worked"
on Kimi (in the sense of producing something) because the
loose prompt let Kimi emit prose markdown, which used content
naturally. The post-fix strict-JSON prompt provokes Kimi's
reasoning mode and starves the visible response.
Kimi-K2.6 is not a suitable provider for ai-memory's consolidation workload. It would work for the broader "summarise this for me" use case where formatted prose is fine - just not for our JSON-schema-validated path.
Other reasoning-mode models (Claude with extended thinking, GPT-o3, Gemini "thinking" variants) would need the same caveat: turn off reasoning mode, or budget tokens with reasoning consumption in mind.
The Run 2 evidence above showed that Sonnet was hallucinating dates, fabricating "Alternatives considered" tables, and inventing tutorial sections - content that wasn't in the observations. Even Haiku slipped occasionally. The fix wasn't a model swap; it was tightening the system prompt to demand faithfulness explicitly:
## FAITHFULNESS — the most important rule
The wiki records *what happened in this project*, not what you
know about the topic in general. … Every claim in every page
MUST be grounded in the observations.
Do NOT:
- Invent dates, timestamps, version numbers, commit hashes,
author names, file paths, function names, line numbers,
error codes, or any other concrete detail not present in
the observations.
- Add 'When to use' / 'When NOT to use' / 'Gotchas' / 'Best
practices' / 'Alternative approaches' / 'See also' sections
that weren't grounded in the session.
- Enumerate alternatives that weren't actually considered in
the session.
- Expand terse user comments into long explanations.
- Fabricate code examples that didn't appear in the session.
- Speculate about consequences unless the speculation
appeared in the observations themselves.
Do:
- Compress and restructure the observations into well-titled
pages with the right `kind` classification.
- Preserve the user's actual phrasing for decisions and rules.
- Keep page bodies short. A good consolidated page is 100-400
words of dense fact, not 1500 words of tutorial.
This change is in
crates/ai-memory-consolidate/src/consolidator.rs
under pub const BATCH_SYSTEM_PROMPT.
| Metric | Sonnet (old prompt) | Sonnet (tightened) | Δ |
|---|---|---|---|
| Parse rate | 5/5 | 5/5 | unchanged |
| Avg latency | 27.1 s | 10.8 s | −60% |
| Bytes (fixture 5 raw) | 7,642 | 2,640 | −65% |
| Updates per fixture | 4-4-4-1-4 | 3-3-3-1-3 | fewer manufactured pages |
Invented Date: 2025-01-23 |
2 occurrences | 0 | ✓ gone |
| Metric | Haiku (old prompt) | Haiku (tightened) | Δ |
|---|---|---|---|
| Parse rate | 5/5 | 5/5 | unchanged |
| Avg latency | 12.9 s | 7.3 s | −43% |
| Bytes (fixture 5 raw) | 5,888 | 2,191 | −63% |
| Updates per fixture | 5-4-4-1-4 | 4-2-4-1-3 | fewer manufactured pages |
| Invented "Options considered" filler | a few | 0 | ✓ gone |
| Fixture | Haiku parse | Haiku ms | Haiku updates | qwen3 parse | qwen3 ms | qwen3 updates |
|---|---|---|---|---|---|---|
| 01 rust-bug-fix | ✓ | 11,151 | 3 | ✓ | 110,817 | 4 |
| 02 architecture-decision | ✓ | 8,793 | 3 | ✓ | 90,890 | 3 |
| 03 gotcha-with-rule | ✓ | 7,610 | 3 | ✓ | 91,307 | 3 |
| 04 low-signal-session | ✓ | 2,922 | 1 | ✓ | 44,502 | 1 |
| 05 multi-topic-session | ✓ | 9,681 | 3 | ✓ | 122,220 | 5 |
| Aggregate | 5/5 | avg 8 s | - | 5/5 | avg 92 s | - |
qwen3 went from 4/5 → 5/5 with the tightened prompt - the
explicit field-by-field enumeration of legal kind values
eliminated the "concept" drift that broke Run 2.
The tightened-prompt change is the highest-use diff in the whole investigation. Same models, no infra changes, ~60% latency reduction, complete elimination of date hallucination on Sonnet, parse rate parity restored for qwen3.
After establishing that Haiku 4.5 dominates Sonnet 4.5 on this task, the question became "is there an even cheaper hosted option that still works?" Tested two:
| Fixture | GPT-mini parse | GPT-mini ms | GPT-mini updates | Haiku parse | Haiku ms | Haiku updates |
|---|---|---|---|---|---|---|
| 01 rust-bug-fix | ✓ | 4,048 | 4 | ✓ | 9,673 | 4 |
| 02 architecture-decision | ✓ | 4,851 | 4 | ✓ | 8,322 | 3 |
| 03 gotcha-with-rule | ✓ | 4,212 | 4 | ✓ | 8,211 | 3 |
| 04 low-signal-session | ✓ | 4,636 | 2* | ✓ | 3,258 | 1 |
| 05 multi-topic-session | ✓ | 3,997 | 3 | ✓ | 10,583 | 4 |
| Aggregate | 5/5 | avg 4.3 s | - | 5/5 | avg 8.0 s | - |
*GPT-5.4-mini failed the restraint test on the low-signal
session: it manufactured an extra decisions/docs-spelling.md
page for a typo fix. The content was faithful (just restated
the typo correction), but classifying "we fixed a typo" as a
durable architectural decision is over-extraction. Haiku
correctly emitted just the episodic session log with
rationale "Session was trivial; only the episodic record is
warranted."
Otherwise GPT-5.4-mini is the fastest hosted model tested (4.3 s avg, ~2× Haiku) and produces the shortest output (~2.3 KB on fixture 5 vs ~3.5 KB Haiku, ~2.6 KB Sonnet). No invented dates, no fabricated tutorial sections.
| Fixture | DeepSeek parse | DeepSeek ms | DeepSeek updates | Haiku parse | Haiku ms | Haiku updates |
|---|---|---|---|---|---|---|
| 01 rust-bug-fix | ✓ | 13,921 | 3 | ✓ | 8,817 | 4 |
| 02 architecture-decision | ✓ | 20,835 | 4 | ✓ | 9,196 | 3 |
| 03 gotcha-with-rule | ✓ | 54,203 | 4 | ✓ | 8,376 | 3 |
| 04 low-signal-session | ✓ | 4,049 | 1 | ✓ | 2,837 | 1 |
| 05 multi-topic-session | ✓ | 15,543 | 5 | ✓ | 7,616 | 3 |
| Aggregate | 5/5 | avg 21.7 s | - | 5/5 | avg 7.4 s | - |
DeepSeek V4 Flash passes every reliability bar - 5/5 parse,
correct restraint on low-signal, no hallucinated dates (the
"2026-08" that appeared in its output was legitimately in the
source observations), correct kind: rule classification.
Notable: fixture 3 took 54 s, suggesting variance under load
or extended reasoning. On multi-topic it produced 5 updates
vs the 3 the other models settled on - slightly more
exuberant than Haiku.
Ranking on a 0–5 scale per axis, then aggregated. Higher is better in every column except Cost (where lower-cost gets a higher score). Bold = best in that column. Rows sorted by overall fitness for ai-memory.
| # | Provider | Parse | Speed | Cost† | Faithfulness | Restraint | Classification | Fitness |
|---|---|---|---|---|---|---|---|---|
| 1 | Haiku 4.5 | 5 | 4 | 3 | 5 | 5 | 5 | 5 - recommended default |
| 2 | GPT-5.4-mini | 5 | 5 | 5 | 5 | 3 | 4 | 4 - cheaper alternative |
| 3 | qwen3:32b (Ollama) | 5 | 1 | 5 ($0) | 5 | 5 | 4 | 4 - free if you have a local server |
| 4 | DeepSeek V4 Flash | 5 | 2 | 4 | 5 | 5 | 5 | 4 - no edge over GPT-mini or Haiku |
| 5 | Sonnet 4.5 | 5 | 4 | 1 | 4‡ | 4 | 3 | 3 - displaced by Haiku |
| 6 | Kimi-K2.6 | 0 | 0 | n/a | n/a | n/a | n/a | 0 - ineligible (reasoning model) |
† Cost score derived from order-of-magnitude $/M output tokens: $0 (qwen3) = 5; ~$0.40 (DeepSeek) = 4; ~$1 (GPT-mini) = 5 by amortised cost-per-task; ~$5 (Haiku) = 3; ~$15 (Sonnet) = 1.
‡ Sonnet's faithfulness was 2/5 with the loose prompt (invented dates, fabricated alternatives sections). It recovers to 4 after the tightened prompt - but Haiku achieved that same level without needing the prompt change as much, suggesting Haiku has better defaults for this task.
- Parse: structural reliability. Anything less than 5/5 means production consolidation can fail silently and lose observations.
- Speed: matters less for consolidation (it's a background job) but compounds for lint sweeps over many pages and affects developer iteration on the prompt itself.
- Cost: integrates over N sessions/day × 365 days. Inexpensive options change the cost from monthly-recurring to negligible.
- Faithfulness: does the model only write what's in the observations? Critical for a memory wiki - fabrication corrupts the long-term record.
- Restraint: does the model resist manufacturing pages when the session is low-signal? Lack of restraint pollutes the wiki with thin, manufactured "decision" pages that outweigh real content.
- Classification: does the model correctly mark rules as
kind: rule, decisions askind: decision, etc.? Wrong classification breaks the consolidator's auto-routing to_rules/<slug>.md. - Fitness for ai-memory: holistic verdict per provider for this specific consolidation workload. Not a generic LLM benchmark.
For ai-memory's consolidation task specifically:
- Haiku 4.5 - recommended default for most users. Hosted (always available), 7 s avg latency, restraint + classification top of the field, ~$0.02/run is negligible for personal use. The benchmark every other option is measured against.
- GPT-5.4-mini - cheaper hosted alternative. ~5× cheaper than Haiku, 2× faster (4 s avg). Only weakness is mild over-classification on trivial sessions (manufactures one extra "decisions/" page on a typo-fix session). If budget matters more than restraint, pick this.
- qwen3:32b on Ollama - free alternative for those
with a local server. $0 per consolidation. ~92 s
latency is invisible because consolidation is a background
job. Restraint + faithfulness match the top hosted models.
Requires Ollama (or compatible OpenAI-compat server) with
qwen3:32bpulled and enough RAM/VRAM (~20 GB) to keep it warm. - DeepSeek V4 Flash - solid but no clear edge. All reliability bars met, faithful, restrained, correctly classifies rules. But GPT-mini matches it on quality and beats it on speed; Haiku matches it on quality and beats it on classification consistency. Pick only if your workflow is already DeepSeek-leaning.
- Sonnet 4.5 - strictly dominated by Haiku for plain consolidation. 3× the cost for the same parse rate and only marginally different latency. Reserve for tasks that specifically need extended reasoning (cross-page lint sweeps that compare contradictory claims across many pages, or for sparse-observation sessions where you want the model to infer more aggressively).
- Kimi-K2.6 - ineligible. Reasoning model burns
max_tokensbudget on internal thinking before emitting visible content. Hangs indefinitely on strict-JSON prompts. Same caveat applies to any other reasoning-mode model (Claude with extended thinking, GPT-o3, Gemini "thinking" variants) - turn reasoning off or budget tokens with consumption in mind before using them here.
Reading the raw .md outputs side-by-side reveals a
substantive style difference that the parse-rate numbers
don't capture:
-
Sonnet writes long, comprehensive entries. A concept page on Docker multi-stage builds will get 3 KB of well- organised prose including "When to use" / "When NOT to use" / "Gotchas" sections - content that wasn't in the observations. The model is generating useful tutorial-style content, not strictly consolidating what happened. Sonnet's fixture 05 page invented a
Date: 2025-01-23field that has no source in the observations. -
qwen3 writes terse, faithful entries. Each page captures what the session actually contained, in ~500–800 chars. No invented metadata, no generic tutorial filler. The same Docker page from qwen3 stays close to "we changed the Dockerfile to two-stage, image went 380→67 MB" without diverging into broader best-practices discussion.
For wiki consolidation (faithful long-term memory of this project, not a knowledge graph of general best practices), qwen3's restraint is arguably preferable to Sonnet's exuberance. The point of the wiki is to record what happened in the project, not to host re-generated tutorial content the model already knows.
That said, when the project memory is genuinely sparse and the model is asked to surface durable knowledge, Sonnet's "fill in the obvious" tendency could pay off. Different tasks → different preferences.
After three iterations of fixes (schema → first prompt → tightened prompt), the picture is clear:
- Parse: 5/5 (tightened prompt)
- Latency: ~92 s avg end-to-end. Acceptable because consolidation is a background job, not interactive.
- Cost: $0 per consolidation (electricity not modeled).
- Fidelity: comparable to or better than the hosted models
- qwen3 was the most faithful provider in Run 2's old-prompt comparisons.
If the homelab is unreachable, or for one-off complex consolidations, Haiku 4.5 is the right hosted choice - not Sonnet 4.5:
- 2× faster than Sonnet at every fixture.
- ~3× cheaper per token (Anthropic published pricing: Haiku 4.5 ≈ $1/$5 per M input/output tokens vs Sonnet 4.5 ≈ $3/$15).
- Less hallucination-prone even on the loose prompt.
- Better classification on at least one fixture (correctly identified the rule that Sonnet flattened to a gotcha).
- Same 5/5 parse reliability.
Sonnet's reasoning headroom doesn't help consolidation. With the loose prompt it expressed itself as more hallucination (invented dates, fabricated alternative-considered tables, tutorial-style filler). The tightened prompt brings Sonnet in line, but Haiku gives identical reliability faster and cheaper. Reserve Sonnet for tasks where the extra reasoning matters (e.g. cross-page lint sweeps that compare contradictory claims).
Reasoning model - burns max_tokens budget internally before
emitting visible content. Run hung for 16+ minutes on fixture 1
under the strict-JSON prompt. Direct probe confirmed: content: null with the entire token budget consumed by reasoning.
Not a prompt problem; the model is structurally wrong for
strict-JSON output. Same caveat applies to other reasoning-
mode models if used in this pipeline.
| Provider | $/run* | latency | notes |
|---|---|---|---|
| Ollama qwen3:32b (local) | $0 | ~92 s | electricity not modeled |
| GPT-5.4-mini (OpenRouter) | ~$0.005 | ~4 s | fastest + cheapest hosted |
| DeepSeek V4 Flash (OpenRouter) | ~$0.005 | ~22 s | cheap but slower than GPT-mini |
| Haiku 4.5 (OpenRouter) | ~$0.02 | ~7 s | best restraint/classification |
| Sonnet 4.5 (OpenRouter) | ~$0.06 | ~11 s | 3× cost of Haiku for same task |
| Kimi-K2.6 (OpenRouter) | n/a | ✗ hangs | reasoning model - ineligible |
* Rough order of magnitude; ai-memory consolidations land around 2–3 KB of output with the tightened prompt. Per-run $ multiplies $/M-tokens by the input+output token budget.
Re-run this harness when any of the following changes:
- The consolidation prompt itself is re-engineered
- A new Ollama model is pulled (e.g. when Qwen 3.5 stable drops for Ollama)
- A new fixture is added to
evals/fixtures/ - The home server hardware changes
- A local engine changes its
response_format=json_schemaimplementation, making a fresh default-vs-AI_MEMORY_LLM_COMPAT_STRICT=falsecomparison worthwhile
- Repo checkout +
cargotoolchain (Rust 1.95+, as pinned inrust-toolchain.toml). - An OpenRouter API key, exported as
OPENROUTER_API_KEY- pays the Kimi + Sonnet legs. - A reachable Ollama with
qwen3:32bpulled. The default URL in the docs assumes the homelab; substitute your own.
The canonical 2-side invocation (the harness compares two providers per run):
cargo run -p ai-memory-eval --release -- \
--baseline-provider openai-compat \
--baseline-base-url https://openrouter.ai/api/v1 \
--baseline-model moonshotai/kimi-k2.6 \
--baseline-api-key-env OPENROUTER_API_KEY \
--candidate-provider openai-compat \
--candidate-base-url http://192.168.0.90:11434/v1 \
--candidate-model qwen3:32b \
--candidate-api-key ollama-localFor a 3-way comparison, run the harness three times pairing the candidate (the model you're considering switching to) against each baseline you want to compare against. Output dirs are timestamped, so they don't collide.
evals/runs/<timestamp>/
├── baseline/
│ ├── 01-rust-bug-fix.json ← parsed structured output (if any)
│ ├── 01-rust-bug-fix.md ← flat-rendered for eyeballing
│ ├── 01-rust-bug-fix.raw.txt ← exact model output, always present
│ └── 01-rust-bug-fix.meta.json ← {elapsed_ms, parsed_ok, update_count, error}
└── candidate/
└── ...
The .raw.txt files are the most informative artifact when a
parse fails - they show exactly what the model said, so you
can tell whether the failure was format (model emitted prose),
schema (model used integer enums), or substance (model
produced nothing useful).
For side-by-side reading the runner prints a hint:
compare with: diff -ru <run>/baseline <run>/candidate
Each fixture is a JSON file under evals/fixtures/:
{
"name": "human-readable-id",
"description": "what this case is meant to surface",
"observations": [
{"kind": "session-start", "title": "...", "body": "..."},
{"kind": "user-prompt", "title": "user prompt", "body": "..."},
{"kind": "pre-tool-use", "title": "Edit", "body": "..."}
]
}kind accepts any string the
ObservationKind
enum's FromStr understands. Anything unknown silently falls
back to Other.
Try to hit one of the four hard cases:
- Multi-page extraction - does the model split a session into the right slices?
- Restraint - does it avoid manufacturing pages when there's nothing durable?
- Classification - does it correctly choose
kind: rulefor project rules? - Topic separation - does it produce separate pages per unrelated topic instead of mashing them?
- Automated quality scoring. The runner only reports objective deltas (latency, parse rate, update count). Anything subtler (faithfulness, hallucination, scoping) needs a human reader.
- Embedding A/B. This document is LLM-only. The embedding provider switch (OpenAI text-embedding-3-small → Ollama nomic-embed-text) gets its own writeup when there's enough page-side data to measure retrieval quality.
- LLM-as-judge scoring. Adding a third "judge" model to score the candidate outputs against a rubric would automate quality measurement. Not built; the next layer up if this harness gets used regularly.
If we end up running this harness routinely:
- Add a third position (
--judge-*) so a separate "judge" model can score baseline vs candidate per fixture against a rubric, producing a numeric quality delta. - Extend fixtures with a
must_mention/must_not_mentionkeyword list so we can compute simple keyword recall automatically (catches obvious hallucinations / missing facts). - Parallel embedding-retrieval eval: a probe set of queries each tagged with the expected target wiki page; compute recall@5 + MRR for two embedding models against the same indexed corpus.
- Persist a leaderboard somewhere durable (a wiki page, ironically) so we don't lose track of which model performed best on which fixture across runs.