perf(normalize): stream Claude Code .jsonl instead of loading it whole (900MB -> 63MB peak)

[Lcstyle]

Problem

normalize_conversations() reads the whole transcript through
_read_transcript_file(), then takes a second full copy with
content.split("\n"). Measured peak RSS on one real 78 MB Claude Code session:

import                      83 MB
read transcript            395 MB   (+312)   <- 4x the file size
+ split into lines         526 MB   (+131)
normalize_conversations()  900 MB peak

Transcripts of 436 MB and 769 MB exist in practice on long-running hosts.
MAX_FILE_SIZE skips above 500 MB, but everything below it is eligible, so one
file can approach available RAM on a smaller box and take the miner with it.

Change

Split _try_claude_code_jsonl into a thin string wrapper plus
_try_claude_code_jsonl_lines(line_iter, verbatim=...) holding the unchanged
body, then try the iterator form against the open file for .jsonl before the
eager read. If it returns a result, return it; if it returns None, fall through
to the existing path untouched.

+27 -1, one file.

Why this is behaviour-preserving

Not a claim — a property of reusing the same parser body. Specifically:

• The streaming path runs the same code the eager path would have run, so
  tool_use_map, is_tool_only merging, strip_noise, assistant-turn merging
  and verbatim all behave identically.
• The short-circuit is safe because _try_normalize_json_split already tries
  _try_claude_code_jsonl first; a successful stream returns exactly what
  the eager path would have returned.
• The fallthrough preserves the ">= 3 lines starting with '>'" passthrough and
  every non-Claude-Code format (Codex, Gemini, claude.ai and ChatGPT bundles,
  plain text), because such a file does not parse as Claude Code JSONL and the
  streaming attempt simply returns None.

This is deliberately narrower than upstream MemPalace#468, which
introduces its own parser and removes .jsonl from the existing dispatch. That
approach would change normalized output — and normalized text is stamped with
NORMALIZE_VERSION, so it would re-mine existing drawers with different content.

Verification

• Output equivalence: 14 real transcripts (up to 4.9M normalized chars),
  comparing sha256 of the joined normalize_conversations() output patched vs
  unpatched — 14/14 identical, 0 differences, 0 errors.
• Streaming confirmed: instrumented the call; the argument reaching the
  parser is a TextIOWrapper, i.e. the open file, not a pre-split list.
• Memory, same 78 MB transcript: peak RSS 900.0 MB -> 63.5 MB (14.2x);
  a 98 MB transcript peaks at 72.8 MB. End-to-end local mine cost fell from
  ~11.5x file size to 1.76x.
• Tests: tests/test_convo_miner.py, tests/test_convo_miner_unit.py,
  tests/test_convo_miner_size_cap.py — 108 passed.

Running on two production palaces (roughly 730k and 1.3M drawers).

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1b971086-1968-4faa-8ef3-350d83eb941c

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}