Feature: Block-based revision pipeline for million-word novels

[liutingqiu]

Problem

The current pipeline (run_pipeline.py) is designed for ~75k word novels (24 chapters). When scaling to million-word novels (150+ chapters), several issues arise:

1. Context overflow: review.py sends the full manuscript to Opus. 150 chapters × 3,500 words = 525k words, which exceeds even the 1M context window when combined with planning docs.
2. Late problem detection: Issues in chapter 5 aren't caught until the entire novel is drafted and reviewed. By then, downstream chapters have inherited the problems.
3. No continuity checking: When chapters are drafted sequentially over days/weeks, subtle continuity breaks accumulate (character state, timeline, physical details).
4. Flat revision: The revision phase edits the whole novel at once, making it hard to focus on specific story arcs.

Proposed Solution: Block-Based Pipeline

We've implemented a block-based revision system that splits the novel into manageable chunks (e.g., 50 chapters each) and processes them independently.

Architecture

Phase 1: Foundation (unchanged — master outline + world + characters + voice)
Phase 2: Block outline generation (NEW — detailed per-block story arcs)
Phase 3: Block drafting + review (NEW — draft → evaluate → review → fix, per block)
Phase 4: Cross-block continuity check + fix (NEW)
Phase 5: Full book review (Opus, all blocks)
Phase 6: Export (unchanged)

New Files

1. gen_block_outline.py — Block-level outline generator

Takes a block of chapters (e.g., 1-50) and generates a detailed, chapter-by-chapter story arc from the master outline. Includes:

• Block arc summary (central tension, entry/exit states)
• Tension curve across the block
• Detailed beats per chapter (more granular than master outline)
• Block-level foreshadowing ledger
• Entry/exit state (how this block connects to adjacent blocks)

uv run python gen_block_outline.py --block 1-50
uv run python gen_block_outline.py --block 51-100 --prev-summary "..."

2. run_block_pipeline.py — Block-based pipeline orchestrator

Full pipeline that processes blocks sequentially:

# Full pipeline from scratch
uv run python run_block_pipeline.py --from-scratch --blocks 1-50,51-100,101-150

# Individual phases
uv run python run_block_pipeline.py --phase block-outlines --blocks 1-50,51-100,101-150
uv run python run_block_pipeline.py --phase block-draft --blocks 1-50,51-100,101-150
uv run python run_block_pipeline.py --phase final-review --blocks 1-50,51-100,101-150
uv run python run_block_pipeline.py --phase export

Each block goes through: draft → evaluate → adversarial edit → reader panel → targeted revisions → block evaluation → Opus review.

Modified Files

3. review.py — Added --block START-END parameter

build_manuscript() now accepts a block parameter to filter chapters. The review prompt includes block context.

uv run python review.py --block 1-50

4. evaluate.py — Added --block parameter for --full mode

evaluate_full() filters chapters by block range. The prompt adapts to describe the scope.

uv run python evaluate.py --full --block 1-50

5. reader_panel.py — Added --block parameter

Loads block-specific arc summary, saves block-specific panel results.

uv run python reader_panel.py --block 1-50

6. adversarial_edit.py — Added range syntax

uv run python adversarial_edit.py 1-50    # block range
uv run python adversarial_edit.py all      # all chapters (auto-detected, no longer hardcoded 1-24)

7. build_arc_summary.py — Added --block parameter

Generates block-specific arc summary files. Removed hardcoded novel premise and chapter range.

uv run python build_arc_summary.py --block 1-50

8. draft_chapter.py — Auto-detects block outlines

When block_outlines/ directory exists, automatically loads the block-level outline for the chapter being drafted and includes it in the context. No CLI change needed.

9. run_pipeline.py — Added --blocks parameter

The original pipeline now supports block-based revision via --blocks:

uv run python run_pipeline.py --phase revision --blocks 1-50,51-100,101-150

Added run_block_revision() function and cross_block_check() for continuity checking between adjacent blocks.

Key Design Decisions

1. Block boundaries align with story arcs: Each block should cover a natural story arc (e.g., Act I, Act II Part 1, etc.), not arbitrary chapter counts.

2. Progressive disclosure: Block outlines are generated sequentially, with the previous block's exit state feeding into the next block's outline generation.

3. Cross-block continuity check: After all blocks are drafted, the boundary chapters (last of block N, first of block N+1) are checked for plot/emotional/physical/timeline/voice continuity. Issues are auto-fixed via gen_revision.py.

4. Checkpoint/resume: block_state.json tracks per-chapter and per-block completion status. If interrupted, re-running the same command resumes from the last completed unit.

5. No breaking changes: All modifications are backward-compatible. Without --block/--blocks flags, the tools behave exactly as before.

Example Workflow for a 150-Chapter Novel

# 1. Foundation (same as before)
uv run python run_block_pipeline.py --phase foundation

# 2. Generate block outlines (sequential, each informed by previous)
uv run python run_block_pipeline.py --phase block-outlines --blocks 1-50,51-100,101-150

# 3. Draft + review each block independently
uv run python run_block_pipeline.py --phase block-draft --blocks 1-50,51-100,101-150
# → Block 1: draft 50 chapters → adversarial edit → reader panel → revise → Opus review
# → Block 2: same process
# → Block 3: same process

# 4. Cross-block continuity + full book review
uv run python run_block_pipeline.py --phase final-review --blocks 1-50,51-100,101-150
# → Check ch50→ch51, ch100→ch101 continuity
# → Fix any issues
# → Full book Opus review

# 5. Export
uv run python run_block_pipeline.py --phase export

Testing

All modified files pass py_compile and linter checks. The changes are additive — no existing functionality is removed or altered.

Related Issues

• Entire Bell story is hard coded into code #9 "Entire Bell story is hard coded into code" — our changes to build_arc_summary.py and adversarial_edit.py remove hardcoded chapter ranges and novel-specific content.
• Master has a lot of the bells novel "baked in" #7 "Master has a lot of the bells novel baked in" — same, the block tools are novel-agnostic.

We'd love feedback on the approach and are happy to submit a PR if there's interest in merging this upstream.