feat: calculate page offsets to align toc margins

Normalize entry positions using baselines registered across pages instead
of simple per-page minimum values. Prevent single far-right columns from
creating spurious indent levels. Skip "Contents at a Glance" pages and
replicated running headers.

Author: mmgeorge

AGENTS.md

@@ -19,6 +19,43 @@ crates/papers-extract/scratch.txt
 
 The `.temp/` directory is gitignored. Create it on demand if it doesn't exist.
 
+## Verify PDF Layout by Rendering and Dumping — Do NOT Infer
+
+When reasoning about a PDF's layout (indent levels, columns, depth, which
+glyphs/numbers a line has, why an entry is mis-parsed), **render the actual
+page to an image and look at it, and dump the page's chars.** Do NOT draw
+conclusions from x-coordinate dumps or text dumps alone, and never guess from
+the parser's output what the source "must" look like. A two-minute render
+settles questions that hours of coordinate-inference get wrong (e.g. "is this a
+sub-section or a sibling section?", "does this chapter have a number?", "is the
+gutter real?"). This is mandatory before concluding a fixture is wrong, a page
+is a certain layout, or a defect is "unfixable".
+
+**Render a page to an image (then Read the image):**
+```bash
+# Windows (Ghostscript). Page numbers are 1-indexed.
+gswin64c -q -dNOPAUSE -dBATCH -dSAFER -sDEVICE=png16m -r140 \
+  -dFirstPage=8 -dLastPage=8 -sOutputFile=.temp/render/page8.png data/<file>.pdf
+# (pdftoppm / mutool draw / magick are equivalents on other platforms)
+```
+
+**Dump a page's chars (codepoint, bbox, font) — for exact x positions, broken
+glyphs, hanging number columns, sub/superscripts:**
+```bash
+cargo run --release --bin dump_chars -- data/<file>.pdf <page_num>   # 1-indexed
+```
+
+**Dump pdfium's reference text for the whole PDF (find a page, see word
+boundaries / wrap markers):**
+```bash
+cargo run --release --bin dump_text -- data/<file>.pdf .temp/<file>_text.txt
+```
+
+Page-index gotcha: Ghostscript `-dFirstPage` and `dump_chars` are **1-indexed**;
+pypdf and our internal `page_idx` are **0-indexed**. An entry can also appear
+twice in `dump_text` (once in the TOC, once in the body) — confirm you are
+looking at the TOC page, not the body section of the same title.
+
 ## Extraction Pipeline Architecture
 
 The PDF extraction pipeline has three stages. Fixes and logic should go

crates/papers-extract/src/toc.rs

@@ -527,6 +527,14 @@ fn find_toc_pages(page_chars: &[(Vec<PdfChar>, f32)]) -> Vec<u32> {
             if trimmed.starts_with("list of") || normalized.starts_with("listof") {
                 continue;
             }
+            // "Contents at a Glance" is a brief summary TOC that duplicates the
+            // chapter/appendix headings of the detailed "Table of Contents".
+            // Skip it as a start page so the detailed TOC is used instead; the
+            // glance page then falls outside the ascending main-TOC run and is
+            // not extracted, avoiding duplicated (and phantom wrapped) headings.
+            if normalized.contains("ataglance") {
+                continue;
+            }
             if normalized == "contents"
                 || normalized == "tableofcontents"
                 || normalized == "detailedcontents"
@@ -780,6 +788,20 @@ fn line_ends_with_number(line: &TocRawLine) -> bool {
 
 // ── Phase 2: Line extraction ──
 
+/// True for an appendix-letter heading fragment: a single uppercase letter
+/// followed by a space and a capitalized word ("B Message Crackers, …",
+/// "A The Build Environment"). Used to keep such a wrapped heading on an
+/// offset-page top band, where its page number sits on the continuation line so
+/// the usual number/leader-dot/heading-number signals are absent. A running
+/// header ("Table of Contents") fails because its second word is lowercase.
+fn is_appendix_letter_heading(text: &str) -> bool {
+    let t = text.trim();
+    let mut chars = t.chars();
+    chars.next().is_some_and(|c| c.is_ascii_uppercase())
+        && chars.next() == Some(' ')
+        && chars.next().is_some_and(|c| c.is_ascii_uppercase())
+}
+
 fn extract_toc_lines(page_chars: &[(Vec<PdfChar>, f32)], toc_pages: &[u32]) -> Vec<TocRawLine> {
     let mut all_lines = Vec::new();
 
@@ -827,7 +849,8 @@ fn extract_toc_lines(page_chars: &[(Vec<PdfChar>, f32)], toc_pages: &[u32]) -> V
                 }
                 return line_ends_with_number(line)
                     || headings::has_leader_dots(&text)
-                    || starts_with_heading_pattern(text.trim());
+                    || starts_with_heading_pattern(text.trim())
+                    || is_appendix_letter_heading(&text);
             }
             true
         });
@@ -882,6 +905,23 @@ fn extract_toc_lines(page_chars: &[(Vec<PdfChar>, f32)], toc_pages: &[u32]) -> V
         all_lines.extend(lines);
     }
 
+    // Drop standalone "Contents" / "Table of Contents" running headers that
+    // REPEAT across pages. A repeated pure-"Contents" line (optionally followed
+    // by a roman page number) sits at an arbitrary, often far-right x and would
+    // otherwise create a spurious indent level and clear the depth-resolution
+    // stack. Gated on ≥2 occurrences so a single real "Contents (p. v)"
+    // front-matter entry (e.g. in `edo`) is preserved. The topmost-line filter
+    // above already handles the once-per-page header in the common case.
+    let is_pure_contents = |l: &TocRawLine| {
+        let norm = normalize_toc_header_text(&l.text());
+        ["contents", "tableofcontents", "detailedcontents"]
+            .iter()
+            .any(|h| norm.starts_with(h) && norm[h.len()..].chars().all(|c| "ivxlcdm".contains(c)))
+    };
+    if all_lines.iter().filter(|l| is_pure_contents(l)).count() >= 2 {
+        all_lines.retain(|l| !is_pure_contents(l));
+    }
+
     all_lines
 }
 
@@ -2551,24 +2591,19 @@ fn classify_and_learn(lines: Vec<TocSplitLine>) -> Vec<ClassifiedEntry> {
         .iter()
         .map(|(l, _, _)| l.x_left)
         .collect();
-    // Normalize x_left per page: subtract each page's minimum x_left so that
-    // indent positions are comparable across pages with different margins.
-    let mut page_min_x: HashMap<u32, f32> = HashMap::new();
-    for (line, _, _) in &entries {
-        let min = page_min_x.entry(line.page_idx).or_insert(f32::MAX);
-        if line.x_left < *min {
-            *min = line.x_left;
-        }
-    }
+    // Normalize x_left per page by a registered baseline so indent positions are
+    // comparable across pages with different margins AND continuation pages that
+    // lack a chapter heading (see `compute_page_offsets`). Indent levels are
+    // count-aware so a stray far-right entry cannot create a spurious deep level.
+    let page_min_x: HashMap<u32, f32> = compute_page_offsets(&entries);
     let x_lefts_normalized: Vec<f32> = entries
         .iter()
         .map(|(l, _, _)| {
             let page_min = page_min_x.get(&l.page_idx).copied().unwrap_or(0.0);
             l.x_left - page_min
         })
         .collect();
-    let indent_levels = compute_indent_levels(&x_lefts_normalized);
-
+    let indent_levels = compute_indent_levels_counted(&x_lefts_normalized, 3);
 
     // Build a mapping from indent level → most common depth among classified
     // entries at that indent. This lets us anchor indent levels to known depths.
@@ -2841,6 +2876,109 @@ fn compute_indent_levels(x_lefts: &[f32]) -> Vec<f32> {
     levels
 }
 
+/// Like `compute_indent_levels` but drops levels backed by fewer than
+/// `min_count` entries. A single far-right entry (a page-number-column artifact
+/// or a two-column right-edge fragment) would otherwise create a spurious deep
+/// indent level that corrupts depth inference. Never returns empty if the
+/// unfiltered set was non-empty.
+fn compute_indent_levels_counted(x_lefts: &[f32], min_count: usize) -> Vec<f32> {
+    let levels = compute_indent_levels(x_lefts);
+    if levels.len() <= 1 {
+        return levels;
+    }
+    let mut counts = vec![0usize; levels.len()];
+    for &x in x_lefts {
+        counts[quantize_indent(x, &levels) as usize] += 1;
+    }
+    let mut filtered: Vec<f32> = Vec::new();
+    for (i, &l) in levels.iter().enumerate() {
+        if counts[i] >= min_count {
+            filtered.push(l);
+        }
+    }
+    if filtered.is_empty() {
+        levels
+    } else {
+        filtered
+    }
+}
+
+/// Per-page x baseline that registers continuation pages to a canonical indent
+/// grid (offset-invariant depth). A naïve per-page *minimum* fails on a
+/// continuation page that carries only a chapter's sections (no chapter
+/// heading): its leftmost entry is at the *section* level, so subtracting the
+/// page min collapses every entry up a depth. Books also shift recto/verso pages
+/// by a binding margin (Windows Internals offsets odd pages +30pt). So:
+///   - Pages with a Chapter/Part heading anchor on that heading's x (the base
+///     indent, which already encodes the page's physical margin).
+///   - Continuation pages are aligned by the offset that best maps their indent
+///     levels onto the canonical grid (inter-level spacings are stable; only the
+///     absolute offset differs). Pages that do not align cleanly fall back to
+///     their own minimum.
+fn compute_page_offsets(entries: &[(TocSplitLine, Classification, u32)]) -> HashMap<u32, f32> {
+    let mut page_min: HashMap<u32, f32> = HashMap::new();
+    let mut pages: Vec<u32> = Vec::new();
+    for (l, _, _) in entries {
+        if !page_min.contains_key(&l.page_idx) {
+            pages.push(l.page_idx);
+        }
+        let m = page_min.entry(l.page_idx).or_insert(f32::MAX);
+        *m = m.min(l.x_left);
+    }
+    // Anchor each page on its CHAPTER level only. Parts/appendix dividers sit
+    // further left than chapters (one indent above), so anchoring a page on its
+    // part would shift that page's baseline relative to chapter-anchored pages
+    // and smear the indent levels. Pages without a chapter (part-only divider
+    // pages, appendix continuations, mid-chapter continuations) are instead
+    // REGISTERED below by aligning their section/subsection levels to the canon.
+    let mut page_offset: HashMap<u32, f32> = HashMap::new();
+    for (l, class, _) in entries {
+        if *class == Classification::Chapter {
+            let e = page_offset.entry(l.page_idx).or_insert(f32::MAX);
+            *e = e.min(l.x_left);
+        }
+    }
+    if page_offset.is_empty() {
+        return page_min; // no chapter anchor — keep per-page minimum
+    }
+    let canon_xs: Vec<f32> = entries
+        .iter()
+        .filter(|(l, _, _)| page_offset.contains_key(&l.page_idx))
+        .map(|(l, _, _)| l.x_left - page_offset[&l.page_idx])
+        .collect();
+    let canon = compute_indent_levels_counted(&canon_xs, 3);
+    for &p in &pages {
+        if page_offset.contains_key(&p) {
+            continue;
+        }
+        let fallback = page_min.get(&p).copied().unwrap_or(0.0);
+        let page_xs: Vec<f32> = entries
+            .iter()
+            .filter(|(l, _, _)| l.page_idx == p)
+            .map(|(l, _, _)| l.x_left)
+            .collect();
+        let page_levels = compute_indent_levels_counted(&page_xs, 2);
+        let mut best_delta = fallback;
+        let mut best_err = f32::MAX;
+        for &pl in &page_levels {
+            for &cl in &canon {
+                let delta = pl - cl;
+                let err: f32 = page_levels
+                    .iter()
+                    .map(|&x| canon.iter().map(|&c| (x - delta - c).abs()).fold(f32::MAX, f32::min))
+                    .sum();
+                if err < best_err {
+                    best_err = err;
+                    best_delta = delta;
+                }
+            }
+        }
+        let avg_err = best_err / page_levels.len().max(1) as f32;
+        page_offset.insert(p, if avg_err <= 6.0 { best_delta } else { fallback });
+    }
+    page_offset
+}
+
 /// Quantize an x-position to an indent level (0-based).
 fn quantize_indent(x: f32, levels: &[f32]) -> u32 {
     levels

crates/papers-extract/tests/fixtures/toc/windows_cpp.md

@@ -35,22 +35,22 @@
     - 3.3.3 Duplicating Object Handles (p. 60)
 - 4 Processes (p. 67)
   - 4.1 Writing Your First Windows Application (p. 68)
-  - 4.2 A Process Instance Handle (p. 73)
-  - 4.3 The CreateProcess Function (p. 89)
-    - 4.3.1 pszApplicationName and pszCommandLine (p. 89)
-  - 4.4 Terminating a Process (p. 104)
-    - 4.4.1 The Primary Thread's Entry-Point Function Returns (p. 104)
-    - 4.4.2 The ExitProcess Function (p. 105)
-    - 4.4.3 The TerminateProcess Function (p. 106)
-    - 4.4.4 When All the Threads in the Process Die (p. 107)
-    - 4.4.5 When a Process Terminates (p. 107)
-  - 4.5 Child Processes (p. 108)
-    - 4.5.1 Running Detached Child Processes (p. 110)
-  - 4.6 When Administrator Runs as a Standard User (p. 110)
-    - 4.6.1 Elevating a Process Automatically (p. 113)
-    - 4.6.2 Elevating a Process by Hand (p. 115)
-    - 4.6.3 What Is the Current Privileges Context? (p. 117)
-    - 4.6.4 Enumerating the Processes Running in the System (p. 118)
+    - 4.1.1 A Process Instance Handle (p. 73)
+  - 4.2 The CreateProcess Function (p. 89)
+    - 4.2.1 pszApplicationName and pszCommandLine (p. 89)
+  - 4.3 Terminating a Process (p. 104)
+    - 4.3.1 The Primary Thread's Entry-Point Function Returns (p. 104)
+    - 4.3.2 The ExitProcess Function (p. 105)
+    - 4.3.3 The TerminateProcess Function (p. 106)
+    - 4.3.4 When All the Threads in the Process Die (p. 107)
+    - 4.3.5 When a Process Terminates (p. 107)
+  - 4.4 Child Processes (p. 108)
+    - 4.4.1 Running Detached Child Processes (p. 110)
+  - 4.5 When Administrator Runs as a Standard User (p. 110)
+    - 4.5.1 Elevating a Process Automatically (p. 113)
+    - 4.5.2 Elevating a Process by Hand (p. 115)
+    - 4.5.3 What Is the Current Privileges Context? (p. 117)
+    - 4.5.4 Enumerating the Processes Running in the System (p. 118)
 - 5 Jobs (p. 125)
   - 5.1 Placing Restrictions on a Job's Processes (p. 129)
   - 5.2 Placing a Process in a Job (p. 136)
@@ -257,8 +257,11 @@
     - 20.1.3 Explicitly Linking to an Exported Symbol (p. 561)
   - 20.2 The DLL's Entry-Point Function (p. 562)
     - 20.2.1 The DLL_PROCESS_ATTACH Notification (p. 563)
-    - 20.2.2 Serialized Calls to DllMain (p. 567)
-    - 20.2.3 DllMain and the C/C++ Run-Time Library (p. 570)
+    - 20.2.2 The DLL_PROCESS_DETACH Notification (p. 564)
+    - 20.2.3 The DLL_THREAD_ATTACH Notification (p. 566)
+    - 20.2.4 The DLL_THREAD_DETACH Notification (p. 567)
+    - 20.2.5 Serialized Calls to DllMain (p. 567)
+    - 20.2.6 DllMain and the C/C++ Run-Time Library (p. 570)
   - 20.3 Delay-Loading a DLL (p. 571)
     - 20.3.1 The DelayLoadApp Sample Application (p. 576)
   - 20.4 Function Forwarders (p. 583)
@@ -366,6 +369,7 @@
     - Forcing the Linker to Look for a (w)WinMain Entry-Point Function (p. 766)
     - Support XP-Theming of the User Interface with pragma (p. 766)
 - B Message Crackers, Child Control Macros, and API Macros (p. 773)
+  - Message Crackers (p. 773)
   - Child Control Macros (p. 776)
   - API Macros (p. 776)
 - Index (p. 779)