fix: algorithm handling

Author: mmgeorge

crates/papers-extract/EXTRACTION.md

@@ -97,6 +97,90 @@ Then:
 | `output.rs` | `dedup_heading_echo()` | Both (runs in reflow) |
 | `output.rs` | `dedup_consecutive_text()` | Both (runs in reflow) |
 
+## Text-Only Detection Logic
+
+### Font-based heading detection
+The **font is the definitive signal** for headings. `partition_heading_chars()`
+extracts chars matching the heading font family (e.g., LinBiolinum vs body
+font LinLibertine). Heading chars are validated against `extract_headings()`
+results. Known headings take **priority over formula zones** — a heading at
+a Y-position overlapping a formula zone is kept because font overrides
+geometric heuristics.
+
+Body blocks should NEVER be classified as headings when font-based detection
+is active (`has_font_headings` flag). Multi-line body blocks that happen to
+start with heading text are not headings — the font is the tell.
+
+### Formula detection
+Two detection paths at the line level:
+1. **Content-based** (`is_likely_formula_text()`) — operators, math symbols,
+   prose word rejection, `$...$` marker density
+2. **Font-based** — ≥3 math italic Unicode chars (U+1D400-1D7FF) with no
+   prose words. These chars ARE the font signal.
+
+Formula detection is **suppressed in algorithm zones** (see below).
+
+**Key issues encountered:**
+- `$`/`{`/`}`/`_`/`^` markers from `extract_region_text()` inflate char
+  counts and dilute math_ratio. Use `content_total` (excluding these
+  formatting chars) for ratio calculations.
+- "fi"/"fifi" ligature artifacts from PDF absolute value bars `|...|` look
+  like prose words. Excluded via `is_ligature_artifact()`.
+- Pseudocode keywords ("if", "for") should only reject formulas when at the
+  START of a line (pseudocode), not mid-line (piecewise formula conditions
+  like "value, if condition").
+- No upper char limit on `is_likely_formula_text()` — piecewise formulas
+  and multi-line formulas can be very long.
+
+### Algorithm detection
+Algorithm zones are detected from **line numbers** ("1:", "15:", "37:") — an
+unambiguous structural signal that display formulas never have. When ≥2
+numbered lines exist in a column, their Y-range defines an algorithm zone.
+
+Inside algorithm zones:
+- **Formula detection suppressed** — pseudocode with math variables is not
+  a display formula
+- **All heuristic breaks suppressed** (y_break, x_break, font_break) —
+  algorithm pseudocode has subscript fragments, varying indentation, and
+  font size changes that would fragment the block
+
+Algorithm caption splitting: when a block starts with "Algorithm N *title*"
+and contains numbered lines, the caption is split into a separate
+`FigureTitle` region and the body becomes an `Algorithm` region.
+
+In the reflow stage, `Algorithm` nodes are **never demoted to Text**.
+They either stay as `Algorithm` (pseudocode) or get promoted to `CodeBlock`
+(actual programming code). `Algorithm` renders as plain text (not fenced
+code blocks) because algorithms can contain `$...$` LaTeX math.
+
+### Subscript / superscript handling
+**Fundamental issue**: PDF text layer chars in math expressions span
+multiple Y positions (subscripts, superscripts, fraction numerators and
+denominators). `group_into_lines()` uses Y-proximity to group chars into
+lines.
+
+**Current approach**: bbox-based line grouping. A char belongs to the current
+line if its Y-center falls within the line's Y bounding box (expanded by
+`avg_height * 0.3` padding). This handles normal subscripts (3-4pt offset)
+but NOT fraction numerators/denominators (6-8pt offset) because they would
+merge actual separate prose lines.
+
+**Remaining limitation**: fraction parts (`1/Δt²`) still fragment into
+separate lines in the text-only path. The ML layout path handles this via
+OCR (GLM-OCR produces per-line LaTeX). Fixing this in the text-only path
+requires X-proximity-aware grouping — chars at different Y but overlapping
+X are part of the same expression. This is a known TODO.
+
+**Fragment break suppression**: tiny lines (≤3 chars) never cause heuristic
+breaks (y_break, x_break, font_break) because they're subscript/superscript
+fragments attached to adjacent content, not separate blocks.
+
+### Overlapping formula deduplication
+`dedup_overlapping_formulas()` **merges** (not picks-one) overlapping formula
+regions. Requires both vertical AND horizontal overlap to prevent merging
+formulas across columns. This handles cases where loose_bounds inflate
+formula bboxes into overlapping territory.
+
 ## Known Limitations
 
 ### Text-only path
@@ -105,6 +189,7 @@ Then:
 - **Missing body content**: Some PDFs (computer_systems, programming_massively_parallel) have body pages that extract to empty.
 - **Font mapping failures**: lambda book — TeX fonts lack Unicode mappings.
 - **Margin notes**: Interleaved with body text in some books (EDO, fluids).
+- **Fraction fragmentation**: Math fractions (`1/Δt²`) produce separate lines for numerator, bar, and denominator in `group_into_lines()`. No Y-threshold alone can merge them without also merging real separate prose lines. Needs X-proximity-aware grouping.
 
 ### Layout path
 - Requires GPU for layout detection, formula OCR, and table OCR.

crates/papers-extract/src/output.rs

@@ -3424,35 +3424,41 @@ fn detect_code_blocks(flat_nodes: &mut Vec<ReflowNode>) {
                 }
             }
             ReflowNode::Algorithm { content } => {
-                // Algorithm regions can be: real pseudocode, programming code, or
-                // misclassified prose. Check structural signals to decide.
                 let is_pseudocode = looks_like_pseudocode(content);
-                if is_pseudocode || looks_like_code(content) || code_score(content) >= 2 {
-                    // Check if it's actually prose with math — full sentences without
-                    // algorithmic structure shouldn't be code blocks
-                    let has_math = content.matches('$').count() >= 2
-                        || content.contains("[[FORMULA")
-                        || count_italic_spans(content) >= 2;
-                    if has_math && !is_pseudocode && !looks_like_code(content) {
-                        // Math-heavy prose misclassified as Algorithm
-                        *node = ReflowNode::Text {
-                            content: content.replace('\n', " "),
-                            footnotes: Vec::new(),
-                        };
-                    } else {
-                        let trimmed = trim_trailing_prose(content);
-                        let lang = if is_pseudocode {
-                            None
-                        } else {
-                            guess_language(&trimmed)
-                        };
-                        *node = ReflowNode::CodeBlock {
-                            content: trimmed,
-                            language: lang,
-                        };
-                    }
+                let is_code = looks_like_code(content);
+                let has_line_nums = crate::text_cleanup::has_algorithm_line_number(content);
+                let has_math = content.matches('$').count() >= 2
+                    || content.contains("[[FORMULA")
+                    || count_italic_spans(content) >= 2;
+                let trimmed = trim_trailing_prose(content);
+                let lang = guess_language(&trimmed);
+
+                if lang.is_some() {
+                    // Recognized programming language → CodeBlock
+                    *node = ReflowNode::CodeBlock {
+                        content: trimmed,
+                        language: lang,
+                    };
+                } else if is_pseudocode || has_line_nums {
+                    // Pseudocode or has line numbers → keep as Algorithm
+                    *node = ReflowNode::Algorithm {
+                        content: trimmed,
+                    };
+                } else if has_math && !is_code {
+                    // Math-heavy content without code signals — ML model
+                    // misclassified prose as Algorithm. Demote to Text.
+                    *node = ReflowNode::Text {
+                        content: content.replace('\n', " "),
+                        footnotes: Vec::new(),
+                    };
+                } else if is_code || code_score(content) >= 2 {
+                    // Code without recognized language → CodeBlock
+                    *node = ReflowNode::CodeBlock {
+                        content: trimmed,
+                        language: None,
+                    };
                 } else {
-                    // Not code — demote to Text so it renders as prose
+                    // No signals at all → demote to Text
                     *node = ReflowNode::Text {
                         content: content.replace('\n', " "),
                         footnotes: Vec::new(),
@@ -3463,7 +3469,7 @@ fn detect_code_blocks(flat_nodes: &mut Vec<ReflowNode>) {
         }
     }
 
-    // Step 2: Merge consecutive CodeBlock nodes
+    // Step 2: Merge consecutive CodeBlock nodes and consecutive Algorithm nodes
     let mut i = 0;
     while i + 1 < flat_nodes.len() {
         let is_code_pair = matches!(&flat_nodes[i], ReflowNode::CodeBlock { .. })
@@ -3473,13 +3479,23 @@ fn detect_code_blocks(flat_nodes: &mut Vec<ReflowNode>) {
                 if let ReflowNode::CodeBlock { content: c1, language: l1 } = &mut flat_nodes[i] {
                     c1.push('\n');
                     c1.push_str(&c2);
-                    // Keep the more specific language
                     if l1.is_none() && l2.is_some() {
                         *l1 = l2;
                     }
                 }
             }
-            continue; // re-check same position for further merging
+            continue;
+        }
+        let is_algo_pair = matches!(&flat_nodes[i], ReflowNode::Algorithm { .. })
+            && matches!(&flat_nodes[i + 1], ReflowNode::Algorithm { .. });
+        if is_algo_pair {
+            if let ReflowNode::Algorithm { content: c2 } = flat_nodes.remove(i + 1) {
+                if let ReflowNode::Algorithm { content: c1 } = &mut flat_nodes[i] {
+                    c1.push('\n');
+                    c1.push_str(&c2);
+                }
+            }
+            continue;
         }
         i += 1;
     }
@@ -3917,8 +3933,20 @@ fn render_children(children: &[ReflowNode], parts: &mut Vec<String>) {
                 }
             }
             ReflowNode::Algorithm { content } => {
-                // Wrap algorithm/code content in fenced code block
-                parts.push(format!("```\n{content}\n```"));
+                // Render algorithm with preserved newlines and indentation.
+                // NOT fenced code blocks (``` kills $...$ LaTeX math).
+                // Trailing "  " on each line = markdown <br> line break.
+                let formatted: Vec<String> = content
+                    .lines()
+                    .map(|line| {
+                        if line.trim().is_empty() {
+                            String::new()
+                        } else {
+                            format!("{line}  ")
+                        }
+                    })
+                    .collect();
+                parts.push(formatted.join("\n"));
             }
             ReflowNode::FigureGroup { path, caption } => {
                 if path.is_empty() {