Skip to content

Commit 48ebfe8

Browse files
committed
feat: migrate formula recognition to GLM-OCR
- Remove PP-FormulaNet predictor and associated Python/Rust code - Consolidate formula and table recognition on GLM-OCR - Update documentation and build configurations - Remove unused `bench_formulas` binary and related scripts
1 parent bdb32b4 commit 48ebfe8

26 files changed

Lines changed: 107 additions & 4169 deletions

AGENTS.md

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -39,10 +39,6 @@ should contain a `README.md` with findings and the raw `results.json` output.
3939
cargo run --release --bin run_glm_ocr -- data/dumps/<paper> \
4040
-o .temp/results/<paper>-glm \
4141
--region-type "Text,DisplayFormula,Algorithm,Table"
42-
43-
# PP-FormulaNet (formulas only)
44-
cargo run --release --bin bench_formulas -- data/dumps/<paper> \
45-
-o .temp/results/<paper>-ppformula
4642
```
4743

4844
3. **Benchmark mode** (`--bench`) runs each image multiple times (default 2)

crates/papers-extract/AGENTS.md

Lines changed: 8 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@
55
Local, pure-Rust PDF processing pipeline using ONNX models and `pdfium-render`.
66
Converts PDFs into structured JSON + Markdown + extracted images. Uses PP-DocLayoutV3 for
77
25-class layout detection with built-in reading order via direct `ort` inference.
8-
Formula recognition uses custom split encoder/decoder ONNX models with CUDA EP.
9-
Table recognition uses `oar-ocr` (SLANet).
8+
Formula recognition uses GLM-OCR (vision-language model with GQA-fused decoder).
9+
Table recognition uses TableFormer or GLM-OCR.
1010

1111
## Architecture
1212

@@ -47,8 +47,8 @@ PDF Input
4747
| `types.rs` | `ExtractionResult`, `Page`, `Region`, `RegionKind` (24 variants), `Metadata` |
4848
| `error.rs` | `ExtractError` enum |
4949
| `layout.rs` | `LayoutDetector` — direct ONNX inference on PP-DocLayoutV3, `DetectedRegion` |
50-
| `formula.rs` | `FormulaPredictor`custom CUDA formula predictor using split encoder/decoder FP16 ONNX models. Persistent IoBinding with pre-allocated GPU buffers + CUDA graphs on decoder + cudarc D2D/H2D memcpy for zero-allocation decoding |
51-
| `pipeline.rs` | `Pipeline` struct — owns pdfium + LayoutDetector + FormulaPredictor + TableStructureRecognitionPredictor, orchestrates per-page processing |
50+
| `glm_ocr/` | `GlmOcrPredictor`GLM-OCR with CUDA (IoBinding + CUDA graphs + GQA), CoreML, and CPU backends |
51+
| `pipeline.rs` | `Pipeline` struct — owns pdfium + LayoutDetector + GlmOcrPredictor + TableFormerPredictor, orchestrates per-page processing |
5252
| `models.rs` | Model download from GitHub releases, predictor + LayoutDetector builders, execution provider config |
5353
| `pdf.rs` | `PdfChar`, `load_pdfium()`, `render_page()`, `extract_page_chars()` |
5454
| `text.rs` | Match pdfium characters to layout regions, reconstruct text with word/paragraph detection. Splices inline formula LaTeX (`$...$`) at correct spatial positions, excluding pdfium chars under formula bboxes. Converts PdfChar Y-up coords to image Y-down space. Also provides `try_extract_inline_formula()` for char-based bypass of simple inline formulas. |
@@ -72,7 +72,7 @@ Seal, Chart, SidebarText
7272
Content routing by kind:
7373
- **Text-bearing**`text` field (pdfium char extraction + inline formula splicing)
7474
- **Table**`html` field (oar-ocr SLANet-Plus)
75-
- **DisplayFormula**`latex` field (custom FormulaPredictor), rendered as `$$...$$`
75+
- **DisplayFormula**`latex` field (GLM-OCR), rendered as `$$...$$`
7676
- **InlineFormula** → char-based bypass if all chars are known LaTeX tokens; otherwise ML OCR. Merged into parent text region as `$...$`; orphans emitted as standalone `$...$` regions
7777
- **Visual** (Image/Chart/Seal) → `image_path` field (cropped PNG)
7878
- **Caption**`text` field + associated with parent via `caption`
@@ -82,7 +82,7 @@ Content routing by kind:
8282
- **Fast** (default): SLANet-Plus (7 MB) — fast table recognition
8383
- **Quality**: PP-LCNet classifier (6.5 MB) + SLANeXt-wired (351 MB) — better accuracy for complex tables
8484

85-
Formula recognition uses the custom split encoder/decoder models (~365 MB total). Simple inline formulas (single variables, Greek letters, basic sub/superscripts) are bypassed via char-based extraction from the PDF text layer, avoiding ML inference.
85+
Formula recognition uses GLM-OCR (~3.3 GB total ONNX models). Simple inline formulas (single variables, Greek letters, basic sub/superscripts) are bypassed via char-based extraction from the PDF text layer, avoiding ML inference.
8686

8787
## Inline Formula Char-Based Bypass
8888

@@ -122,9 +122,8 @@ Formula recognition uses the custom split encoder/decoder models (~365 MB total)
122122

123123
## Model Management
124124

125-
Layout/table models auto-download from `github.com/GreatV/oar-ocr/releases` on first use.
126-
Formula models (`encoder_fp16.onnx`, `decoder_fp16_argmax.onnx`) must be pre-exported via
127-
`py/pp-formulanet/cuda/export.py` and placed in the model cache directory.
125+
Layout model auto-downloads from `github.com/GreatV/oar-ocr/releases` on first use.
126+
GLM-OCR models must be pre-exported via `py/glm-ocr/cuda/export.py` and placed in the model cache directory.
128127
Cache directory: `{dirs::cache_dir()}/papers/models/` (override: `PAPERS_MODEL_DIR` env var).
129128

130129
## Execution Providers

crates/papers-extract/Cargo.toml

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,10 +16,6 @@ path = "src/bin/dump_chars.rs"
1616
name = "find_gaps"
1717
path = "src/bin/find_gaps.rs"
1818

19-
[[bin]]
20-
name = "bench_formulas"
21-
path = "src/bin/bench_formulas.rs"
22-
2319
[[bin]]
2420
name = "run_glm_ocr"
2521
path = "src/bin/run_glm_ocr.rs"

0 commit comments

Comments
 (0)