Handle glyphs with no Unicode mapping: glyph-index recovery + OCR sidecar

Readme.md

@@ -45,6 +45,7 @@ Usage: pdfalto [options] <PDF-file> [<xml-file>]
   -readingOrder                 : blocks follow the reading order
   -noText                       : do not extract textual objects (might be useful, but non-valid ALTO)
   -charReadingOrderAttr         : include TYPE attribute to String elements to indicate right-to-left reading order (might be useful, but non-valid ALTO)
+  -ocr                          : emit a sidecar listing glyphs with no Unicode mapping (<xml>_data/ocr-regions.json); render + OCR the bounding boxes externally (e.g. pdftoppm).
   -fullFontName                 : fonts names are not normalized
   -nsURI <string>               : add the specified namespace URI
   -opw <string>                 : owner password (for encrypted files)
@@ -74,6 +75,46 @@ files are generated:
   use `-onlyGraphsCoord` (or the deprecated alias `-noImage`) to keep extracting image coordinates without dumping the
   image files. To skip all graphics processing (bitmap and vectorial), use `-skipGraphs`.
 
+### OCR sidecar for glyphs with no Unicode mapping
+
+Some PDFs contain glyphs that have no valid Unicode mapping — typically custom
+Type 3 fonts, symbolic/dingbat fonts with broken `ToUnicode` CMaps, or Private
+Use Area characters. pdfalto first tries to *recover* the real characters
+(including MSTT `GNN`/`gNNN`/`cidNNN` glyph-index naming); `-ocr` only captures
+the *residual* glyphs that recovery genuinely cannot map. For those, pdfalto:
+
+1. Substitutes each glyph with a unique placeholder codepoint allocated
+   sequentially from the Private Use Area (`U+E000` onward) inside the ALTO
+   `<String CONTENT>`, so the output stays valid UTF-8 and each distinct
+   `(fontName, charCode)` gets its own collision-free placeholder.
+2. Writes `<xml>_data/ocr-regions.json` listing every unique `(fontName,
+   charCode)` pair that triggered a placeholder, with a per-occurrence page
+   number and bounding box in page-space points (xpdf bottom-up convention,
+   flip with page height for ALTO top-down).
+
+pdfalto intentionally does not rasterise these regions — this keeps the tool
+dependency-free and lets the caller pick a rendering backend, DPI, and OCR
+engine. A typical pipeline is:
+
+```sh
+pdfalto -ocr in.pdf out.xml
+# render each sidecar occurrence to a bitmap (pdftoppm is one option)
+jq -c '.glyphs[] | .occurrences[] as $o | {placeholder, charCode, page: $o.page, bbox: $o}' \
+    out.xml_data/ocr-regions.json | while read region; do
+  # ... invoke your OCR engine here, then write a corrections JSON ...
+done
+# apply corrections back: replace placeholder codepoints in out.xml
+#   with the recovered Unicode, matched by (fontName, charCode) — the canonical
+#   sidecar key — or by (page, bbox)
+```
+
+Each distinct `(fontName, charCode)` gets its own placeholder, so the codepoint
+alone is a stable key for documents with up to 6400 distinct unmapped glyphs;
+beyond that the allocation saturates, so prefer `(fontName, charCode)` as the
+authoritative key when applying corrections. The placeholder codepoints in the
+sidecar are identical to the ones in the
+ALTO file, so a corrections step can simply substitute them in-place.
+
 ### Extra script to get only text content
 
 The goal of pdfalto is to extract all the content of a PDF, not just text, but also layout, style, font, vector
@@ -136,12 +177,12 @@ building, see described workaround.
 
 ## Future work
 
-- Text like containing block element characters (https://unicode.org/charts/PDF/U2B00.pdf) are used as placeholders for
-  unknown character unicodes, instead of what would be expected when visually inspecting the text. The reason for these
-  unsolved character unicode values is that the actual characters are glyphs that are embedded in the PDF document which
-  use free unicode range for embedded fonts, not the right unicode. The only way to extract the valid text for those
-  special characters is to use OCR at glyph level . This is our targeted main future enhancement, relying on a custom
-  Deep Learning approach.
+- Under `-ocr`, glyphs with no recoverable Unicode are substituted with Private Use Area placeholder codepoints
+  (`U+E000` onward, see the OCR sidecar section above) instead of what would be expected when visually inspecting the
+  text. The reason for these unsolved character unicode values is that the actual characters are glyphs embedded in the
+  PDF document which use a free unicode range for embedded fonts, not the right unicode. The only way to extract the
+  valid text for those special characters is to use OCR at glyph level, driven by the `ocr-regions.json` sidecar. This
+  is our targeted main future enhancement, relying on a custom Deep Learning approach.
 
 - map special characters in secondary fonts to their expected unicode
 

languages/xpdf-others/others.nameToUnicode

@@ -0,0 +1,4 @@
+2113 lscript
+2202 partialdiff
+21A9 arrowhookleft
+21A9 arrowrighttophalf

languages/xpdf-others/sc.nameToUnicode

@@ -0,0 +1,27 @@
+0061 a.sc
+0062 b.sc
+0063 c.sc
+0064 d.sc
+0065 e.sc
+0066 f.sc
+0067 g.sc
+0068 h.sc
+0069 i.sc
+006a j.sc
+006b k.sc
+006c l.sc
+006d m.sc
+006e n.sc
+006f o.sc
+0070 p.sc
+0071 q.sc
+0072 r.sc
+0073 s.sc
+0074 t.sc
+0075 u.sc
+0076 v.sc
+0077 w.sc
+0078 x.sc
+0079 y.sc
+007a z.sc
+002d hyphen.sc

languages/xpdf-others/taboldstyle.nameToUnicode

@@ -1 +1,10 @@
-0038 eight.taboldstyle
+0030 zero.taboldstyle
+0031 one.taboldstyle
+0032 two.taboldstyle
+0033 three.taboldstyle
+0034 four.taboldstyle
+0035 five.taboldstyle
+0036 six.taboldstyle
+0037 seven.taboldstyle
+0038 eight.taboldstyle
+0039 nine.taboldstyle