fix: emit glyph marker instead of fabricated encoding text for symbolic-font cmap misses

[wittjeff]

Fixes the text-extraction corruption reported in docling-project/docling#3802 (Arabic في coming out as #). Full root-cause analysis: docling-project/docling#3802 (comment)

Problem

When a character code is not covered by /Encoding/Differences or the font's /ToUnicode CMap, get_correct_character() falls through to get_character_from_encoding(), which resolves the code through the declared base encoding and then StandardEncoding. For symbolic fonts (e.g. subsetted Arabic fonts, whose codes are assigned sequentially from 0x21 and bear no relation to Latin encodings) this fabricates ASCII text: an Arabic ligature glyph subsetted at code 0x23 whose ToUnicode entry the generator omitted — a common generator bug for one-glyph-to-many-codepoints ligatures — comes out as a literal #.

Because the fabricated character looks like plausible text, downstream consumers cannot detect the corruption (docling's rate_text_quality() gate specifically watches for GLYPH<...> markers, which never appear on this path).

Fix

The unmapped-code fallback call sites in get_correct_character() are routed through a new resolve_unmapped_character(): when an explicit /ToUnicode (or predefined CID) CMap exists but does not cover the code, and the code cannot meaningfully resolve through the standard Latin encodings — the font is symbolic (/FontDescriptor /Flags bit 3) or composite (Type0, whose codes are CIDs) — it emits the existing GLYPH<code> marker instead of fabricating text from the encoding tables.

• The guard applies only to the encoding-table fallbacks. Authoritative sources stay ahead of it and are untouched: /Differences, the cmaps, and the known base-font tables (fm.has(c) — e.g. TeX math fonts resolving ∑).
• Non-symbolic simple fonts keep the current encoding fallback unchanged (a partial ToUnicode on a Latin font still resolves the rest through WinAnsi/Standard, where e.g. 0x23 genuinely is #).
• Fonts with no ToUnicode at all are unchanged.
• With default decoding (keep_glyphs=False) the marker is already rendered as a space, so consumers see whitespace instead of a wrong character; with keep_glyphs=True the marker is visible and can trigger OCR fallback logic.
• /FontDescriptor /Flags parsing is hoisted into init_font_flags() and reused by build_embedded_font_blob(), which previously parsed it inline.

Tests

tests/test_tounicode_fallback.py builds the PDFs in memory (same pattern as test_standard_font_widths.py): a simple TrueType font, text bytes 0x21 0x22 0x23, ToUnicode covering 0x21/0x22:

variant	before	after
symbolic, ToUnicode omits 0x23	# (fabricated)	GLYPH<35>
symbolic, ToUnicode maps <23> <0641064A>	في	في (unchanged)
non-symbolic, ToUnicode omits 0x23	#	# (unchanged)

The reference regression suite (test_reference_documents_from_filenames) passes unchanged, and a real-world Arabic PDF with complete ToUnicode (UAE labor law, ADJD mirror) extracts identically before/after.

🤖 Generated with Claude Code

[github-actions]

✅ DCO Check Passed

Thanks @wittjeff, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?: