feat(docx): detect code blocks via paragraph styles and monospaced fonts#3735
feat(docx): detect code blocks via paragraph styles and monospaced fonts#3735LucasArray wants to merge 6 commits into
Conversation
Signed-off-by: Lucas Araujo <29403436+LucasArray@users.noreply.github.com>
Signed-off-by: Lucas Araujo <29403436+LucasArray@users.noreply.github.com>
Signed-off-by: Lucas Araujo <29403436+LucasArray@users.noreply.github.com>
Signed-off-by: Lucas Araujo <29403436+LucasArray@users.noreply.github.com>
Signed-off-by: Lucas Araujo <29403436+LucasArray@users.noreply.github.com>
|
✅ DCO Check Passed Thanks @LucasArray, all your commits are properly signed off. 🎉 |
Merge Protections🔴 1 of 2 protections blocking · waiting on 👀 reviews
🔴 Require two reviewer for test updatesWaiting for
This rule is failing.When test data is updated, we require two reviewers
Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
📄 Knowledge review✏️ Suggested updates1 page suggestion needs review.
📝 What are the detailed pipeline options and processing behaviors for PDF, DOCX, PPTX, and XLSX files in the Python SDK?@@ -122,6 +122,7 @@
- **Header/Footer Export**: Only supported via Python API by setting `included_content_layers={ContentLayer.BODY, ContentLayer.FURNITURE}`; default export excludes header/footer
- **Processing**:
- **Checkboxes**: DOCX checkboxes are automatically detected and parsed using Word 2010+ XML elements (w14:checkbox). Checkbox text is labeled as `CHECKBOX_SELECTED` (for checked checkboxes) or `CHECKBOX_UNSELECTED` (for unchecked checkboxes). Common checkbox symbols (☐, ☑, ☒, etc.) are automatically removed from the text content. This feature works automatically without any configuration required.
+ - **Code Blocks**: Docling detects code blocks through paragraph styles (like "Source Code", "Code Block") and monospaced fonts (like Consolas, Courier). Consecutive code paragraphs merge into single `CodeItem`s, preserving multi-line code structure and indentation.
- **Multiple Equations in Paragraphs**: When a DOCX paragraph contains multiple sibling OMML equations (e.g., multiple `<m:oMath>` elements), each equation is extracted as a separate `FORMULA` item in the document structure. This applies to both:
- **Standalone equation paragraphs**: Paragraphs containing only equations (no surrounding text) produce multiple separate `FORMULA` items, one for each equation
- **Inline equations**: Multiple equations within text-containing paragraphs are preserved as distinct formula items |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Signed-off-by: Lucas Araujo <29403436+LucasArray@users.noreply.github.com>
ceberam
left a comment
There was a problem hiding this comment.
Thanks @LucasArray for your contribution. This is an interesting new feature for docx documents.
Here is some initial feedback. I have also added other inline comments.
- We have recently done some changes in the imports of the declarative backends. It would be good to rebase your branch to
main. That would resolve the existing conflict. - Do not create a new test module for this particular new feature. Reuse the
test_backend_msword.pymodule. You can then leverage the fixtures that are already created. - Do we have any evidence of large, publicly available collections of
docxthat include code snippets in monospaced fonts? I'm just trying to figure out if we could sustain many of the heuristics you include in the backend with real documents.
| code_item = doc.add_code( | ||
| text=code_text, | ||
| orig=code_text, | ||
| parent=parent, | ||
| content_layer=self.content_layer, | ||
| ) |
There was a problem hiding this comment.
The function doc.add_code() accepts a code_language parameter. This PR omits code_language entirely, leaving every DOCX code block with code_language = UNKNOWN.
After the merge-or-create decision, you could call detect_code_language with the final text and pass it to the doc.add_code() function. Check how the HTML backend and the Markdown backend both call detect_code_language(text).
You could even leverage this function as a code signal too, not just for labeling. If detect_code_language returns anything other than UNKNOWN, the paragraph is definitively code.
| _CODE_DEF_PATTERN: Final[re.Pattern[str]] = re.compile( | ||
| r"^(?:async\s+)?" | ||
| r"(?:def|class|if|elif|while|for|with|except|catch|switch" | ||
| r"|function|func|fn|sub|proc)" | ||
| r"\s+[\w.]+\(.*\)\s*:$" | ||
| ) |
There was a problem hiding this comment.
This regex successfully matches some code signals as block headers, like def fib(n): but other common patterns like for x in range(n): would be missed. Please, check my previous comment, since detect_code_language could help.
| font = getattr(style, "font", None) | ||
| font_name = getattr(font, "name", None) |
There was a problem hiding this comment.
Style comment: the getattr usage is not fully justified. These are not "malformed basedOn chain" situations, they are normal attributes on every ParagraphStyle in python-docx. Please, use direct attribute access, with None check if necessary (already done for style).
Refer to the code standards section in AGENTS.md. If you are using an AI coding agent, make sure it leverages this file.
Issue resolved by this Pull Request:
Resolves #1945
Description
This PR adds code-block detection to the DOCX backend. Paragraphs that carry a code style, or that are set in a monospaced font and look like code, are now emitted as
CodeItems, and consecutive code paragraphs merge into a single block. Previously every code line came out as a plaintextparagraph, so multi-line snippets were shredded and lost their structure.Detection follows the two signals proposed in #1945:
Source Code,Code,Verbatim,HTML Preformatted, ...), matched exactly (case-folded) through thebasedOninheritance chain, the same way pandoc's docx reader recognizes code. Ambiguous caption-style names likeListingare deliberately excluded.{};=<>beyond a lone semicolon, a call shape likeset()orprint(sys.argv), or a keyword-led definition line likedef fib(n):. It never fires in headers/footers, table cells, list items or caption-style paragraphs, and the document default font does not count as evidence, so Courier-set letters, forms, contracts and screenplays stay text.Changes
_is_code_style,_is_code_by_fontand friends): the two tiers above, private to the backend, with the paragraph style resolved once per paragraph (see Performance).CodeItemonly when it is the parent's last child, the most recent text item, and in the same content layer, so pictures, tables, headings, resumed lists and header/footer boundaries all end the block. Interior blank lines are preserved; trailing blank paragraphs are dropped.code_languagestays unset: language detection is owned by Add code language detection #2733 / feat: add code language detection #3721; theCodeFormulaModelenrichment can fill it in later.CodeItem. Working on this surfaced a pre-existing crash (anUnboundLocalErrorwhen a rich-flagged cell walk emits no items, silently swallowed as "could not parse a table"), fixed in its own first commit since the new clause makes that path more reachable.Testing
tests/data/docx/sources/docx_code_blocks.docxwith regenerated groundtruth (DOCLING_GEN_TEST_DATA=1): styled and font-only positives, merged multi-line blocks with indentation, and negative cases (inline mono word,Source Referencebibliography style, pure-Courier prose). I am aware reference-data changes require two approvals.tests/test_backend_msword_code.py: regression tests built with python-docx covering merge boundaries (pictures, 1x1 furniture tables, blank lines), style-name traps (Barcode,Unicode,Area Code,Listingcaptions), font-tier negatives (Courier letterheads, bracket-checkbox forms, typewriter contracts, monospace-default documents, proportional tracked insertions), list interactions (lists closing before code, lists resuming after code, straynumPr), style inheritance, table cells, and header/footer isolation.ruff,ty,tach,dprintanduv lock --lockedare clean.Performance
The paragraph style is resolved once per paragraph and the cheap checks run first, so the hot path gets faster rather than slower: on a 6000-paragraph code-free document the backend runs about 20% faster than
main(11.8s vs 15.0s, min of 6 interleaved runs on identical fixtures), and about 14% faster on a code-heavy document.Checklist: