Skip to content

feat(docx): detect code blocks via paragraph styles and monospaced fonts#3735

Open
LucasArray wants to merge 6 commits into
docling-project:mainfrom
LucasArray:feat/docx-code-detection
Open

feat(docx): detect code blocks via paragraph styles and monospaced fonts#3735
LucasArray wants to merge 6 commits into
docling-project:mainfrom
LucasArray:feat/docx-code-detection

Conversation

@LucasArray

Copy link
Copy Markdown

Issue resolved by this Pull Request:
Resolves #1945

Description

This PR adds code-block detection to the DOCX backend. Paragraphs that carry a code style, or that are set in a monospaced font and look like code, are now emitted as CodeItems, and consecutive code paragraphs merge into a single block. Previously every code line came out as a plain text paragraph, so multi-line snippets were shredded and lost their structure.

Detection follows the two signals proposed in #1945:

  • Style tier (authoritative): a curated set of code style names/ids (Source Code, Code, Verbatim, HTML Preformatted, ...), matched exactly (case-folded) through the basedOn inheritance chain, the same way pandoc's docx reader recognizes code. Ambiguous caption-style names like Listing are deliberately excluded.
  • Font tier (guarded fallback): fires only when at least 90% of the paragraph's characters are set in a monospaced family (including runs inside hyperlinks and tracked insertions) and the text carries a code signal: structural punctuation {};=<> beyond a lone semicolon, a call shape like set() or print(sys.argv), or a keyword-led definition line like def fib(n):. It never fires in headers/footers, table cells, list items or caption-style paragraphs, and the document default font does not count as evidence, so Courier-set letters, forms, contracts and screenplays stay text.

Changes

  • Detection helpers (_is_code_style, _is_code_by_font and friends): the two tiers above, private to the backend, with the paragraph style resolved once per paragraph (see Performance).
  • Block merging: consecutive code paragraphs merge into one CodeItem only when it is the parent's last child, the most recent text item, and in the same content layer, so pictures, tables, headings, resumed lists and header/footer boundaries all end the block. Interior blank lines are preserved; trailing blank paragraphs are dropped.
  • code_language stays unset: language detection is owned by Add code language detection #2733 / feat: add code language detection #3721; the CodeFormulaModel enrichment can fill it in later.
  • Rich table cells: a non-empty code-styled cell is walked as rich content so it can emit a CodeItem. Working on this surfaced a pre-existing crash (an UnboundLocalError when a rich-flagged cell walk emits no items, silently swallowed as "could not parse a table"), fixed in its own first commit since the new clause makes that path more reachable.

Testing

  • New fixture tests/data/docx/sources/docx_code_blocks.docx with regenerated groundtruth (DOCLING_GEN_TEST_DATA=1): styled and font-only positives, merged multi-line blocks with indentation, and negative cases (inline mono word, Source Reference bibliography style, pure-Courier prose). I am aware reference-data changes require two approvals.
  • tests/test_backend_msword_code.py: regression tests built with python-docx covering merge boundaries (pictures, 1x1 furniture tables, blank lines), style-name traps (Barcode, Unicode, Area Code, Listing captions), font-tier negatives (Courier letterheads, bracket-checkbox forms, typewriter contracts, monospace-default documents, proportional tracked insertions), list interactions (lists closing before code, lists resuming after code, stray numPr), style inheritance, table cells, and header/footer isolation.
  • Full msword suite passes; ruff, ty, tach, dprint and uv lock --locked are clean.

Performance

The paragraph style is resolved once per paragraph and the cheap checks run first, so the hot path gets faster rather than slower: on a 6000-paragraph code-free document the backend runs about 20% faster than main (11.8s vs 15.0s, min of 6 interleaved runs on identical fixtures), and about 14% faster on a code-heavy document.

Checklist:

  • Documentation has been updated, if necessary.
  • Examples have been added, if necessary.
  • Tests have been added, if necessary.

Signed-off-by: Lucas Araujo <29403436+LucasArray@users.noreply.github.com>
Signed-off-by: Lucas Araujo <29403436+LucasArray@users.noreply.github.com>
Signed-off-by: Lucas Araujo <29403436+LucasArray@users.noreply.github.com>
Signed-off-by: Lucas Araujo <29403436+LucasArray@users.noreply.github.com>
Signed-off-by: Lucas Araujo <29403436+LucasArray@users.noreply.github.com>
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

DCO Check Passed

Thanks @LucasArray, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🔴 1 of 2 protections blocking · waiting on 👀 reviews

Protection Waiting on
🔴 Require two reviewer for test updates 👀 reviews
🟢 Enforce conventional commit

🔴 Require two reviewer for test updates

Waiting for

  • #approved-reviews-by >= 2
This rule is failing.

When test data is updated, we require two reviewers

  • #approved-reviews-by >= 2

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

@LucasArray LucasArray marked this pull request as ready for review July 2, 2026 16:47
@dosubot

dosubot Bot commented Jul 2, 2026

Copy link
Copy Markdown

📄 Knowledge review

✏️ Suggested updates

1 page suggestion needs review.

Page Library Status
What are the detailed pipeline options and processing behaviors for PDF, DOCX, PPTX, and XLSX files in the Python SDK? Docling 🟡 Review
📝 What are the detailed pipeline options and processing behaviors for PDF, DOCX, PPTX, and XLSX files in the Python SDK?
@@ -122,6 +122,7 @@
     - **Header/Footer Export**: Only supported via Python API by setting `included_content_layers={ContentLayer.BODY, ContentLayer.FURNITURE}`; default export excludes header/footer
 - **Processing**:
     - **Checkboxes**: DOCX checkboxes are automatically detected and parsed using Word 2010+ XML elements (w14:checkbox). Checkbox text is labeled as `CHECKBOX_SELECTED` (for checked checkboxes) or `CHECKBOX_UNSELECTED` (for unchecked checkboxes). Common checkbox symbols (☐, ☑, ☒, etc.) are automatically removed from the text content. This feature works automatically without any configuration required.
+      - **Code Blocks**: Docling detects code blocks through paragraph styles (like "Source Code", "Code Block") and monospaced fonts (like Consolas, Courier). Consecutive code paragraphs merge into single `CodeItem`s, preserving multi-line code structure and indentation.
     - **Multiple Equations in Paragraphs**: When a DOCX paragraph contains multiple sibling OMML equations (e.g., multiple `<m:oMath>` elements), each equation is extracted as a separate `FORMULA` item in the document structure. This applies to both:
         - **Standalone equation paragraphs**: Paragraphs containing only equations (no surrounding text) produce multiple separate `FORMULA` items, one for each equation
         - **Inline equations**: Multiple equations within text-containing paragraphs are preserved as distinct formula items

Accept · Edit · Decline


Leave Feedback Ask Dosu about docling Add Dosu to your team

@PeterStaar-IBM PeterStaar-IBM requested a review from ceberam July 3, 2026 05:48
@codecov

codecov Bot commented Jul 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.37349% with 11 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
docling/backend/msword_backend.py 93.37% 11 Missing ⚠️

📢 Thoughts on this report? Let us know!

Signed-off-by: Lucas Araujo <29403436+LucasArray@users.noreply.github.com>

@ceberam ceberam left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @LucasArray for your contribution. This is an interesting new feature for docx documents.
Here is some initial feedback. I have also added other inline comments.

  • We have recently done some changes in the imports of the declarative backends. It would be good to rebase your branch to main. That would resolve the existing conflict.
  • Do not create a new test module for this particular new feature. Reuse the test_backend_msword.py module. You can then leverage the fixtures that are already created.
  • Do we have any evidence of large, publicly available collections of docx that include code snippets in monospaced fonts? I'm just trying to figure out if we could sustain many of the heuristics you include in the backend with real documents.

Comment on lines +1899 to +1904
code_item = doc.add_code(
text=code_text,
orig=code_text,
parent=parent,
content_layer=self.content_layer,
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function doc.add_code() accepts a code_language parameter. This PR omits code_language entirely, leaving every DOCX code block with code_language = UNKNOWN.
After the merge-or-create decision, you could call detect_code_language with the final text and pass it to the doc.add_code() function. Check how the HTML backend and the Markdown backend both call detect_code_language(text).
You could even leverage this function as a code signal too, not just for labeling. If detect_code_language returns anything other than UNKNOWN, the paragraph is definitively code.

Comment on lines +148 to +153
_CODE_DEF_PATTERN: Final[re.Pattern[str]] = re.compile(
r"^(?:async\s+)?"
r"(?:def|class|if|elif|while|for|with|except|catch|switch"
r"|function|func|fn|sub|proc)"
r"\s+[\w.]+\(.*\)\s*:$"
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This regex successfully matches some code signals as block headers, like def fib(n): but other common patterns like for x in range(n): would be missed. Please, check my previous comment, since detect_code_language could help.

Comment on lines +950 to +951
font = getattr(style, "font", None)
font_name = getattr(font, "name", None)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style comment: the getattr usage is not fully justified. These are not "malformed basedOn chain" situations, they are normal attributes on every ParagraphStyle in python-docx. Please, use direct attribute access, with None check if necessary (already done for style).

Refer to the code standards section in AGENTS.md. If you are using an AI coding agent, make sure it leverages this file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Detect code in docx

2 participants