feat(pdf): recover non-repeating headers/footers mis-tagged as furniture

[DanielNg0729]

Summary

This PR resolves #3693. The layout model labels regions as PAGE_HEADER/PAGE_FOOTER from per-page position alone. It has no notion of whether the text actually repeats across pages. A one-off heading sitting at the top of a page therefore gets tagged as a page header, moved to the FURNITURE content layer by the reading-order model, and silently dropped from the default export (which only emits BODY). On the affected PDF this removes obvious section headings.

This PR adds an opt-in document-level pass, HeaderFooterModel, that runs just before the reading-order model, the first stage with whole-document context, and demotes header/footer detections that are not genuine running furniture back into the body.

Heuristic

A PAGE_HEADER/PAGE_FOOTER detection is kept as furniture when either:

1. Repetition — its normalized text (digits stripped, so "Page 3 of 10" matches "Page 4 of 10") appears on ≥ min_repeat_pages pages (a consistent running header/footer); or
2. Coverage — the side it belongs to (header vs footer) is present on ≥ min_coverage of the pages. A real running header/footer appears on (almost) every page, so a side that only shows up on a few scattered pages is most likely mis-detected content, while a header whose text legitimately changes per section is still trusted because it covers most pages.

Otherwise it is reclassified: a non-repeating header → SECTION_HEADER (so the existing heading-hierarchy step can level it), a non-repeating footer → TEXT. The pass never drops, adds or reorders items, and never touches genuine repeating headers/footers. It only runs on documents with ≥ min_pages pages, since repetition/coverage aren't observable on 1–2 page docs.

Configuration

New HeaderFooterOptions, exposed as PdfPipelineOptions.header_footer_options:

field	default	meaning
enabled	False	turn the recovery pass on
min_pages	3	minimum pages for the pass to run
min_repeat_pages	2	a text repeating on this many pages is genuine furniture
min_coverage	0.5	a side present on this fraction of pages is genuine
promote_headers_to_section	True	recovered header → SECTION_HEADER (else TEXT)

Tests

tests/test_header_footer.py focused unit tests for the heuristic (both sample-PDF scenarios, section-varying headers, short-doc guard, footer/text targets, blank detections, disabled no-op) and the in-place relabeling + bucket move. All pass.

Notes

• Wired into both standard_pdf_pipeline and legacy_standard_pdf_pipeline, before reading order.
• No new dependencies. Public API stays backward compatible (new field has a default).

[github-actions]

✅ DCO Check Passed

Thanks @DanielNg0729, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[codecov]

Codecov Report

❌ Patch coverage is 97.50000% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...models/stages/header_footer/header_footer_model.py	97.01%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[DanielNg0729]

Hi Dr @PeterStaar-IBM, I've resolved the merge conflict. This is ready for review whenever you have a moment. Thank you very much!

[PeterStaar-IBM]

Hi Dr @PeterStaar-IBM, I've resolved the merge conflict. This is ready for review whenever you have a moment. Thank you very much!

re-running the CI.

[DanielNg0729]

Hi Dr. @PeterStaar-IBM Dr. @cau-git, just a gentle reminder that this PR is ready for review whenever you have the chance. Thank you very much!