Fix line number recognition for sparse and accumulated numbering - #241
Open
lfoppiano wants to merge 1 commit into
Open
Fix line number recognition for sparse and accumulated numbering#241lfoppiano wants to merge 1 commit into
lfoppiano wants to merge 1 commit into
Conversation
lfoppiano
force-pushed
the
fix/sparse-line-number-recognition
branch
from
July 16, 2026 09:01
592dc60 to
04262d1
Compare
…n't strip footnotes markLineNumber() missed valid line-number columns in two layouts the size-scaled coverage threshold from #226 does not cover, and a naive fix would strip footnote/reference numbers. Detection (low-coverage escape hatch): accept a margin cluster as line numbers when its values form a regular arithmetic progression (dominant constant increment) at the extreme page margin, via either - "big numbers": the smallest value exceeds the page's line count (accumulated numbering that a figure or column break may split, e.g. 428..444 on a late page), or - even vertical pitch (fresh sparse numbering such as every 5th line). Footnote guard (document level): footnote and bibliographic reference numbers also accumulate to large values at a margin, but their block sits at the bottom of the page and never covers it, whereas a line-number column reaches the top of the page's text on some page and runs down every page. So a low-coverage column is accepted only if it reaches the top of the text here, or was already confirmed as a line-number column on an earlier page (2-column pages and figures push the numbers down, so a confirmed column may start mid-page). Confirmed column x-positions are carried across pages via XmlAltoOutputDev. Validated on the GROBID end-to-end corpora and the grobid-footnote-flavour corpus: - bioRxiv line numbers stripped on ~24-28% of docs (test-2000 and train-6000), removing only digit tokens; 94% of the pre-guard recall retained. - grobid-footnote-flavour: 0/100 docs changed (no footnote/page numbers stripped). - eLife/PLOS/PMC (no line numbers): unchanged. Figure panel labels preserved.
lfoppiano
force-pushed
the
fix/sparse-line-number-recognition
branch
from
July 21, 2026 15:41
55695c6 to
f023bf0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TextPage::markLineNumber()misses valid line-number columns in two layouts the size-scaledcoverage threshold from #226 does not cover — and a naive fix wrongly strips footnote/reference
numbers (which also accumulate to large values at a margin). This recovers the missed line numbers
while leaving footnote/reference numbers intact, using a document-level guard.
Detection (low-coverage escape hatch)
Accept a margin cluster as line numbers when its values form a regular arithmetic progression
(dominant constant increment) at the extreme page margin, via either:
figure or column break may split (e.g.
428..444on a late page), or229336v1: coverage~0.17 vs the required 0.5, so 242 margin numbers used to survive).
Footnote guard (document level)
Footnote / bibliographic reference numbers also accumulate to large values at a margin, but their
block sits at the bottom of the page and never covers it, whereas a line-number column reaches
the top of the page's text on some page and runs down every page. So a low-coverage column is
accepted only if it reaches the top of the text here, or was already confirmed as a line-number
column on an earlier page (2-column pages and figures push the numbers down, so a confirmed
column may legitimately start mid-page). Confirmed column x-positions are carried across pages via
XmlAltoOutputDev. Per-page geometry alone cannot separate the two — a figure above the textpushes line numbers into the same vertical band as footnotes; the document-level view does.
Validation
Measured on the GROBID end-to-end corpora and the
grobid-footnote-flavourcorpus, diffingpdfalto output unpatched vs patched.
Recall — line numbers stripped (ALTO token level):
Only digit tokens are removed (verified token-by-token on the heaviest strips). Vs the un-guarded
version, 94% of recall retained on test-2000 and 97% on train-6000; the residual are
2-column/figure docs whose line-number column never reaches the page top on any page (inherently
ambiguous with footnotes).
Precision — nothing that isn't a line number is removed:
Stress-tested against the full
mpilhlt/grobid-footnote-flavourcorpus — 100 two-column, footnote-heavy legal/humanities PDFs (all 12 batches), the exact layout
where footnote/reference numbers accumulate to large values at a margin and look like line numbers:
005124v1, inset1..6) preserved.Notes
Conservative constants: increment 1–50, pitch CV < 0.30, margin within 6% of page width, "reaches
top" = topmost number within the top 40% of the page's text, confirmed-column x tolerance 15 pt.