Skip to content

Fix line number recognition for sparse and accumulated numbering - #241

Open
lfoppiano wants to merge 1 commit into
masterfrom
fix/sparse-line-number-recognition
Open

Fix line number recognition for sparse and accumulated numbering#241
lfoppiano wants to merge 1 commit into
masterfrom
fix/sparse-line-number-recognition

Conversation

@lfoppiano

@lfoppiano lfoppiano commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

TextPage::markLineNumber() misses valid line-number columns in two layouts the size-scaled
coverage threshold from #226 does not cover — and a naive fix wrongly strips footnote/reference
numbers (which also accumulate to large values at a margin). This recovers the missed line numbers
while leaving footnote/reference numbers intact, using a document-level guard.

Detection (low-coverage escape hatch)

Accept a margin cluster as line numbers when its values form a regular arithmetic progression
(dominant constant increment) at the extreme page margin, via either:

  • big numbers — the smallest value exceeds the page's line count: accumulated numbering that a
    figure or column break may split (e.g. 428..444 on a late page), or
  • even vertical pitch — fresh sparse numbering such as every 5th line (229336v1: coverage
    ~0.17 vs the required 0.5, so 242 margin numbers used to survive).

Footnote guard (document level)

Footnote / bibliographic reference numbers also accumulate to large values at a margin, but their
block sits at the bottom of the page and never covers it, whereas a line-number column reaches
the top of the page's text on some page and runs down every page. So a low-coverage column is
accepted only if it reaches the top of the text here, or was already confirmed as a line-number
column on an earlier page
(2-column pages and figures push the numbers down, so a confirmed
column may legitimately start mid-page). Confirmed column x-positions are carried across pages via
XmlAltoOutputDev. Per-page geometry alone cannot separate the two — a figure above the text
pushes line numbers into the same vertical band as footnotes; the document-level view does.

Validation

Measured on the GROBID end-to-end corpora and the grobid-footnote-flavour corpus, diffing
pdfalto output unpatched vs patched.

Recall — line numbers stripped (ALTO token level):

corpus docs docs stripped tokens over-strip negative deltas
bioRxiv-10k-test-2000 2000 468 (23%) 17,570 0 0
bioRxiv-10k-train-6000 6000 1,655 (28%) 59,165 0 0

Only digit tokens are removed (verified token-by-token on the heaviest strips). Vs the un-guarded
version, 94% of recall retained on test-2000 and 97% on train-6000; the residual are
2-column/figure docs whose line-number column never reaches the page top on any page (inherently
ambiguous with footnotes).

Precision — nothing that isn't a line number is removed:

Stress-tested against the full mpilhlt/grobid-footnote-flavour
corpus — 100 two-column, footnote-heavy legal/humanities PDFs (all 12 batches), the exact layout
where footnote/reference numbers accumulate to large values at a margin and look like line numbers:

corpus docs docs changed (before guard → after)
grobid-footnote-flavour (all batches) 100 2 → 0 — no footnote/page numbers stripped
  • eLife/PLOS/PMC (no line numbers): 0 changed, unchanged.
  • Figure-panel labels (005124v1, inset 1..6) preserved.

Notes

Conservative constants: increment 1–50, pitch CV < 0.30, margin within 6% of page width, "reaches
top" = topmost number within the top 40% of the page's text, confirmed-column x tolerance 15 pt.

@lfoppiano
lfoppiano force-pushed the fix/sparse-line-number-recognition branch from 592dc60 to 04262d1 Compare July 16, 2026 09:01
@lfoppiano lfoppiano added this to the 0.7.0 milestone Jul 19, 2026
…n't strip footnotes

markLineNumber() missed valid line-number columns in two layouts the size-scaled coverage
threshold from #226 does not cover, and a naive fix would strip footnote/reference numbers.

Detection (low-coverage escape hatch): accept a margin cluster as line numbers when its values
form a regular arithmetic progression (dominant constant increment) at the extreme page margin,
via either
  - "big numbers": the smallest value exceeds the page's line count (accumulated numbering that
    a figure or column break may split, e.g. 428..444 on a late page), or
  - even vertical pitch (fresh sparse numbering such as every 5th line).

Footnote guard (document level): footnote and bibliographic reference numbers also accumulate to
large values at a margin, but their block sits at the bottom of the page and never covers it,
whereas a line-number column reaches the top of the page's text on some page and runs down every
page. So a low-coverage column is accepted only if it reaches the top of the text here, or was
already confirmed as a line-number column on an earlier page (2-column pages and figures push the
numbers down, so a confirmed column may start mid-page). Confirmed column x-positions are carried
across pages via XmlAltoOutputDev.

Validated on the GROBID end-to-end corpora and the grobid-footnote-flavour corpus:
- bioRxiv line numbers stripped on ~24-28% of docs (test-2000 and train-6000), removing only
  digit tokens; 94% of the pre-guard recall retained.
- grobid-footnote-flavour: 0/100 docs changed (no footnote/page numbers stripped).
- eLife/PLOS/PMC (no line numbers): unchanged. Figure panel labels preserved.
@lfoppiano
lfoppiano force-pushed the fix/sparse-line-number-recognition branch from 55695c6 to f023bf0 Compare July 21, 2026 15:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant