P1M_benchmarks.md

Pub1M Benchmarks

Create Pub1M evaluation datasets. This downloads from Huggingface the Pub1M_OTSL and runs the evaluations for TableFormer using the first 1000 samples.

# Make the ground-truth
docling-eval create-gt --benchmark Pub1M --output-dir ./benchmarks/Pub1M/ 

# Make predictions for tables.
docling-eval create-eval \
  --benchmark DPBench \
  --output-dir ./benchmarks/Pub1M/ \
  --end-index 1000 \
  --prediction-provider TableFormer # use tableformer predictions only

Tableformer Evaluation

Create the evaluation report:

docling-eval evaluate \
  --modality table_structure \
  --benchmark Pub1M \
  --output-dir ./benchmarks/Pub1M/ 

Tableformer evaluation json

Visualize the report:

docling-eval visualize \
  --modality table_structure \
  --benchmark Pub1M \
  --output-dir ./benchmarks/Pub1M/ 

TEDS struct only report

TEDS struct with text report