-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy path06-technical-report.html
More file actions
3058 lines (3025 loc) · 424 KB
/
Copy path06-technical-report.html
File metadata and controls
3058 lines (3025 loc) · 424 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"><head>
<meta charset="utf-8">
<meta name="generator" content="quarto-1.5.57">
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
<title>7 Technical Report – Portfolio</title>
<style>
code{white-space: pre-wrap;}
span.smallcaps{font-variant: small-caps;}
div.columns{display: flex; gap: min(4vw, 1.5em);}
div.column{flex: auto; overflow-x: auto;}
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
ul.task-list{list-style: none;}
ul.task-list li input[type="checkbox"] {
width: 0.8em;
margin: 0 0.8em 0.2em -1em; /* quarto-specific, see https://github.com/quarto-dev/quarto-cli/issues/4556 */
vertical-align: middle;
}
/* CSS for syntax highlighting */
pre > code.sourceCode { white-space: pre; position: relative; }
pre > code.sourceCode > span { line-height: 1.25; }
pre > code.sourceCode > span:empty { height: 1.2em; }
.sourceCode { overflow: visible; }
code.sourceCode > span { color: inherit; text-decoration: inherit; }
div.sourceCode { margin: 1em 0; }
pre.sourceCode { margin: 0; }
@media screen {
div.sourceCode { overflow: auto; }
}
@media print {
pre > code.sourceCode { white-space: pre-wrap; }
pre > code.sourceCode > span { display: inline-block; text-indent: -5em; padding-left: 5em; }
}
pre.numberSource code
{ counter-reset: source-line 0; }
pre.numberSource code > span
{ position: relative; left: -4em; counter-increment: source-line; }
pre.numberSource code > span > a:first-child::before
{ content: counter(source-line);
position: relative; left: -1em; text-align: right; vertical-align: baseline;
border: none; display: inline-block;
-webkit-touch-callout: none; -webkit-user-select: none;
-khtml-user-select: none; -moz-user-select: none;
-ms-user-select: none; user-select: none;
padding: 0 4px; width: 4em;
}
pre.numberSource { margin-left: 3em; padding-left: 4px; }
div.sourceCode
{ }
@media screen {
pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; }
}
</style>
<script src="site_libs/quarto-nav/quarto-nav.js"></script>
<script src="site_libs/quarto-nav/headroom.min.js"></script>
<script src="site_libs/clipboard/clipboard.min.js"></script>
<script src="site_libs/quarto-search/autocomplete.umd.js"></script>
<script src="site_libs/quarto-search/fuse.min.js"></script>
<script src="site_libs/quarto-search/quarto-search.js"></script>
<meta name="quarto:offset" content="./">
<link href="./summary.html" rel="next">
<link href="./05-aim3.html" rel="prev">
<link href="./pic/cover.png" rel="icon" type="image/png">
<script src="site_libs/quarto-html/quarto.js"></script>
<script src="site_libs/quarto-html/popper.min.js"></script>
<script src="site_libs/quarto-html/tippy.umd.min.js"></script>
<script src="site_libs/quarto-html/anchor.min.js"></script>
<link href="site_libs/quarto-html/tippy.css" rel="stylesheet">
<link href="site_libs/quarto-html/quarto-syntax-highlighting.css" rel="stylesheet" id="quarto-text-highlighting-styles">
<script src="site_libs/bootstrap/bootstrap.min.js"></script>
<link href="site_libs/bootstrap/bootstrap-icons.css" rel="stylesheet">
<link href="site_libs/bootstrap/bootstrap.min.css" rel="stylesheet" id="quarto-bootstrap" data-mode="light">
<script id="quarto-search-options" type="application/json">{
"location": "sidebar",
"copy-button": false,
"collapse-after": 3,
"panel-placement": "start",
"type": "textbox",
"limit": 50,
"keyboard-shortcut": [
"f",
"/",
"s"
],
"show-item-context": false,
"language": {
"search-no-results-text": "No results",
"search-matching-documents-text": "matching documents",
"search-copy-link-title": "Copy link to search",
"search-hide-matches-text": "Hide additional matches",
"search-more-match-text": "more match in this document",
"search-more-matches-text": "more matches in this document",
"search-clear-button-title": "Clear",
"search-text-placeholder": "",
"search-detached-cancel-button-title": "Cancel",
"search-submit-button-title": "Submit",
"search-label": "Search"
}
}</script>
<style>html{ scroll-behavior: smooth; }</style>
<script src="https://cdnjs.cloudflare.com/polyfill/v3/polyfill.min.js?features=es6"></script>
<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-chtml-full.js" type="text/javascript"></script>
<script type="text/javascript">
const typesetMath = (el) => {
if (window.MathJax) {
// MathJax Typeset
window.MathJax.typeset([el]);
} else if (window.katex) {
// KaTeX Render
var mathElements = el.getElementsByClassName("math");
var macros = [];
for (var i = 0; i < mathElements.length; i++) {
var texText = mathElements[i].firstChild;
if (mathElements[i].tagName == "SPAN") {
window.katex.render(texText.data, mathElements[i], {
displayMode: mathElements[i].classList.contains('display'),
throwOnError: false,
macros: macros,
fleqn: false
});
}
}
}
}
window.Quarto = {
typesetMath
};
</script>
</head>
<body class="nav-sidebar docked">
<div id="quarto-search-results"></div>
<header id="quarto-header" class="headroom fixed-top">
<nav class="quarto-secondary-nav">
<div class="container-fluid d-flex">
<button type="button" class="quarto-btn-toggle btn" data-bs-toggle="collapse" role="button" data-bs-target=".quarto-sidebar-collapse-item" aria-controls="quarto-sidebar" aria-expanded="false" aria-label="Toggle sidebar navigation" onclick="if (window.quartoToggleHeadroom) { window.quartoToggleHeadroom(); }">
<i class="bi bi-layout-text-sidebar-reverse"></i>
</button>
<nav class="quarto-page-breadcrumbs" aria-label="breadcrumb"><ol class="breadcrumb"><li class="breadcrumb-item"><a href="./06-technical-report.html"><span class="chapter-number">7</span> <span class="chapter-title">Technical Report</span></a></li></ol></nav>
<a class="flex-grow-1" role="navigation" data-bs-toggle="collapse" data-bs-target=".quarto-sidebar-collapse-item" aria-controls="quarto-sidebar" aria-expanded="false" aria-label="Toggle sidebar navigation" onclick="if (window.quartoToggleHeadroom) { window.quartoToggleHeadroom(); }">
</a>
<button type="button" class="btn quarto-search-button" aria-label="Search" onclick="window.quartoOpenSearch();">
<i class="bi bi-search"></i>
</button>
</div>
</nav>
</header>
<!-- content -->
<div id="quarto-content" class="quarto-container page-columns page-rows-contents page-layout-article">
<!-- sidebar -->
<nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal quarto-sidebar-collapse-item sidebar-navigation docked overflow-auto">
<div class="pt-lg-2 mt-2 text-left sidebar-header">
<div class="sidebar-title mb-0 py-0">
<a href="./">Portfolio</a>
<div class="sidebar-tools-main">
<a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
<div class="quarto-reader-toggle-btn">
<i class="bi"></i>
</div>
</a>
</div>
</div>
</div>
<div class="mt-2 flex-shrink-0 align-items-center">
<div class="sidebar-search">
<div id="quarto-search" class="" title="Search"></div>
</div>
</div>
<div class="sidebar-menu-container">
<ul class="list-unstyled mt-1">
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./index.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Preface</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./intro.html" class="sidebar-item-text sidebar-link">
<span class="menu-text"><span class="chapter-number">1</span> <span class="chapter-title">Introduction</span></span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./01-proposal.html" class="sidebar-item-text sidebar-link">
<span class="menu-text"><span class="chapter-number">2</span> <span class="chapter-title">Literature Analytics and Methodological Gap Analysis in Geospatial Epidemiology</span></span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./02-data-collection.html" class="sidebar-item-text sidebar-link">
<span class="menu-text"><span class="chapter-number">3</span> <span class="chapter-title">Data Collection</span></span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./03-literature-review.html" class="sidebar-item-text sidebar-link">
<span class="menu-text"><span class="chapter-number">4</span> <span class="chapter-title">Literature Review</span></span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./04-prelim-analysis.html" class="sidebar-item-text sidebar-link">
<span class="menu-text"><span class="chapter-number">5</span> <span class="chapter-title">Preliminary Analysis</span></span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./05-aim3.html" class="sidebar-item-text sidebar-link">
<span class="menu-text"><span class="chapter-number">6</span> <span class="chapter-title">Proposed Deep Learning Framework for UOG Site Monitoring (Aim 3)</span></span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./06-technical-report.html" class="sidebar-item-text sidebar-link active">
<span class="menu-text"><span class="chapter-number">7</span> <span class="chapter-title">Technical Report</span></span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./summary.html" class="sidebar-item-text sidebar-link">
<span class="menu-text"><span class="chapter-number">8</span> <span class="chapter-title">Summary</span></span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./references.html" class="sidebar-item-text sidebar-link">
<span class="menu-text"><span class="chapter-number">9</span> <span class="chapter-title">References</span></span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./update.html" class="sidebar-item-text sidebar-link">
<span class="menu-text">Project Update</span></a>
</div>
</li>
<li class="sidebar-item sidebar-item-section">
<div class="sidebar-item-container">
<a class="sidebar-item-text sidebar-link text-start" data-bs-toggle="collapse" data-bs-target="#quarto-sidebar-section-1" role="navigation" aria-expanded="true">
<span class="menu-text">Appendices</span></a>
<a class="sidebar-item-toggle text-start" data-bs-toggle="collapse" data-bs-target="#quarto-sidebar-section-1" role="navigation" aria-expanded="true" aria-label="Toggle section">
<i class="bi bi-chevron-right ms-2"></i>
</a>
</div>
<ul id="quarto-sidebar-section-1" class="collapse list-unstyled sidebar-section depth1 show">
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./appendix-a-etl-rag.html" class="sidebar-item-text sidebar-link">
<span class="menu-text"><span class="chapter-number">A</span> <span class="chapter-title">appendix-a-etl-rag.html</span></span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./appendix-b-lca-rf.html" class="sidebar-item-text sidebar-link">
<span class="menu-text"><span class="chapter-number">B</span> <span class="chapter-title">Preliminary LCA Cover crop Analysis</span></span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./appendix-c-hardware.html" class="sidebar-item-text sidebar-link">
<span class="menu-text"><span class="chapter-number">C</span> <span class="chapter-title">Appendix C: Computational Hardware</span></span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./appendix-d-neo4j-queries.html" class="sidebar-item-text sidebar-link">
<span class="menu-text"><span class="chapter-number">D</span> <span class="chapter-title">Knowledge Graph Queries and Supporting Data</span></span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./appendix-e-db-queries.html" class="sidebar-item-text sidebar-link">
<span class="menu-text"><span class="chapter-number">E</span> <span class="chapter-title">Database Queries</span></span></a>
</div>
</li>
</ul>
</li>
</ul>
</div>
</nav>
<div id="quarto-sidebar-glass" class="quarto-sidebar-collapse-item" data-bs-toggle="collapse" data-bs-target=".quarto-sidebar-collapse-item"></div>
<!-- margin-sidebar -->
<div id="quarto-margin-sidebar" class="sidebar margin-sidebar">
<nav id="TOC" role="doc-toc" class="toc-active">
<h2 id="toc-title">Table of contents</h2>
<ul>
<li><a href="#the-etl-pipeline-from-raw-literature-to-structured-knowledge" id="toc-the-etl-pipeline-from-raw-literature-to-structured-knowledge" class="nav-link active" data-scroll-target="#the-etl-pipeline-from-raw-literature-to-structured-knowledge"><span class="header-section-number">7.1</span> The ETL Pipeline: From Raw Literature to Structured Knowledge</a>
<ul>
<li><a href="#corpus-size-and-variability" id="toc-corpus-size-and-variability" class="nav-link" data-scroll-target="#corpus-size-and-variability"><span class="header-section-number">7.1.1</span> Corpus Size and Variability</a></li>
<li><a href="#stage-1-multi-source-data-acquisition-and-enrichment" id="toc-stage-1-multi-source-data-acquisition-and-enrichment" class="nav-link" data-scroll-target="#stage-1-multi-source-data-acquisition-and-enrichment"><span class="header-section-number">7.1.2</span> Stage 1: Multi-Source Data Acquisition and Enrichment</a>
<ul class="collapse">
<li><a href="#parallelized-metadata-fetching" id="toc-parallelized-metadata-fetching" class="nav-link" data-scroll-target="#parallelized-metadata-fetching"><span class="header-section-number">7.1.2.1</span> Parallelized Metadata Fetching:</a></li>
<li><a href="#data-fusion-and-deduplication" id="toc-data-fusion-and-deduplication" class="nav-link" data-scroll-target="#data-fusion-and-deduplication"><span class="header-section-number">7.1.2.2</span> Data Fusion and Deduplication:</a></li>
<li><a href="#metadata-enrichment" id="toc-metadata-enrichment" class="nav-link" data-scroll-target="#metadata-enrichment"><span class="header-section-number">7.1.2.3</span> Metadata Enrichment:</a></li>
</ul></li>
<li><a href="#stage-2-asynchronous-full-text-pdf-retrieval" id="toc-stage-2-asynchronous-full-text-pdf-retrieval" class="nav-link" data-scroll-target="#stage-2-asynchronous-full-text-pdf-retrieval"><span class="header-section-number">7.1.3</span> Stage 2: Asynchronous Full-Text PDF Retrieval</a></li>
<li><a href="#stage-3-bibliographic-management-and-citation-rendering" id="toc-stage-3-bibliographic-management-and-citation-rendering" class="nav-link" data-scroll-target="#stage-3-bibliographic-management-and-citation-rendering"><span class="header-section-number">7.1.4</span> Stage 3: Bibliographic Management and Citation Rendering</a></li>
<li><a href="#stage-4-content-extraction-via-document-layout-analysis" id="toc-stage-4-content-extraction-via-document-layout-analysis" class="nav-link" data-scroll-target="#stage-4-content-extraction-via-document-layout-analysis"><span class="header-section-number">7.1.5</span> Stage 4: Content Extraction via Document Layout Analysis</a>
<ul class="collapse">
<li><a href="#core-technology-and-identified-limitations" id="toc-core-technology-and-identified-limitations" class="nav-link" data-scroll-target="#core-technology-and-identified-limitations"><span class="header-section-number">7.1.5.1</span> Core Technology and Identified Limitations</a></li>
<li><a href="#custom-enhancements-and-implementation" id="toc-custom-enhancements-and-implementation" class="nav-link" data-scroll-target="#custom-enhancements-and-implementation"><span class="header-section-number">7.1.5.2</span> Custom Enhancements and Implementation</a></li>
<li><a href="#initial-challenge-inaccurate-bounding-boxes-and-model-inefficiency" id="toc-initial-challenge-inaccurate-bounding-boxes-and-model-inefficiency" class="nav-link" data-scroll-target="#initial-challenge-inaccurate-bounding-boxes-and-model-inefficiency"><span class="header-section-number">7.1.5.3</span> Initial Challenge: Inaccurate Bounding Boxes and Model Inefficiency</a></li>
<li><a href="#bounding-box-and-image-preprocessing-modifications" id="toc-bounding-box-and-image-preprocessing-modifications" class="nav-link" data-scroll-target="#bounding-box-and-image-preprocessing-modifications"><span class="header-section-number">7.1.5.4</span> Bounding Box and Image Preprocessing Modifications</a></li>
<li><a href="#implementation-of-selective-content-masking" id="toc-implementation-of-selective-content-masking" class="nav-link" data-scroll-target="#implementation-of-selective-content-masking"><span class="header-section-number">7.1.5.5</span> Implementation of Selective Content Masking</a></li>
<li><a href="#heuristic-based-merging-of-fragmented-layout-clusters" id="toc-heuristic-based-merging-of-fragmented-layout-clusters" class="nav-link" data-scroll-target="#heuristic-based-merging-of-fragmented-layout-clusters"><span class="header-section-number">7.1.5.6</span> Heuristic-Based Merging of Fragmented Layout Clusters</a></li>
<li><a href="#misclassification-of-page-layouts" id="toc-misclassification-of-page-layouts" class="nav-link" data-scroll-target="#misclassification-of-page-layouts"><span class="header-section-number">7.1.5.7</span> Misclassification of Page Layouts</a></li>
<li><a href="#heuristic-based-re-classification-of-misidentified-clusters" id="toc-heuristic-based-re-classification-of-misidentified-clusters" class="nav-link" data-scroll-target="#heuristic-based-re-classification-of-misidentified-clusters"><span class="header-section-number">7.1.5.8</span> Heuristic-Based Re-classification of Misidentified Clusters</a></li>
<li><a href="#pre-emptive-filtering-of-page-level-artifacts" id="toc-pre-emptive-filtering-of-page-level-artifacts" class="nav-link" data-scroll-target="#pre-emptive-filtering-of-page-level-artifacts"><span class="header-section-number">7.1.5.9</span> Pre-emptive Filtering of Page-Level Artifacts</a></li>
<li><a href="#refinements-to-the-underlying-layout-predictor" id="toc-refinements-to-the-underlying-layout-predictor" class="nav-link" data-scroll-target="#refinements-to-the-underlying-layout-predictor"><span class="header-section-number">7.1.5.10</span> Refinements to the Underlying Layout Predictor</a></li>
<li><a href="#final-implementation-in-the-etl-pipeline" id="toc-final-implementation-in-the-etl-pipeline" class="nav-link" data-scroll-target="#final-implementation-in-the-etl-pipeline"><span class="header-section-number">7.1.5.11</span> Final Implementation in the ETL Pipeline</a></li>
</ul></li>
<li><a href="#stage-5-structured-field-extraction-using-large-language-models" id="toc-stage-5-structured-field-extraction-using-large-language-models" class="nav-link" data-scroll-target="#stage-5-structured-field-extraction-using-large-language-models"><span class="header-section-number">7.1.6</span> Stage 5: Structured Field Extraction using Large Language Models</a>
<ul class="collapse">
<li><a href="#methodology-iterative-context-aware-extraction-with-local-llms" id="toc-methodology-iterative-context-aware-extraction-with-local-llms" class="nav-link" data-scroll-target="#methodology-iterative-context-aware-extraction-with-local-llms"><span class="header-section-number">7.1.6.1</span> Methodology: Iterative, Context-Aware Extraction with Local LLMs</a></li>
<li><a href="#parallelized-execution-and-data-merging" id="toc-parallelized-execution-and-data-merging" class="nav-link" data-scroll-target="#parallelized-execution-and-data-merging"><span class="header-section-number">7.1.6.2</span> Parallelized Execution and Data Merging</a></li>
</ul></li>
<li><a href="#stage-6-thematic-analysis-via-topic-modeling" id="toc-stage-6-thematic-analysis-via-topic-modeling" class="nav-link" data-scroll-target="#stage-6-thematic-analysis-via-topic-modeling"><span class="header-section-number">7.1.7</span> Stage 6: Thematic Analysis via Topic Modeling</a>
<ul class="collapse">
<li><a href="#methodology-probabilistic-topic-modeling-with-lda" id="toc-methodology-probabilistic-topic-modeling-with-lda" class="nav-link" data-scroll-target="#methodology-probabilistic-topic-modeling-with-lda"><span class="header-section-number">7.1.7.1</span> Methodology: Probabilistic Topic Modeling with LDA</a></li>
<li><a href="#data-preprocessing-and-hyperparameter-optimization" id="toc-data-preprocessing-and-hyperparameter-optimization" class="nav-link" data-scroll-target="#data-preprocessing-and-hyperparameter-optimization"><span class="header-section-number">7.1.7.2</span> Data Preprocessing and Hyperparameter Optimization</a></li>
<li><a href="#model-evaluation-topic-coherence" id="toc-model-evaluation-topic-coherence" class="nav-link" data-scroll-target="#model-evaluation-topic-coherence"><span class="header-section-number">7.1.7.3</span> Model Evaluation: Topic Coherence</a></li>
<li><a href="#interpretation-and-visualization" id="toc-interpretation-and-visualization" class="nav-link" data-scroll-target="#interpretation-and-visualization"><span class="header-section-number">7.1.7.4</span> Interpretation and Visualization</a></li>
</ul></li>
<li><a href="#stage-7-the-semantic-unification-pipeline" id="toc-stage-7-the-semantic-unification-pipeline" class="nav-link" data-scroll-target="#stage-7-the-semantic-unification-pipeline"><span class="header-section-number">7.1.8</span> Stage 7: The Semantic Unification Pipeline</a>
<ul class="collapse">
<li><a href="#the-challenge-of-terminological-heterogeneity-in-automated-extraction" id="toc-the-challenge-of-terminological-heterogeneity-in-automated-extraction" class="nav-link" data-scroll-target="#the-challenge-of-terminological-heterogeneity-in-automated-extraction"><span class="header-section-number">7.1.8.1</span> The Challenge of Terminological Heterogeneity in Automated Extraction</a></li>
<li><a href="#a-vector-space-model-for-semantic-representation" id="toc-a-vector-space-model-for-semantic-representation" class="nav-link" data-scroll-target="#a-vector-space-model-for-semantic-representation"><span class="header-section-number">7.1.8.2</span> A Vector Space Model for Semantic Representation</a></li>
<li><a href="#pre-computation-of-the-canonical-knowledge-base" id="toc-pre-computation-of-the-canonical-knowledge-base" class="nav-link" data-scroll-target="#pre-computation-of-the-canonical-knowledge-base"><span class="header-section-number">7.1.8.3</span> Pre-computation of the Canonical Knowledge Base</a></li>
<li><a href="#quantifying-semantic-similarity-the-cosine-similarity-metric" id="toc-quantifying-semantic-similarity-the-cosine-similarity-metric" class="nav-link" data-scroll-target="#quantifying-semantic-similarity-the-cosine-similarity-metric"><span class="header-section-number">7.1.8.4</span> Quantifying Semantic Similarity: The Cosine Similarity Metric</a></li>
<li><a href="#threshold-based-mapping-for-ontological-consistency" id="toc-threshold-based-mapping-for-ontological-consistency" class="nav-link" data-scroll-target="#threshold-based-mapping-for-ontological-consistency"><span class="header-section-number">7.1.8.5</span> Threshold-Based Mapping for Ontological Consistency</a></li>
<li><a href="#knowledge-graph-integration" id="toc-knowledge-graph-integration" class="nav-link" data-scroll-target="#knowledge-graph-integration"><span class="header-section-number">7.1.8.6</span> Knowledge Graph Integration</a></li>
<li><a href="#dynamic-graph-construction" id="toc-dynamic-graph-construction" class="nav-link" data-scroll-target="#dynamic-graph-construction"><span class="header-section-number">7.1.8.7</span> Dynamic Graph Construction</a></li>
<li><a href="#querying-for-methodological-insights" id="toc-querying-for-methodological-insights" class="nav-link" data-scroll-target="#querying-for-methodological-insights"><span class="header-section-number">7.1.8.8</span> Querying for Methodological Insights</a></li>
</ul></li>
</ul></li>
<li><a href="#etl-pipeline-summary-and-output" id="toc-etl-pipeline-summary-and-output" class="nav-link" data-scroll-target="#etl-pipeline-summary-and-output"><span class="header-section-number">7.2</span> ETL Pipeline Summary and Output</a></li>
<li><a href="#hysemrag-qa-framework-an-agentic-approach-for-verifiable-generation" id="toc-hysemrag-qa-framework-an-agentic-approach-for-verifiable-generation" class="nav-link" data-scroll-target="#hysemrag-qa-framework-an-agentic-approach-for-verifiable-generation"><span class="header-section-number">8</span> HySemRAG-QA Framework: An Agentic Approach for Verifiable Generation</a></li>
</ul>
</nav>
</div>
<!-- main -->
<main class="content" id="quarto-document-content">
<header id="title-block-header" class="quarto-title-block default">
<div class="quarto-title">
<div class="quarto-title-block"><div><h1 class="title"><span class="chapter-number">7</span> <span class="chapter-title">Technical Report</span></h1><button type="button" class="btn code-tools-button dropdown-toggle" id="quarto-code-tools-menu" data-bs-toggle="dropdown" aria-expanded="false"><i class="bi"></i> Code</button><ul class="dropdown-menu dropdown-menu-end" aria-labelelledby="quarto-code-tools-menu"><li><a id="quarto-show-all-code" class="dropdown-item" href="javascript:void(0)" role="button">Show All Code</a></li><li><a id="quarto-hide-all-code" class="dropdown-item" href="javascript:void(0)" role="button">Hide All Code</a></li><li><hr class="dropdown-divider"></li><li><a id="quarto-view-source" class="dropdown-item" href="javascript:void(0)" role="button">View Source</a></li></ul></div></div>
</div>
<div class="quarto-title-meta">
</div>
</header>
<p>#06-technical-report.qmd</p>
<section id="the-etl-pipeline-from-raw-literature-to-structured-knowledge" class="level2" data-number="7.1">
<h2 data-number="7.1" class="anchored" data-anchor-id="the-etl-pipeline-from-raw-literature-to-structured-knowledge"><span class="header-section-number">7.1</span> The ETL Pipeline: From Raw Literature to Structured Knowledge</h2>
<p>This chapter will server as the technical report for the ETL+HySemRAG methodology and is a work in progress. This chapter will be updated once the draft is complete to describe all phases and theories of operation of the system.</p>
<div id="fig-graph-abstract" class="quarto-float quarto-figure quarto-figure-left anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-graph-abstract-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="figures/ETL_RAG.png" class="img-fluid figure-img">
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-graph-abstract-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure 7.1: Graphical Abstract
</figcaption>
</figure>
</div>
<section id="corpus-size-and-variability" class="level3" data-number="7.1.1">
<h3 data-number="7.1.1" class="anchored" data-anchor-id="corpus-size-and-variability"><span class="header-section-number">7.1.1</span> Corpus Size and Variability</h3>
<p>The final corpus size in each ETL run is not fixed, rather emerges naturally based on the availability of open-access full-text PDFs via the Unpaywall API. Typically, from an initial candidate pool of approximately 3,400 metadata records retrieved from PubMed, OpenAlex, and Scopus, the number of successfully retrieved full-text documents ranges from around 500 to 1,300. This inherent variability reflects realistic constraints encountered during automated scholarly literature retrieval, specifically the reliance on open-access publications. Importantly, this variability underscores the pipeline’s robustness and adaptability, as subsequent processing stages maintain stable performance metrics (accuracy of data extraction, knowledge graph quality, semantic consistency) regardless of the exact corpus size. This adaptability positions the ETL pipeline as particularly suited to real-world research scenarios where comprehensive subscription-based access may not always be available.</p>
<p><strong>Demonstration Video</strong></p>
<p>Here is a video demonstrating the system’s capabilities:</p>
<div class="quarto-video ratio ratio-16x9"><iframe data-external="1" src="https://www.youtube.com/embed/ZCy5ESJ1gVE?si=K8CttwgTj7yGrWjn" title="" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe></div>
<hr>
</section>
<section id="stage-1-multi-source-data-acquisition-and-enrichment" class="level3" data-number="7.1.2">
<h3 data-number="7.1.2" class="anchored" data-anchor-id="stage-1-multi-source-data-acquisition-and-enrichment"><span class="header-section-number">7.1.2</span> Stage 1: Multi-Source Data Acquisition and Enrichment</h3>
<p>The initial and most foundational stage of the ETL pipeline is the acquisition of a comprehensive, relevant, and clean corpus of scholarly literature. The objective of this phase is to systematically gather article metadata from multiple disparate sources, each with unique strengths, and then intelligently fuse them into a single, deduplicated, and enriched dataset. This process is orchestrated by a series of high-performance Python modules designed for scalability and robustness, ensuring that the subsequent stages of the pipeline operate on a high-quality foundation.</p>
<section id="parallelized-metadata-fetching" class="level4" data-number="7.1.2.1">
<h4 data-number="7.1.2.1" class="anchored" data-anchor-id="parallelized-metadata-fetching"><span class="header-section-number">7.1.2.1</span> Parallelized Metadata Fetching:</h4>
<p>To achieve a comprehensive corpus, the system employs a multi-source fetching strategy, simultaneously querying three major scholarly databases: PubMed, for its depth in biomedical and life sciences literature; OpenAlex, for its broad multidisciplinary coverage and rich citation network data; and Elsevier’s Scopus API, for its extensive abstract and citation database, particularly for STEM fields. The GUI allows the user to select any combination of these sources for a given run.</p>
<p>Recognizing that fetching metadata for thousands of articles via sequential API calls would present a significant performance bottleneck, the system was architected for high-throughput, concurrent data retrieval. The implementation is tailored to the specific architecture and rate-limiting policies of each API. For PubMed, the scripts/fast_pubmed.py module first performs a single, efficient esearch call to retrieve up to 10,000 PubMed IDs (PMIDs) matching a given query. Subsequently, it uses a ThreadPoolExecutor to create a pool of worker threads that fetch the full article metadata in parallel batches using efetch, while adhering to NCBI’s polite API usage policy of approximately 3 queries per second. For OpenAlex and Scopus, the system leverages modern asynchronous I/O via the asyncio and aiohttp libraries within the scripts/fast_openalex.py and scripts/etl_elsevier.py modules. This approach allows the system to initiate hundreds of concurrent HTTP requests, managing them efficiently without being blocked by network latency. An asyncio.Semaphore is used as a throttling mechanism to ensure the number of requests per second does not exceed the official rate limits for each API, preventing errors and ensuring reliable access.</p>
</section>
<section id="data-fusion-and-deduplication" class="level4" data-number="7.1.2.2">
<h4 data-number="7.1.2.2" class="anchored" data-anchor-id="data-fusion-and-deduplication"><span class="header-section-number">7.1.2.2</span> Data Fusion and Deduplication:</h4>
<p>Once the parallel fetching processes are complete, the metadata from each source, now held in separate pandas DataFrames, must be combined into a single, unified dataset. This is handled by the merge_and_deduplicate function within the scripts/etl_utils_refined.py module. The function first concatenates the list of DataFrames into one large, heterogeneous dataset.</p>
<div id="merge-deduplicate" class="cell" data-execution_count="2">
<details class="code-fold">
<summary>Code</summary>
<div class="sourceCode cell-code" id="cb1"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> merge_and_deduplicate(from_pubmed_all: List[pd.DataFrame], </span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a> from_openalex_all: List[pd.DataFrame], </span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a> from_elsevier_all: Optional[List[pd.DataFrame]] <span class="op">=</span> <span class="va">None</span>) <span class="op">-></span> pd.DataFrame:</span>
<span id="cb1-4"><a href="#cb1-4" aria-hidden="true" tabindex="-1"></a> all_dfs <span class="op">=</span> []</span>
<span id="cb1-5"><a href="#cb1-5" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> from_pubmed_all: all_dfs.extend(from_pubmed_all)</span>
<span id="cb1-6"><a href="#cb1-6" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> from_openalex_all: all_dfs.extend(from_openalex_all)</span>
<span id="cb1-7"><a href="#cb1-7" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> from_elsevier_all: all_dfs.extend(from_elsevier_all)</span>
<span id="cb1-8"><a href="#cb1-8" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> <span class="kw">not</span> all_dfs:</span>
<span id="cb1-9"><a href="#cb1-9" aria-hidden="true" tabindex="-1"></a> logging.warning(<span class="st">"No DataFrames provided to merge_and_deduplicate."</span>)</span>
<span id="cb1-10"><a href="#cb1-10" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> pd.DataFrame()</span>
<span id="cb1-11"><a href="#cb1-11" aria-hidden="true" tabindex="-1"></a> combined_df <span class="op">=</span> pd.concat(all_dfs, ignore_index<span class="op">=</span><span class="va">True</span>)</span>
<span id="cb1-12"><a href="#cb1-12" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> combined_df.empty:</span>
<span id="cb1-13"><a href="#cb1-13" aria-hidden="true" tabindex="-1"></a> logging.info(<span class="st">"Combined DataFrame is empty before deduplication."</span>)</span>
<span id="cb1-14"><a href="#cb1-14" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> combined_df</span>
<span id="cb1-15"><a href="#cb1-15" aria-hidden="true" tabindex="-1"></a> deduplicated_df <span class="op">=</span> deduplicate_by_doi_title(combined_df)</span>
<span id="cb1-16"><a href="#cb1-16" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> deduplicated_df</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
</details>
</div>
<p>The subsequent critical step is deduplication, as the same article may be retrieved from multiple databases. A simple row-based drop would be insufficient. Therefore, the system implements a robust, hierarchical deduplication algorithm. First, it normalizes all Digital Object Identifiers (DOIs) by converting them to a consistent lowercase format and removing whitespace. It then identifies and retains the first occurrence of each unique DOI.</p>
<div id="deduplicate-doi" class="cell" data-execution_count="3">
<details class="code-fold">
<summary>Code</summary>
<div class="sourceCode cell-code" id="cb2"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> deduplicate_by_doi_title(df: pd.DataFrame) <span class="op">-></span> pd.DataFrame:</span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a> <span class="co"># Create normalized columns</span></span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> <span class="st">"DOI"</span> <span class="kw">not</span> <span class="kw">in</span> df.columns: df[<span class="st">"DOI"</span>] <span class="op">=</span> <span class="st">""</span></span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> <span class="st">"Title"</span> <span class="kw">not</span> <span class="kw">in</span> df.columns: df[<span class="st">"Title"</span>] <span class="op">=</span> <span class="st">""</span></span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a> df[<span class="st">"DOI"</span>] <span class="op">=</span> df[<span class="st">"DOI"</span>].fillna(<span class="st">""</span>)</span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a> df[<span class="st">"Title"</span>] <span class="op">=</span> df[<span class="st">"Title"</span>].fillna(<span class="st">""</span>)</span>
<span id="cb2-7"><a href="#cb2-7" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-8"><a href="#cb2-8" aria-hidden="true" tabindex="-1"></a> df[<span class="st">"DOI_clean"</span>] <span class="op">=</span> df[<span class="st">"DOI"</span>].astype(<span class="bu">str</span>).<span class="bu">str</span>.lower().<span class="bu">str</span>.strip()</span>
<span id="cb2-9"><a href="#cb2-9" aria-hidden="true" tabindex="-1"></a> df[<span class="st">"Title_clean"</span>] <span class="op">=</span> df[<span class="st">"Title"</span>].astype(<span class="bu">str</span>).<span class="bu">str</span>.lower().<span class="bu">str</span>.strip()</span>
<span id="cb2-10"><a href="#cb2-10" aria-hidden="true" tabindex="-1"></a> df.sort_values(by<span class="op">=</span>[<span class="st">"DOI_clean"</span>, <span class="st">"Title_clean"</span>], inplace<span class="op">=</span><span class="va">True</span>, kind<span class="op">=</span><span class="st">'stable'</span>)</span>
<span id="cb2-11"><a href="#cb2-11" aria-hidden="true" tabindex="-1"></a> indices_to_keep <span class="op">=</span> <span class="bu">set</span>()</span>
<span id="cb2-12"><a href="#cb2-12" aria-hidden="true" tabindex="-1"></a> seen_dois <span class="op">=</span> <span class="bu">set</span>()</span>
<span id="cb2-13"><a href="#cb2-13" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span> index, doi <span class="kw">in</span> df.loc[df[<span class="st">"DOI_clean"</span>] <span class="op">!=</span> <span class="st">""</span>, <span class="st">"DOI_clean"</span>].items():</span>
<span id="cb2-14"><a href="#cb2-14" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> doi <span class="kw">not</span> <span class="kw">in</span> seen_dois:</span>
<span id="cb2-15"><a href="#cb2-15" aria-hidden="true" tabindex="-1"></a> indices_to_keep.add(index)</span>
<span id="cb2-16"><a href="#cb2-16" aria-hidden="true" tabindex="-1"></a> seen_dois.add(doi)</span>
<span id="cb2-17"><a href="#cb2-17" aria-hidden="true" tabindex="-1"></a> empty_doi_mask <span class="op">=</span> (df[<span class="st">"DOI_clean"</span>] <span class="op">==</span> <span class="st">""</span>) <span class="op">&</span> (<span class="op">~</span>df.index.isin(indices_to_keep))</span>
<span id="cb2-18"><a href="#cb2-18" aria-hidden="true" tabindex="-1"></a> seen_titles_empty_doi <span class="op">=</span> <span class="bu">set</span>()</span>
<span id="cb2-19"><a href="#cb2-19" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span> index, title <span class="kw">in</span> df.loc[empty_doi_mask, <span class="st">"Title_clean"</span>].items():</span>
<span id="cb2-20"><a href="#cb2-20" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> title <span class="kw">not</span> <span class="kw">in</span> seen_titles_empty_doi:</span>
<span id="cb2-21"><a href="#cb2-21" aria-hidden="true" tabindex="-1"></a> indices_to_keep.add(index)</span>
<span id="cb2-22"><a href="#cb2-22" aria-hidden="true" tabindex="-1"></a> seen_titles_empty_doi.add(title)</span>
<span id="cb2-23"><a href="#cb2-23" aria-hidden="true" tabindex="-1"></a> df_deduplicated <span class="op">=</span> df.loc[<span class="bu">list</span>(indices_to_keep)].copy()</span>
<span id="cb2-24"><a href="#cb2-24" aria-hidden="true" tabindex="-1"></a> df_deduplicated.drop(columns<span class="op">=</span>[<span class="st">"DOI_clean"</span>, <span class="st">"Title_clean"</span>], inplace<span class="op">=</span><span class="va">True</span>)</span>
<span id="cb2-25"><a href="#cb2-25" aria-hidden="true" tabindex="-1"></a> df_deduplicated.reset_index(drop<span class="op">=</span><span class="va">True</span>, inplace<span class="op">=</span><span class="va">True</span>)</span>
<span id="cb2-26"><a href="#cb2-26" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> df_deduplicated</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
</details>
</div>
<p>For the remaining records that may lack a DOI, the algorithm performs a secondary deduplication based on a similarly normalized article title. This two-pass strategy ensures that each unique scholarly work is represented by a single, canonical record in the final dataset, preventing data redundancy in all downstream analyses.</p>
</section>
<section id="metadata-enrichment" class="level4" data-number="7.1.2.3">
<h4 data-number="7.1.2.3" class="anchored" data-anchor-id="metadata-enrichment"><span class="header-section-number">7.1.2.3</span> Metadata Enrichment:</h4>
<p>The initial, merged dataset, while comprehensive, may contain records with incomplete information (e.g., a record from PubMed might be missing a PDF link that OpenAlex has). The final step in the acquisition phase is therefore a dedicated enrichment process, orchestrated by scripts/metadata_enrichment.py. This module takes the entire list of unique DOIs from the deduplicated dataset and once again uses an asynchronous approach to concurrently query both the OpenAlex and Crossref APIs for every article.</p>
<div id="enrich-metadata" class="cell" data-execution_count="4">
<details class="code-fold">
<summary>Code</summary>
<div class="sourceCode cell-code" id="cb3"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> enrich_metadata_sync(df: pd.DataFrame, crossref_email: Optional[<span class="bu">str</span>] <span class="op">=</span> <span class="va">None</span>) <span class="op">-></span> pd.DataFrame:</span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a> email <span class="op">=</span> crossref_email <span class="kw">or</span> MAILTO</span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a> dois <span class="op">=</span> df[<span class="st">"DOI"</span>].dropna().unique().tolist()</span>
<span id="cb3-4"><a href="#cb3-4" aria-hidden="true" tabindex="-1"></a> enrichment <span class="op">=</span> asyncio.run(enrich_metadata(dois, email))</span>
<span id="cb3-5"><a href="#cb3-5" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb3-6"><a href="#cb3-6" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span> idx, row <span class="kw">in</span> df.iterrows():</span>
<span id="cb3-7"><a href="#cb3-7" aria-hidden="true" tabindex="-1"></a> doi <span class="op">=</span> row[<span class="st">"DOI"</span>]</span>
<span id="cb3-8"><a href="#cb3-8" aria-hidden="true" tabindex="-1"></a> metadata <span class="op">=</span> enrichment.get(doi, {})</span>
<span id="cb3-9"><a href="#cb3-9" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> metadata:</span>
<span id="cb3-10"><a href="#cb3-10" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> pd.isna(row.get(<span class="st">"Abstract"</span>)) <span class="kw">or</span> <span class="kw">not</span> <span class="bu">str</span>(row[<span class="st">"Abstract"</span>]).strip():</span>
<span id="cb3-11"><a href="#cb3-11" aria-hidden="true" tabindex="-1"></a> df.at[idx, <span class="st">"Abstract"</span>] <span class="op">=</span> metadata.get(<span class="st">"Abstract"</span>, row[<span class="st">"Abstract"</span>])</span>
<span id="cb3-12"><a href="#cb3-12" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> pd.isna(row.get(<span class="st">"ItemType"</span>)) <span class="kw">or</span> <span class="kw">not</span> <span class="bu">str</span>(row[<span class="st">"ItemType"</span>]).strip() <span class="kw">or</span> row[<span class="st">"ItemType"</span>] <span class="op">==</span> <span class="st">"N/A"</span>:</span>
<span id="cb3-13"><a href="#cb3-13" aria-hidden="true" tabindex="-1"></a> df.at[idx, <span class="st">"ItemType"</span>] <span class="op">=</span> metadata.get(<span class="st">"ItemType"</span>, row.get(<span class="st">"ItemType"</span>, <span class="st">"N/A"</span>))</span>
<span id="cb3-14"><a href="#cb3-14" aria-hidden="true" tabindex="-1"></a> citation_count <span class="op">=</span> metadata.get(<span class="st">"CitationCount"</span>, row.get(<span class="st">"CitationCount"</span>))</span>
<span id="cb3-15"><a href="#cb3-15" aria-hidden="true" tabindex="-1"></a> df.at[idx, <span class="st">"CitationCount"</span>] <span class="op">=</span> <span class="bu">int</span>(citation_count) <span class="cf">if</span> citation_count <span class="kw">is</span> <span class="kw">not</span> <span class="va">None</span> <span class="cf">else</span> <span class="dv">0</span></span>
<span id="cb3-16"><a href="#cb3-16" aria-hidden="true" tabindex="-1"></a> primary_topic <span class="op">=</span> metadata.get(<span class="st">"primary_topic"</span>)</span>
<span id="cb3-17"><a href="#cb3-17" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> primary_topic:</span>
<span id="cb3-18"><a href="#cb3-18" aria-hidden="true" tabindex="-1"></a> df.at[idx, <span class="st">"primary_topic"</span>] <span class="op">=</span> json.dumps(primary_topic)</span>
<span id="cb3-19"><a href="#cb3-19" aria-hidden="true" tabindex="-1"></a> <span class="cf">else</span>:</span>
<span id="cb3-20"><a href="#cb3-20" aria-hidden="true" tabindex="-1"></a> df.at[idx, <span class="st">"primary_topic"</span>] <span class="op">=</span> pd.NA</span>
<span id="cb3-21"><a href="#cb3-21" aria-hidden="true" tabindex="-1"></a> df.at[idx, <span class="st">"is_published"</span>] <span class="op">=</span> metadata.get(<span class="st">"is_published"</span>, row.get(<span class="st">"is_published"</span>))</span>
<span id="cb3-22"><a href="#cb3-22" aria-hidden="true" tabindex="-1"></a> df.at[idx, <span class="st">"is_retracted"</span>] <span class="op">=</span> metadata.get(<span class="st">"is_retracted"</span>, row.get(<span class="st">"is_retracted"</span>))</span>
<span id="cb3-23"><a href="#cb3-23" aria-hidden="true" tabindex="-1"></a> df.at[idx, <span class="st">"OpenAlexID"</span>] <span class="op">=</span> metadata.get(<span class="st">"OpenAlexID"</span>, row.get(<span class="st">"OpenAlexID"</span>))</span>
<span id="cb3-24"><a href="#cb3-24" aria-hidden="true" tabindex="-1"></a> df[<span class="st">"CitationCount"</span>] <span class="op">=</span> df[<span class="st">"CitationCount"</span>].fillna(<span class="dv">0</span>).astype(<span class="bu">int</span>)</span>
<span id="cb3-25"><a href="#cb3-25" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb3-26"><a href="#cb3-26" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> df</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
</details>
</div>
<p>The results are then intelligently merged back into the main DataFrame. This process is non-destructive; it is designed to backfill missing information rather than overwrite existing data. For instance, an abstract retrieved from Crossref will only be added to a record if that record’s Abstract field is currently empty. Similarly, fields like ItemType or CitationCount are updated only if the existing data is null or a placeholder. This ensures that the final output is the most complete possible version of each record, combining the strengths of all queried databases.</p>
<hr>
</section>
</section>
<section id="stage-2-asynchronous-full-text-pdf-retrieval" class="level3" data-number="7.1.3">
<h3 data-number="7.1.3" class="anchored" data-anchor-id="stage-2-asynchronous-full-text-pdf-retrieval"><span class="header-section-number">7.1.3</span> Stage 2: Asynchronous Full-Text PDF Retrieval</h3>
<p>Following the creation of a unified and deduplicated list of scholarly articles, the next critical stage of the pipeline is the acquisition of the full-text content for each entry. This is essential for the deep content analysis, entity extraction, and topic modeling performed in subsequent stages. The system orchestrates this process by leveraging the Digital Object Identifier (DOI) of each article to query the <em>Unpaywall API</em>, a service that indexes legally available open-access versions of scholarly publications. Given that this stage involves potentially thousands of network requests, a high-throughput, asynchronous architecture was implemented to manage this I/O-bound task efficiently.</p>
<p>The core retrieval logic, encapsulated in <em>scripts/async_unpaywall.py</em>, is built upon Python’s <em>asyncio</em> and <em>aiohttp</em> libraries. This approach allows the system to initiate and manage thousands of concurrent HTTP requests, preventing the pipeline from being blocked by network latency from any single request. To ensure compliance with Unpaywall’s API usage policies and to maintain system stability, an <em>asyncio.Semaphore</em> is employed as a robust rate-limiting mechanism. The semaphore is initialized with the <em>API_RATE_LIMIT</em> value specified in the project’s configuration file (e.g., 8 queries per second). Each asynchronous worker task must acquire the semaphore before making an API call, guaranteeing that the overall request rate does not exceed the defined limit. The worker function first queries Unpaywall for a given DOI to get a list of potential PDF locations, prioritizing the <em>best_oa_location</em>, and then iteratively attempts to download the file from each candidate URL until a valid PDF is secured.</p>
<div id="unpaywall-worker" class="cell" data-execution_count="5">
<details class="code-fold">
<summary>Code</summary>
<div class="sourceCode cell-code" id="cb4"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="co"># In scripts/async_unpaywall.py</span></span>
<span id="cb4-2"><a href="#cb4-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb4-3"><a href="#cb4-3" aria-hidden="true" tabindex="-1"></a><span class="cf">async</span> <span class="kw">def</span> _worker(row_idx, doi, title, session):</span>
<span id="cb4-4"><a href="#cb4-4" aria-hidden="true" tabindex="-1"></a> <span class="co">"""Return (row_idx, pdf_path|None, status_string)."""</span></span>
<span id="cb4-5"><a href="#cb4-5" aria-hidden="true" tabindex="-1"></a> api_url <span class="op">=</span> <span class="ss">f"https://api.unpaywall.org/v2/</span><span class="sc">{</span>doi<span class="sc">}</span><span class="ss">?email=</span><span class="sc">{</span>UNPAYWALL_EMAIL<span class="sc">}</span><span class="ss">"</span></span>
<span id="cb4-6"><a href="#cb4-6" aria-hidden="true" tabindex="-1"></a> <span class="co"># _fetch_json is rate-limited by a semaphore</span></span>
<span id="cb4-7"><a href="#cb4-7" aria-hidden="true" tabindex="-1"></a> data <span class="op">=</span> <span class="cf">await</span> _fetch_json(session, api_url)</span>
<span id="cb4-8"><a href="#cb4-8" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> <span class="kw">not</span> data:</span>
<span id="cb4-9"><a href="#cb4-9" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> row_idx, <span class="va">None</span>, <span class="st">"Unpaywall_fail"</span></span>
<span id="cb4-10"><a href="#cb4-10" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb4-11"><a href="#cb4-11" aria-hidden="true" tabindex="-1"></a> <span class="co"># Build a list of candidate URLs, prioritizing the best_oa_location</span></span>
<span id="cb4-12"><a href="#cb4-12" aria-hidden="true" tabindex="-1"></a> candidates <span class="op">=</span> []</span>
<span id="cb4-13"><a href="#cb4-13" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> (b <span class="op">:=</span> data.get(<span class="st">"best_oa_location"</span>)): candidates.append(b)</span>
<span id="cb4-14"><a href="#cb4-14" aria-hidden="true" tabindex="-1"></a> candidates <span class="op">+=</span> data.get(<span class="st">"oa_locations"</span>, [])</span>
<span id="cb4-15"><a href="#cb4-15" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb4-16"><a href="#cb4-16" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span> loc <span class="kw">in</span> candidates:</span>
<span id="cb4-17"><a href="#cb4-17" aria-hidden="true" tabindex="-1"></a> pdf_url <span class="op">=</span> loc.get(<span class="st">"url_for_pdf"</span>)</span>
<span id="cb4-18"><a href="#cb4-18" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> <span class="kw">not</span> pdf_url:</span>
<span id="cb4-19"><a href="#cb4-19" aria-hidden="true" tabindex="-1"></a> <span class="cf">continue</span></span>
<span id="cb4-20"><a href="#cb4-20" aria-hidden="true" tabindex="-1"></a> <span class="co"># Attempt to download the PDF from the candidate URL</span></span>
<span id="cb4-21"><a href="#cb4-21" aria-hidden="true" tabindex="-1"></a> pdf_bytes <span class="op">=</span> <span class="cf">await</span> _download_pdf(session, pdf_url)</span>
<span id="cb4-22"><a href="#cb4-22" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> pdf_bytes:</span>
<span id="cb4-23"><a href="#cb4-23" aria-hidden="true" tabindex="-1"></a> <span class="co"># Sanitize filename and save the file</span></span>
<span id="cb4-24"><a href="#cb4-24" aria-hidden="true" tabindex="-1"></a> safe_name <span class="op">=</span> re.sub(<span class="vs">r'[\\/*?:"<>|()]+'</span>, <span class="st">''</span>, title)[:<span class="dv">100</span>] <span class="kw">or</span> doi</span>
<span id="cb4-25"><a href="#cb4-25" aria-hidden="true" tabindex="-1"></a> fname <span class="op">=</span> <span class="ss">f"</span><span class="sc">{</span>safe_name<span class="sc">}</span><span class="ss">_</span><span class="sc">{</span>row_idx<span class="sc">}</span><span class="ss">.pdf"</span></span>
<span id="cb4-26"><a href="#cb4-26" aria-hidden="true" tabindex="-1"></a> path <span class="op">=</span> os.path.join(PDF_SAVE_FOLDER, fname)</span>
<span id="cb4-27"><a href="#cb4-27" aria-hidden="true" tabindex="-1"></a> <span class="cf">with</span> <span class="bu">open</span>(path, <span class="st">"wb"</span>) <span class="im">as</span> f:</span>
<span id="cb4-28"><a href="#cb4-28" aria-hidden="true" tabindex="-1"></a> f.write(pdf_bytes)</span>
<span id="cb4-29"><a href="#cb4-29" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> row_idx, path, <span class="st">"Saved"</span></span>
<span id="cb4-30"><a href="#cb4-30" aria-hidden="true" tabindex="-1"></a> </span>
<span id="cb4-31"><a href="#cb4-31" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> row_idx, <span class="va">None</span>, <span class="st">"No_PDF"</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
</details>
</div>
<p>Upon successful download, the PDF is saved to a local directory defined in the configuration, and the main DataFrame is updated with the local file path for that article record. Records for which a full-text PDF could not be successfully retrieved are filtered out at the conclusion of this stage. The resulting DataFrame, now containing only articles with corresponding local PDF files, is then passed to the next stage of the pipeline for content extraction.</p>
<hr>
</section>
<section id="stage-3-bibliographic-management-and-citation-rendering" class="level3" data-number="7.1.4">
<h3 data-number="7.1.4" class="anchored" data-anchor-id="stage-3-bibliographic-management-and-citation-rendering"><span class="header-section-number">7.1.4</span> Stage 3: Bibliographic Management and Citation Rendering</h3>
<p>Following the acquisition and deduplication of article metadata and the retrieval of corresponding full-text PDFs, the pipeline proceeds to a crucial data management and enrichment stage. The objective here is twofold: first, to create a permanent, centralized, and queryable library of all processed literature using the Zotero reference management system; and second, to pre-generate authoritative citation metadata for each article. This pre-rendered citation data serves as a verifiable “ground truth” for the downstream Retrieval-Augmented Generation (RAG) agent, ensuring the academic rigor and traceability of the final synthesized output. This entire process is orchestrated by the s<em>cripts/fast_zotero_gui.py</em> module.</p>
<p>The integration with Zotero is architected for high-throughput processing. The <em>integrate</em> function within the module receives the DataFrame of articles with valid PDF paths and uses the <em>pyzotero</em> library to communicate with the Zotero API. To maximize efficiency, new items are created in batches of up to 50 articles per API call. Subsequently, the system utilizes a <em>ThreadPoolExecutor</em> to upload the associated PDF files to the newly created Zotero entries in parallel, significantly reducing the time required for this I/O-bound task.</p>
<p>A key technical innovation in this module is the method of citation generation. Rather than relying on slow, repeated API calls to Zotero for formatting each citation, the system leverages the <em>citeproc-py</em> library to perform this rendering locally. This approach uses standard Citation Style Language (CSL) files (e.g., apa.csl, ieee.csl), which are selectable in the GUI, to instantly format both the full bibliographic citation and the corresponding in-text citation for every article. The module also includes a robust fallback mechanism that can query the Zotero API for the citation if the local <em>citeproc</em> rendering encounters an error for a specific entry.</p>
<div id="zotero-citation-rendering" class="cell" data-execution_count="6">
<details class="code-fold">
<summary>Code</summary>
<div class="sourceCode cell-code" id="cb5"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="co"># In scripts/fast_zotero_gui.py</span></span>
<span id="cb5-2"><a href="#cb5-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb5-3"><a href="#cb5-3" aria-hidden="true" tabindex="-1"></a><span class="co"># ... inside the integrate function loop ...</span></span>
<span id="cb5-4"><a href="#cb5-4" aria-hidden="true" tabindex="-1"></a><span class="cf">for</span> idx, zkey <span class="kw">in</span> attach.items():</span>
<span id="cb5-5"><a href="#cb5-5" aria-hidden="true" tabindex="-1"></a> df.at[idx, <span class="st">"ZoteroKey"</span>] <span class="op">=</span> zkey</span>
<span id="cb5-6"><a href="#cb5-6" aria-hidden="true" tabindex="-1"></a> meta <span class="op">=</span> {c: df.at[idx, c] <span class="cf">for</span> c <span class="kw">in</span> (</span>
<span id="cb5-7"><a href="#cb5-7" aria-hidden="true" tabindex="-1"></a> <span class="st">"Title"</span>, <span class="st">"DOI"</span>, <span class="st">"Authors"</span>, <span class="st">"Date"</span>,</span>
<span id="cb5-8"><a href="#cb5-8" aria-hidden="true" tabindex="-1"></a> <span class="st">"Journal"</span>, <span class="st">"Volume"</span>, <span class="st">"Issue"</span>, <span class="st">"Pages"</span></span>
<span id="cb5-9"><a href="#cb5-9" aria-hidden="true" tabindex="-1"></a> )}</span>
<span id="cb5-10"><a href="#cb5-10" aria-hidden="true" tabindex="-1"></a> <span class="cf">try</span>:</span>
<span id="cb5-11"><a href="#cb5-11" aria-hidden="true" tabindex="-1"></a> <span class="co"># Attempt local citation rendering first</span></span>
<span id="cb5-12"><a href="#cb5-12" aria-hidden="true" tabindex="-1"></a> in_text, full <span class="op">=</span> _render_citations(meta, csl_path)</span>
<span id="cb5-13"><a href="#cb5-13" aria-hidden="true" tabindex="-1"></a> <span class="cf">except</span> <span class="pp">Exception</span> <span class="im">as</span> exc:</span>
<span id="cb5-14"><a href="#cb5-14" aria-hidden="true" tabindex="-1"></a> <span class="co"># If local rendering fails, use the fallback</span></span>
<span id="cb5-15"><a href="#cb5-15" aria-hidden="true" tabindex="-1"></a> logging.error(<span class="ss">f"citeproc rendering failed for row </span><span class="sc">{</span>idx<span class="sc">}</span><span class="ss">: </span><span class="sc">{</span>exc<span class="sc">}</span><span class="ss">"</span>)</span>
<span id="cb5-16"><a href="#cb5-16" aria-hidden="true" tabindex="-1"></a> in_text, full <span class="op">=</span> fallback(zkey, csl_path) <span class="cf">if</span> <span class="bu">callable</span>(fallback) <span class="cf">else</span> (<span class="st">""</span>, <span class="st">""</span>)</span>
<span id="cb5-17"><a href="#cb5-17" aria-hidden="true" tabindex="-1"></a> df.at[idx, <span class="st">"InTextCitation"</span>] <span class="op">=</span> in_text <span class="kw">or</span> <span class="st">""</span></span>
<span id="cb5-18"><a href="#cb5-18" aria-hidden="true" tabindex="-1"></a> df.at[idx, <span class="st">"FullCitation"</span>] <span class="op">=</span> full <span class="kw">or</span> <span class="st">""</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
</details>
</div>
<p>The impact of this stage on the overall system is critical. The generated <em>ZoteroKey</em>, <em>InTextCitation</em>, and <em>FullCitation</em> generated in this stage are not merely for user reference; they are programmatically injected back into the DataFrame. This enriched data is subsequently passed as part of the context to the downstream Retrieval-Augmented Generation (RAG) agent.</p>
<p>This architectural choice is fundamental to ensuring the factuality and academic rigor of the final synthesized output. The RAG agent is explicitly instructed to populate its findings into a structured JSON object that requires these specific metadata fields for each piece of evidence it presents. By providing this pre-generated, validated citation data directly within the context, the system constrains the Large Language Model. This prevents the LLM from attempting to generate its own citations, a task for which they are notoriously unreliable and prone to hallucination.</p>
<p>Instead of showing the entire internal prompt, the following illustrates the structure of the metadata object that the RAG agent must populate for each observation it makes. The presence of the Zotero-derived fields highlights their essential role in the validation process.</p>
<div id="metadata-injection" class="cell" data-execution_count="7">
<details class="code-fold">
<summary>Code</summary>
<div class="sourceCode cell-code" id="cb6"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true" tabindex="-1"></a><span class="op">//</span> Example Metadata Schema <span class="cf">for</span> a single RAG Observation</span>
<span id="cb6-2"><a href="#cb6-2" aria-hidden="true" tabindex="-1"></a>{</span>
<span id="cb6-3"><a href="#cb6-3" aria-hidden="true" tabindex="-1"></a> <span class="st">"metadata"</span>: {</span>
<span id="cb6-4"><a href="#cb6-4" aria-hidden="true" tabindex="-1"></a> <span class="st">"PDF_DocIndex"</span>: <span class="st">"148"</span>,</span>
<span id="cb6-5"><a href="#cb6-5" aria-hidden="true" tabindex="-1"></a> <span class="st">"PDF_ChunkIndex"</span>: <span class="st">"0"</span>,</span>
<span id="cb6-6"><a href="#cb6-6" aria-hidden="true" tabindex="-1"></a> <span class="st">"Struct_DocIndex"</span>: <span class="st">"N/A"</span>,</span>
<span id="cb6-7"><a href="#cb6-7" aria-hidden="true" tabindex="-1"></a> <span class="st">"Struct_ChunkIndex"</span>: <span class="st">"N/A"</span>,</span>
<span id="cb6-8"><a href="#cb6-8" aria-hidden="true" tabindex="-1"></a> <span class="st">"DOI"</span>: <span class="st">"10.1016/j.jobe.2021.103722"</span>,</span>
<span id="cb6-9"><a href="#cb6-9" aria-hidden="true" tabindex="-1"></a> <span class="st">"ZoteroKey"</span>: <span class="st">"ABC123DE"</span>,</span>
<span id="cb6-10"><a href="#cb6-10" aria-hidden="true" tabindex="-1"></a> <span class="st">"InTextCitation"</span>: <span class="st">"(Author et al., 2021)"</span>,</span>
<span id="cb6-11"><a href="#cb6-11" aria-hidden="true" tabindex="-1"></a> <span class="st">"FullCitation"</span>: <span class="st">"Author, A., Author, B., & Author, C. (2021). Title of the article. Journal Name, 1(2), 100-110. https://doi.org/10.1016/j.jobe.2021.103722"</span></span>
<span id="cb6-12"><a href="#cb6-12" aria-hidden="true" tabindex="-1"></a> }</span>
<span id="cb6-13"><a href="#cb6-13" aria-hidden="true" tabindex="-1"></a>}</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
</details>
</div>
<p>This architectural choice constrains the Large Language Model, preventing it from generating its own, potentially hallucinated, citations. It ensures that every piece of evidence presented in the final answer is verifiably and accurately linked back to a specific source document managed within the Zotero library, creating a fully traceable and academically sound output.</p>
<hr>
</section>
<section id="stage-4-content-extraction-via-document-layout-analysis" class="level3" data-number="7.1.5">
<h3 data-number="7.1.5" class="anchored" data-anchor-id="stage-4-content-extraction-via-document-layout-analysis"><span class="header-section-number">7.1.5</span> Stage 4: Content Extraction via Document Layout Analysis</h3>
<p><strong>Objective: High-Fidelity Extraction from Scholarly PDFs</strong></p>
<p>A foundational requirement for the ETL pipeline is the ability to accurately and efficiently extract content from thousands of scholarly articles in PDF format. This task is notoriously challenging due to the format’s complex, variable layouts and the need to preserve structural context. The objective for this stage was to move beyond simple text extraction and reliably parse not only the full textual body of each article but also complex embedded elements crucial for scientific analysis, such as tabular data and mathematical formulas formatted in LaTeX.</p>
<section id="core-technology-and-identified-limitations" class="level4" data-number="7.1.5.1">
<h4 data-number="7.1.5.1" class="anchored" data-anchor-id="core-technology-and-identified-limitations"><span class="header-section-number">7.1.5.1</span> Core Technology and Identified Limitations</h4>
<p>To meet this objective, the system leverages IBM’s Docling library as its core document analysis engine, a powerful framework that uses machine learning for sophisticated page layout detection. However, initial implementation and testing revealed several significant limitations with the out-of-the-box library that necessitated a substantial re-engineering effort:</p>
<ol type="1">
<li><p><strong>Performance Bottleneck:</strong> The framework lacked native support for efficient parallelization, making the processing of thousands of documents a prohibitively time-consuming, single-threaded operation.</p></li>
<li><p><strong>Model Inefficiency:</strong> TThe original formula recognition model, based on the <em>SamOPTForCausalLM</em> architecture, was found to be excessively resource-intensive, requiring approximately 30 GB of VRAM per instance, constraining the ability to parallelize effectively.</p></li>
<li><p><strong>Layout Inaccuracies:</strong> The layout detection model frequently produced suboptimal bounding boxes. This manifested in two primary ways:</p>
<ul>
<li><strong>Formula Fragmentation:</strong> Single, multi-line mathematical formulas were often incorrectly segmented into several independent clusters.</li>
<li><strong>Layout Misclassification:</strong> Pages containing structured elements like line numbers were often misclassified in their entirety as a single, large TABLE cluster, losing all other semantic information.</li>
</ul></li>
<li><p><strong>Stability Issues:</strong> Persistent glyph-parsing errors within the core library caused instability during large-scale processing runs.</p></li>
</ol>
<p>To address these significant challenges, a substantial re-engineering of the Docling library was undertaken. The resulting modified codebase, along with testing scripts, sample data, and a validation notebook, has been made publicly available in a dedicated GitHub repository to ensure transparency and allow for independent validation of the enhancements.</p>
<p><a href="https://github.com/agodinezmm2007/docling_mod">Github Repository</a></p>
</section>
<section id="custom-enhancements-and-implementation" class="level4" data-number="7.1.5.2">
<h4 data-number="7.1.5.2" class="anchored" data-anchor-id="custom-enhancements-and-implementation"><span class="header-section-number">7.1.5.2</span> Custom Enhancements and Implementation</h4>
<p>To overcome these challenges, the system was transformed into a high-performance, accurate, and efficient processing system through a series of targeted enhancements, primarily orchestrated by <em>scripts/docling_multi_mp_gui.py</em>.</p>
<p>The performance bottleneck was addressed first by designing a custom architecture around Python’s <em>concurrent.futures.ProcessPoolExecutor</em>. A key innovation was the implementation of a custom <em>worker_initializer</em> function that facilitates true parallel processing across multiple GPUs. This initializer uses a locking mechanism to assign each new worker process to a specific GPU from the available hardware pool, effectively converting Docling into a distributed application and dramatically reducing processing time.</p>
<p>The accuracy and inefficiency issues of the formula recognition model were solved by replacing it entirely. The core <em>code_formula_predictor.py</em> was reconfigured to integrate a more efficient and accurate model, <em>“SmolDocling,”</em> which reduced VRAM requirements to ~8GB.</p>
<p>Finally, the layout inaccuracies and stability issues were addressed through direct modifications to the Docling codebase. This included engineering a sophisticated post-processing step in <em>docling/utils/layout_postprocessor.py</em> with rule-based heuristics to merge fragmented formulas and re-classify entire misidentified pages. The persistent glyph errors required low-level debugging by reverse-engineering the compiled C-extension library (<em>pdf_parsers.cpython-312-x86_64-linux-gnu.so</em>) using the <em>Ghidra</em> software reverse engineering suite to diagnose and patch the root cause.</p>
<p>These extensive modifications are reflected in the numerous library files that were changed during development:</p>
<ul>
<li><p><strong>Primary Docling Library Modifications:</strong></p>
<ul>
<li>docling/backend/docling_parse_v4_backend.py</li>
<li>docling/datamodel/base_models.py</li>
<li>docling/datamodel/pipeline_options.py</li>
<li>docling/models/code_formula_model.py</li>
<li>docling/models/layout_model.py</li>
<li>docling/pipeline/vlm_pipeline.py</li>
<li>docling/utils/layout_postprocessor.py</li>
</ul></li>
<li><p><strong>IBM Model Predictor Modifications:</strong></p>
<ul>
<li>docling_ibm_models/code_formula_model/code_formula_predictor.py</li>
<li>docling_ibm_models/layoutmodel/layout_predictor.py</li>
</ul></li>
</ul>
</section>
<section id="initial-challenge-inaccurate-bounding-boxes-and-model-inefficiency" class="level4" data-number="7.1.5.3">
<h4 data-number="7.1.5.3" class="anchored" data-anchor-id="initial-challenge-inaccurate-bounding-boxes-and-model-inefficiency"><span class="header-section-number">7.1.5.3</span> Initial Challenge: Inaccurate Bounding Boxes and Model Inefficiency</h4>
<p>The first major issue identified was the precision of the layout analysis model. The bounding boxes generated for mathematical formulas were often too tight or included extraneous visual artifacts from surrounding text. When these imperfect image snippets were passed to the original formula recognition model <strong>(ds4sd/CodeFormula)</strong>, it frequently failed, resulting in generative feedback loops that produced repetitive, malformed LaTeX output (as exemplified in <a href="#fig-bad-form" class="quarto-xref">Figure <span>7.2</span></a>, showing a poorly cropped snippet).</p>
<div id="fig-bad-form" class="quarto-float quarto-figure quarto-figure-left anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-bad-form-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="figures/bad_form_17.png" class="img-fluid figure-img">
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-bad-form-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure 7.2: Bad Formula
</figcaption>
</figure>
</div>
<p>Furthermore, the original model, based on the <strong>SamOPTForCausalLM</strong> architecture, was found to be excessively resource-intensive, requiring approximately 30 GB of VRAM per instance. This made large-scale parallelization across multiple GPUs prohibitively expensive and inefficient.</p>
</section>
<section id="bounding-box-and-image-preprocessing-modifications" class="level4" data-number="7.1.5.4">
<h4 data-number="7.1.5.4" class="anchored" data-anchor-id="bounding-box-and-image-preprocessing-modifications"><span class="header-section-number">7.1.5.4</span> Bounding Box and Image Preprocessing Modifications</h4>
<p>The initial attempt to resolve the formula recognition issue focused on improving the quality of the image snippets before they were sent to the model. This involved significant modifications to <em>docling/models/code_formula_model.py</em>.</p>
<ul>
<li><strong>Dynamic Bounding Box Expansion:</strong> Logic was introduced in the <em>prepare_element</em> method to dynamically expand the bounding boxes of detected formulas. Instead of using the original, often tightly-cropped box, the new implementation expands the box by a configurable ratio of the snippet’s height and width. This ensures more visual context is captured as demonstrated in <a href="#fig-good-form" class="quarto-xref">Figure <span>7.3</span></a>.</li>
</ul>
<div id="fig-good-form" class="quarto-float quarto-figure quarto-figure-left anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-good-form-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="figures/good_form_17.png" class="img-fluid figure-img">
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-good-form-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure 7.3: Good Formula
</figcaption>
</figure>
</div>
<ul>
<li><p><strong>Ratio-Based Image Padding:</strong> The original *_pad_with_most_frequent_edge_color* function, which used static pixel padding, was entirely replaced. The new version accepts floating-point ratios, allowing it to apply padding that is proportional to the image’s dimensions. This creates a more consistent and contextually appropriate input for the model, regardless of the formula’s original size.</p></li>
<li><p><strong>Conditional Image Masking:</strong> To further isolate formulas from surrounding text, conditional logic was added to use Docling’s <em>page.get_masked_image()</em> method exclusively for items labeled as DocItemLabel.FORMULA.</p></li>
</ul>
</section>
<section id="implementation-of-selective-content-masking" class="level4" data-number="7.1.5.5">
<h4 data-number="7.1.5.5" class="anchored" data-anchor-id="implementation-of-selective-content-masking"><span class="header-section-number">7.1.5.5</span> Implementation of Selective Content Masking</h4>
<p>The most critical enhancement was the development of a method to isolate formula elements from distracting adjacent text. When the layout model produced imperfect bounding boxes, snippets would often contain partial lines of text from the main body, confusing the formula recognition model. To solve this, a selective masking capability was engineered directly into the <em>Page</em> class within <em>docling/datamodel/base_models.py</em>.</p>
<p>This was achieved by introducing two new methods, <em>get_masked_image</em> and the internal *_create_masked_image<em>. The </em>_create_masked_image* function operates by first taking a complete, high-resolution image of a document page. It then iterates through all layout clusters previously identified by the layout model. For every cluster that is not labeled as a <em>DocItemLabel.FORMULA</em>, the function programmatically draws a white rectangle over its bounding box.</p>
<div id="create-masked-image" class="cell" data-execution_count="8">
<details class="code-fold">
<summary>Code</summary>
<div class="sourceCode cell-code" id="cb7"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb7-1"><a href="#cb7-1" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> _create_masked_image(<span class="va">self</span>, scale: <span class="bu">float</span>, pdf_identifier: Optional[<span class="bu">str</span>] <span class="op">=</span> <span class="va">None</span>) <span class="op">-></span> Optional[Image]:</span>
<span id="cb7-2"><a href="#cb7-2" aria-hidden="true" tabindex="-1"></a> <span class="cf">try</span>:</span>
<span id="cb7-3"><a href="#cb7-3" aria-hidden="true" tabindex="-1"></a> original_image <span class="op">=</span> <span class="va">self</span>.get_image(scale<span class="op">=</span>scale)</span>
<span id="cb7-4"><a href="#cb7-4" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> original_image <span class="kw">is</span> <span class="va">None</span> <span class="kw">or</span> <span class="va">self</span>.size <span class="kw">is</span> <span class="va">None</span>:</span>
<span id="cb7-5"><a href="#cb7-5" aria-hidden="true" tabindex="-1"></a> _log.warning(<span class="st">"Original image or size is None for page </span><span class="sc">%s</span><span class="st">."</span>, <span class="va">self</span>.page_no)</span>
<span id="cb7-6"><a href="#cb7-6" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> <span class="va">None</span></span>
<span id="cb7-7"><a href="#cb7-7" aria-hidden="true" tabindex="-1"></a> masked <span class="op">=</span> original_image.copy()</span>
<span id="cb7-8"><a href="#cb7-8" aria-hidden="true" tabindex="-1"></a> draw <span class="op">=</span> ImageDraw.Draw(masked)</span>
<span id="cb7-9"><a href="#cb7-9" aria-hidden="true" tabindex="-1"></a> scale_x <span class="op">=</span> masked.width <span class="op">/</span> <span class="va">self</span>.size.width</span>
<span id="cb7-10"><a href="#cb7-10" aria-hidden="true" tabindex="-1"></a> scale_y <span class="op">=</span> masked.height <span class="op">/</span> <span class="va">self</span>.size.height</span>
<span id="cb7-11"><a href="#cb7-11" aria-hidden="true" tabindex="-1"></a> top_expansion_factor <span class="op">=</span> <span class="fl">0.045</span></span>
<span id="cb7-12"><a href="#cb7-12" aria-hidden="true" tabindex="-1"></a> bottom_expansion_factor <span class="op">=</span> <span class="fl">0.045</span></span>
<span id="cb7-13"><a href="#cb7-13" aria-hidden="true" tabindex="-1"></a> cluster_count <span class="op">=</span> <span class="bu">len</span>(<span class="va">self</span>.predictions.layout.clusters)</span>
<span id="cb7-14"><a href="#cb7-14" aria-hidden="true" tabindex="-1"></a> _log.debug(<span class="st">"Creating masked image for page </span><span class="sc">%s</span><span class="st"> with </span><span class="sc">%d</span><span class="st"> clusters."</span>, <span class="va">self</span>.page_no, cluster_count)</span>
<span id="cb7-15"><a href="#cb7-15" aria-hidden="true" tabindex="-1"></a> <span class="co"># Mask non-formula clusters explicitly here with correct scaling</span></span>
<span id="cb7-16"><a href="#cb7-16" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span> cl <span class="kw">in</span> <span class="va">self</span>.predictions.layout.clusters:</span>
<span id="cb7-17"><a href="#cb7-17" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> cl.label <span class="op">!=</span> DocItemLabel.FORMULA:</span>
<span id="cb7-18"><a href="#cb7-18" aria-hidden="true" tabindex="-1"></a> bbox <span class="op">=</span> cl.bbox.to_top_left_origin(page_height<span class="op">=</span><span class="va">self</span>.size.height)</span>
<span id="cb7-19"><a href="#cb7-19" aria-hidden="true" tabindex="-1"></a> bbox_height <span class="op">=</span> bbox.b <span class="op">-</span> bbox.t</span>
<span id="cb7-20"><a href="#cb7-20" aria-hidden="true" tabindex="-1"></a> expanded_bbox <span class="op">=</span> BoundingBox(</span>
<span id="cb7-21"><a href="#cb7-21" aria-hidden="true" tabindex="-1"></a> l<span class="op">=</span>bbox.l,</span>
<span id="cb7-22"><a href="#cb7-22" aria-hidden="true" tabindex="-1"></a> t<span class="op">=</span>bbox.t <span class="op">-</span> bbox_height <span class="op">*</span> top_expansion_factor,</span>
<span id="cb7-23"><a href="#cb7-23" aria-hidden="true" tabindex="-1"></a> r<span class="op">=</span>bbox.r,</span>
<span id="cb7-24"><a href="#cb7-24" aria-hidden="true" tabindex="-1"></a> b<span class="op">=</span>bbox.b <span class="op">+</span> bbox_height <span class="op">*</span> bottom_expansion_factor,</span>
<span id="cb7-25"><a href="#cb7-25" aria-hidden="true" tabindex="-1"></a> coord_origin<span class="op">=</span>bbox.coord_origin,</span>
<span id="cb7-26"><a href="#cb7-26" aria-hidden="true" tabindex="-1"></a> ) </span>
<span id="cb7-27"><a href="#cb7-27" aria-hidden="true" tabindex="-1"></a> scaled_bbox <span class="op">=</span> BoundingBox(</span>
<span id="cb7-28"><a href="#cb7-28" aria-hidden="true" tabindex="-1"></a> l<span class="op">=</span>expanded_bbox.l <span class="op">*</span> scale_x,</span>
<span id="cb7-29"><a href="#cb7-29" aria-hidden="true" tabindex="-1"></a> t<span class="op">=</span>expanded_bbox.t <span class="op">*</span> scale_y,</span>
<span id="cb7-30"><a href="#cb7-30" aria-hidden="true" tabindex="-1"></a> r<span class="op">=</span>expanded_bbox.r <span class="op">*</span> scale_x,</span>
<span id="cb7-31"><a href="#cb7-31" aria-hidden="true" tabindex="-1"></a> b<span class="op">=</span>expanded_bbox.b <span class="op">*</span> scale_y,</span>
<span id="cb7-32"><a href="#cb7-32" aria-hidden="true" tabindex="-1"></a> coord_origin<span class="op">=</span>expanded_bbox.coord_origin</span>
<span id="cb7-33"><a href="#cb7-33" aria-hidden="true" tabindex="-1"></a> )</span>
<span id="cb7-34"><a href="#cb7-34" aria-hidden="true" tabindex="-1"></a> draw.rectangle(scaled_bbox.as_tuple(), fill<span class="op">=</span><span class="st">"white"</span>)</span>
<span id="cb7-35"><a href="#cb7-35" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> masked</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
</details>
</div>
<p>The result is a new, “masked” image of the page where all non-formula content has been completely obscured, as demonstrated in the comparison between an original page layout (<a href="#fig-unmasked" class="quarto-xref">Figure <span>7.4</span></a>) and its masked counterpart (<a href="#fig-masked" class="quarto-xref">Figure <span>7.5</span></a>).</p>
<div id="fig-unmasked" class="quarto-float quarto-figure quarto-figure-left anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-unmasked-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="figures/postprocessed_layout_page_00020.png" class="img-fluid figure-img">
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-unmasked-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure 7.4: Unmasked Page
</figcaption>
</figure>
</div>
<div id="fig-masked" class="quarto-float quarto-figure quarto-figure-left anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-masked-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="figures/masked_page_00020_scale_2.6.png" class="img-fluid figure-img">
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-masked-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure 7.5: Masked Page
</figcaption>
</figure>
</div>
<p>When the downstream <em>prepare_element</em> method in <em>code_formula_model.py</em> is called, it now uses this <em>get_masked_image</em> function instead of the standard <em>get_image</em>. By cropping the formula’s bounding box from this pre-masked page, the system guarantees that the final image snippet sent to the SmolDocling model contains only the formula itself, significantly improving the signal-to-noise ratio and preventing recognition errors caused by textual artifacts.</p>
</section>
<section id="heuristic-based-merging-of-fragmented-layout-clusters" class="level4" data-number="7.1.5.6">
<h4 data-number="7.1.5.6" class="anchored" data-anchor-id="heuristic-based-merging-of-fragmented-layout-clusters"><span class="header-section-number">7.1.5.6</span> Heuristic-Based Merging of Fragmented Layout Clusters</h4>
<p>Beyond improving the quality of individual image snippets, a second significant challenge was identified in the output of the base layout analysis model: the fragmentation of single, cohesive document elements into multiple, distinct bounding boxes. This issue was particularly prevalent with complex, multi-line mathematical formulas, which were often incorrectly segmented into several independent <em>FORMULA</em> clusters, as exemplified in <a href="#fig-fragmented" class="quarto-xref">Figure <span>7.6</span></a>. This fragmentation would lead to the downstream formula recognition model receiving only partial equations, resulting in incomplete and unusable LaTeX output.</p>
<div id="fig-fragmented" class="quarto-float quarto-figure quarto-figure-left anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-fragmented-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="figures/postprocessed_layout_page_00009_bad.png" class="img-fluid figure-img">
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-fragmented-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure 7.6: Fragmented Formulas
</figcaption>
</figure>
</div>
<p>To resolve this, a sophisticated post-processing step was engineered within <em>docling/utils/layout_postprocessor.py</em>. The purpose of this step is to apply a series of rule-based heuristics to refine and correct the raw output of the initial layout model before the final document structure is assembled.</p>
<div id="merge-clusters" class="cell" data-execution_count="9">
<details class="code-fold">
<summary>Code</summary>
<div class="sourceCode cell-code" id="cb8"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb8-1"><a href="#cb8-1" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> _merge_vertically_adjacent_formulas(</span>
<span id="cb8-2"><a href="#cb8-2" aria-hidden="true" tabindex="-1"></a> <span class="va">self</span>,</span>
<span id="cb8-3"><a href="#cb8-3" aria-hidden="true" tabindex="-1"></a> clusters,</span>
<span id="cb8-4"><a href="#cb8-4" aria-hidden="true" tabindex="-1"></a> vertical_threshold_factor<span class="op">=</span><span class="fl">1.8</span>,</span>
<span id="cb8-5"><a href="#cb8-5" aria-hidden="true" tabindex="-1"></a> horizontal_overlap_threshold<span class="op">=</span><span class="fl">0.7</span>,</span>
<span id="cb8-6"><a href="#cb8-6" aria-hidden="true" tabindex="-1"></a> padding<span class="op">=</span><span class="dv">50</span>,</span>
<span id="cb8-7"><a href="#cb8-7" aria-hidden="true" tabindex="-1"></a> alignment_threshold<span class="op">=</span><span class="dv">20</span>, <span class="co"># pixel-based threshold</span></span>
<span id="cb8-8"><a href="#cb8-8" aria-hidden="true" tabindex="-1"></a> max_alignment_ratio<span class="op">=</span><span class="fl">0.2</span>, <span class="co"># ratio-based threshold (e.g., 0.2 means 20% mismatch)</span></span>
<span id="cb8-9"><a href="#cb8-9" aria-hidden="true" tabindex="-1"></a>):</span>
<span id="cb8-10"><a href="#cb8-10" aria-hidden="true" tabindex="-1"></a> <span class="co">"""</span></span>
<span id="cb8-11"><a href="#cb8-11" aria-hidden="true" tabindex="-1"></a><span class="co"> merge vertically adjacent FORMULA clusters using Union-Find.</span></span>
<span id="cb8-12"><a href="#cb8-12" aria-hidden="true" tabindex="-1"></a><span class="co"> New logic:</span></span>
<span id="cb8-13"><a href="#cb8-13" aria-hidden="true" tabindex="-1"></a><span class="co"> 1) Compute a dynamic vertical threshold based on median formula height.</span></span>
<span id="cb8-14"><a href="#cb8-14" aria-hidden="true" tabindex="-1"></a><span class="co"> 2) Expand bounding boxes horizontally using padding to compute the horizontal overlap.</span></span>
<span id="cb8-15"><a href="#cb8-15" aria-hidden="true" tabindex="-1"></a><span class="co"> 3) Compute both absolute (pixel) differences and ratio-based alignment.</span></span>
<span id="cb8-16"><a href="#cb8-16" aria-hidden="true" tabindex="-1"></a><span class="co"> 4) Extract formula numbers from cell texts. If both clusters have numbers and they differ, skip merge.</span></span>
<span id="cb8-17"><a href="#cb8-17" aria-hidden="true" tabindex="-1"></a><span class="co"> 5) If neither cluster has a formula number, require that the vertical gap is very small (<= 10 px)</span></span>
<span id="cb8-18"><a href="#cb8-18" aria-hidden="true" tabindex="-1"></a><span class="co"> and that the horizontal overlap is nearly complete (>= 0.9).</span></span>
<span id="cb8-19"><a href="#cb8-19" aria-hidden="true" tabindex="-1"></a><span class="co"> 6) Otherwise, use the normal criteria based on vertical gap (and derived thresholds).</span></span>
<span id="cb8-20"><a href="#cb8-20" aria-hidden="true" tabindex="-1"></a><span class="co"> """</span></span>
<span id="cb8-21"><a href="#cb8-21" aria-hidden="true" tabindex="-1"></a> <span class="kw">def</span> compute_alignment_factor(c1, c2):</span>
<span id="cb8-22"><a href="#cb8-22" aria-hidden="true" tabindex="-1"></a> <span class="co"># Compute the average width and return the maximum normalized edge difference.</span></span>
<span id="cb8-23"><a href="#cb8-23" aria-hidden="true" tabindex="-1"></a> w1 <span class="op">=</span> c1.bbox.r <span class="op">-</span> c1.bbox.l</span>
<span id="cb8-24"><a href="#cb8-24" aria-hidden="true" tabindex="-1"></a> w2 <span class="op">=</span> c2.bbox.r <span class="op">-</span> c2.bbox.l</span>
<span id="cb8-25"><a href="#cb8-25" aria-hidden="true" tabindex="-1"></a> avg_width <span class="op">=</span> <span class="bu">max</span>((w1 <span class="op">+</span> w2) <span class="op">/</span> <span class="fl">2.0</span>, <span class="fl">1e-6</span>)</span>
<span id="cb8-26"><a href="#cb8-26" aria-hidden="true" tabindex="-1"></a> left_diff <span class="op">=</span> <span class="bu">abs</span>(c1.bbox.l <span class="op">-</span> c2.bbox.l)</span>
<span id="cb8-27"><a href="#cb8-27" aria-hidden="true" tabindex="-1"></a> right_diff <span class="op">=</span> <span class="bu">abs</span>(c1.bbox.r <span class="op">-</span> c2.bbox.r)</span>
<span id="cb8-28"><a href="#cb8-28" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> <span class="bu">max</span>(left_diff <span class="op">/</span> avg_width, right_diff <span class="op">/</span> avg_width)</span>
<span id="cb8-29"><a href="#cb8-29" aria-hidden="true" tabindex="-1"></a> <span class="co"># Only process clusters labelled as FORMULA.</span></span>
<span id="cb8-30"><a href="#cb8-30" aria-hidden="true" tabindex="-1"></a> formula_clusters <span class="op">=</span> [c <span class="cf">for</span> c <span class="kw">in</span> clusters <span class="cf">if</span> c.label <span class="op">==</span> DocItemLabel.FORMULA]</span>
<span id="cb8-31"><a href="#cb8-31" aria-hidden="true" tabindex="-1"></a> non_formula_clusters <span class="op">=</span> [c <span class="cf">for</span> c <span class="kw">in</span> clusters <span class="cf">if</span> c.label <span class="op">!=</span> DocItemLabel.FORMULA]</span>
<span id="cb8-32"><a href="#cb8-32" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> <span class="kw">not</span> formula_clusters:</span>
<span id="cb8-33"><a href="#cb8-33" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> clusters</span>
<span id="cb8-34"><a href="#cb8-34" aria-hidden="true" tabindex="-1"></a> <span class="co"># Sort by the top coordinate.</span></span>
<span id="cb8-35"><a href="#cb8-35" aria-hidden="true" tabindex="-1"></a> formula_clusters.sort(key<span class="op">=</span><span class="kw">lambda</span> c: c.bbox.t)</span>
<span id="cb8-36"><a href="#cb8-36" aria-hidden="true" tabindex="-1"></a> heights <span class="op">=</span> [c.bbox.b <span class="op">-</span> c.bbox.t <span class="cf">for</span> c <span class="kw">in</span> formula_clusters]</span>
<span id="cb8-37"><a href="#cb8-37" aria-hidden="true" tabindex="-1"></a> median_height <span class="op">=</span> np.median(heights) <span class="cf">if</span> heights <span class="cf">else</span> <span class="dv">0</span></span>
<span id="cb8-38"><a href="#cb8-38" aria-hidden="true" tabindex="-1"></a> vertical_threshold <span class="op">=</span> median_height <span class="op">*</span> vertical_threshold_factor</span>
<span id="cb8-39"><a href="#cb8-39" aria-hidden="true" tabindex="-1"></a> <span class="co"># Build a mapping from cluster id to cluster instance.</span></span>
<span id="cb8-40"><a href="#cb8-40" aria-hidden="true" tabindex="-1"></a> id_to_cluster <span class="op">=</span> {c.<span class="bu">id</span>: c <span class="cf">for</span> c <span class="kw">in</span> formula_clusters}</span>
<span id="cb8-41"><a href="#cb8-41" aria-hidden="true" tabindex="-1"></a> uf <span class="op">=</span> UnionFind(<span class="bu">list</span>(id_to_cluster.keys()))</span>
<span id="cb8-42"><a href="#cb8-42" aria-hidden="true" tabindex="-1"></a> n <span class="op">=</span> <span class="bu">len</span>(formula_clusters)</span>
<span id="cb8-43"><a href="#cb8-43" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span> i <span class="kw">in</span> <span class="bu">range</span>(n):</span>
<span id="cb8-44"><a href="#cb8-44" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span> j <span class="kw">in</span> <span class="bu">range</span>(i <span class="op">+</span> <span class="dv">1</span>, n):</span>
<span id="cb8-45"><a href="#cb8-45" aria-hidden="true" tabindex="-1"></a> c1 <span class="op">=</span> formula_clusters[i]</span>
<span id="cb8-46"><a href="#cb8-46" aria-hidden="true" tabindex="-1"></a> c2 <span class="op">=</span> formula_clusters[j]</span>
<span id="cb8-47"><a href="#cb8-47" aria-hidden="true" tabindex="-1"></a> <span class="co"># Compute vertical gap (c2 is assumed to be below c1)</span></span>
<span id="cb8-48"><a href="#cb8-48" aria-hidden="true" tabindex="-1"></a> vertical_gap <span class="op">=</span> c2.bbox.t <span class="op">-</span> c1.bbox.b</span>
<span id="cb8-49"><a href="#cb8-49" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> vertical_gap <span class="op"><</span> <span class="dv">0</span> <span class="kw">or</span> vertical_gap <span class="op">></span> vertical_threshold:</span>
<span id="cb8-50"><a href="#cb8-50" aria-hidden="true" tabindex="-1"></a> <span class="cf">continue</span></span>
<span id="cb8-51"><a href="#cb8-51" aria-hidden="true" tabindex="-1"></a> <span class="co"># Expand bounding boxes horizontally using the given padding.</span></span>
<span id="cb8-52"><a href="#cb8-52" aria-hidden="true" tabindex="-1"></a> expanded_bbox_c1 <span class="op">=</span> <span class="bu">type</span>(c1.bbox)(</span>
<span id="cb8-53"><a href="#cb8-53" aria-hidden="true" tabindex="-1"></a> l<span class="op">=</span>c1.bbox.l <span class="op">-</span> padding,</span>
<span id="cb8-54"><a href="#cb8-54" aria-hidden="true" tabindex="-1"></a> t<span class="op">=</span>c1.bbox.t,</span>
<span id="cb8-55"><a href="#cb8-55" aria-hidden="true" tabindex="-1"></a> r<span class="op">=</span>c1.bbox.r <span class="op">+</span> padding,</span>
<span id="cb8-56"><a href="#cb8-56" aria-hidden="true" tabindex="-1"></a> b<span class="op">=</span>c1.bbox.b</span>
<span id="cb8-57"><a href="#cb8-57" aria-hidden="true" tabindex="-1"></a> )</span>
<span id="cb8-58"><a href="#cb8-58" aria-hidden="true" tabindex="-1"></a> expanded_bbox_c2 <span class="op">=</span> <span class="bu">type</span>(c2.bbox)(</span>
<span id="cb8-59"><a href="#cb8-59" aria-hidden="true" tabindex="-1"></a> l<span class="op">=</span>c2.bbox.l <span class="op">-</span> padding,</span>
<span id="cb8-60"><a href="#cb8-60" aria-hidden="true" tabindex="-1"></a> t<span class="op">=</span>c2.bbox.t,</span>
<span id="cb8-61"><a href="#cb8-61" aria-hidden="true" tabindex="-1"></a> r<span class="op">=</span>c2.bbox.r <span class="op">+</span> padding,</span>
<span id="cb8-62"><a href="#cb8-62" aria-hidden="true" tabindex="-1"></a> b<span class="op">=</span>c2.bbox.b</span>
<span id="cb8-63"><a href="#cb8-63" aria-hidden="true" tabindex="-1"></a> )</span>
<span id="cb8-64"><a href="#cb8-64" aria-hidden="true" tabindex="-1"></a> horizontal_overlap <span class="op">=</span> <span class="bu">min</span>(expanded_bbox_c1.r, expanded_bbox_c2.r) <span class="op">-</span> <span class="bu">max</span>(expanded_bbox_c1.l, expanded_bbox_c2.l)</span>
<span id="cb8-65"><a href="#cb8-65" aria-hidden="true" tabindex="-1"></a> min_width <span class="op">=</span> <span class="bu">min</span>(expanded_bbox_c1.r <span class="op">-</span> expanded_bbox_c1.l, expanded_bbox_c2.r <span class="op">-</span> expanded_bbox_c2.l)</span>
<span id="cb8-66"><a href="#cb8-66" aria-hidden="true" tabindex="-1"></a> overlap_ratio <span class="op">=</span> horizontal_overlap <span class="op">/</span> min_width <span class="cf">if</span> min_width <span class="op">></span> <span class="dv">0</span> <span class="cf">else</span> <span class="dv">0</span></span>
<span id="cb8-67"><a href="#cb8-67" aria-hidden="true" tabindex="-1"></a> left_diff <span class="op">=</span> <span class="bu">abs</span>(c1.bbox.l <span class="op">-</span> c2.bbox.l)</span>
<span id="cb8-68"><a href="#cb8-68" aria-hidden="true" tabindex="-1"></a> right_diff <span class="op">=</span> <span class="bu">abs</span>(c1.bbox.r <span class="op">-</span> c2.bbox.r)</span>
<span id="cb8-69"><a href="#cb8-69" aria-hidden="true" tabindex="-1"></a> alignment_factor <span class="op">=</span> compute_alignment_factor(c1, c2)</span>
<span id="cb8-70"><a href="#cb8-70" aria-hidden="true" tabindex="-1"></a> <span class="co"># Extract formula numbers using _extract_formula_number.</span></span>
<span id="cb8-71"><a href="#cb8-71" aria-hidden="true" tabindex="-1"></a> num1 <span class="op">=</span> <span class="va">self</span>._extract_formula_number(c1)</span>
<span id="cb8-72"><a href="#cb8-72" aria-hidden="true" tabindex="-1"></a> num2 <span class="op">=</span> <span class="va">self</span>._extract_formula_number(c2)</span>
<span id="cb8-73"><a href="#cb8-73" aria-hidden="true" tabindex="-1"></a> <span class="co"># Branch 4) If both have numbers and differ, skip</span></span>
<span id="cb8-74"><a href="#cb8-74" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> num1 <span class="kw">and</span> num2 <span class="kw">and</span> num1 <span class="op">!=</span> num2:</span>
<span id="cb8-75"><a href="#cb8-75" aria-hidden="true" tabindex="-1"></a> <span class="cf">continue</span></span>
<span id="cb8-76"><a href="#cb8-76" aria-hidden="true" tabindex="-1"></a> <span class="co"># Normal merging criteria based on vertical gap and alignment.</span></span>
<span id="cb8-77"><a href="#cb8-77" aria-hidden="true" tabindex="-1"></a> <span class="co"># Pre-calculate default geometry-based threshold</span></span>
<span id="cb8-78"><a href="#cb8-78" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> vertical_gap <span class="op"><=</span> (<span class="fl">0.5</span> <span class="op">*</span> vertical_threshold):</span>
<span id="cb8-79"><a href="#cb8-79" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> left_diff <span class="op"><=</span> alignment_threshold <span class="kw">and</span> right_diff <span class="op"><=</span> alignment_threshold:</span>
<span id="cb8-80"><a href="#cb8-80" aria-hidden="true" tabindex="-1"></a> required_overlap <span class="op">=</span> horizontal_overlap_threshold</span>
<span id="cb8-81"><a href="#cb8-81" aria-hidden="true" tabindex="-1"></a> <span class="cf">else</span>:</span>
<span id="cb8-82"><a href="#cb8-82" aria-hidden="true" tabindex="-1"></a> required_overlap <span class="op">=</span> <span class="fl">0.85</span></span>
<span id="cb8-83"><a href="#cb8-83" aria-hidden="true" tabindex="-1"></a> <span class="cf">elif</span> vertical_gap <span class="op"><=</span> vertical_threshold:</span>
<span id="cb8-84"><a href="#cb8-84" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> alignment_factor <span class="op"><=</span> max_alignment_ratio:</span>
<span id="cb8-85"><a href="#cb8-85" aria-hidden="true" tabindex="-1"></a> required_overlap <span class="op">=</span> <span class="fl">0.85</span></span>
<span id="cb8-86"><a href="#cb8-86" aria-hidden="true" tabindex="-1"></a> <span class="cf">else</span>:</span>
<span id="cb8-87"><a href="#cb8-87" aria-hidden="true" tabindex="-1"></a> <span class="cf">continue</span></span>
<span id="cb8-88"><a href="#cb8-88" aria-hidden="true" tabindex="-1"></a> <span class="cf">else</span>:</span>
<span id="cb8-89"><a href="#cb8-89" aria-hidden="true" tabindex="-1"></a> <span class="cf">continue</span></span>
<span id="cb8-90"><a href="#cb8-90" aria-hidden="true" tabindex="-1"></a> <span class="co"># New branch: if no formula number is detected in either cluster,</span></span>
<span id="cb8-91"><a href="#cb8-91" aria-hidden="true" tabindex="-1"></a> <span class="co"># then enforce a very small vertical gap and nearly complete horizontal overlap.</span></span>
<span id="cb8-92"><a href="#cb8-92" aria-hidden="true" tabindex="-1"></a> <span class="co"># Branch 2) Both None => require vertical_gap ≤10 and overlap ≥0.9</span></span>
<span id="cb8-93"><a href="#cb8-93" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> num1 <span class="kw">is</span> <span class="va">None</span> <span class="kw">and</span> num2 <span class="kw">is</span> <span class="va">None</span>:</span>
<span id="cb8-94"><a href="#cb8-94" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> vertical_gap <span class="op">></span> <span class="dv">3</span>: <span class="co"># Change this constant if needed</span></span>
<span id="cb8-95"><a href="#cb8-95" aria-hidden="true" tabindex="-1"></a> <span class="cf">continue</span></span>
<span id="cb8-96"><a href="#cb8-96" aria-hidden="true" tabindex="-1"></a> <span class="co"># Require a higher overlap; allow slight relaxation if the normal criterion is higher.</span></span>
<span id="cb8-97"><a href="#cb8-97" aria-hidden="true" tabindex="-1"></a> required_overlap <span class="op">=</span> <span class="bu">max</span>(required_overlap, <span class="fl">0.9</span>)</span>
<span id="cb8-98"><a href="#cb8-98" aria-hidden="true" tabindex="-1"></a> <span class="co"># Branch 3) One number missing => require vertical_gap ≤5 and overlap ≥0.95</span></span>
<span id="cb8-99"><a href="#cb8-99" aria-hidden="true" tabindex="-1"></a> <span class="cf">elif</span> (num1 <span class="kw">is</span> <span class="va">None</span>) <span class="op">!=</span> (num2 <span class="kw">is</span> <span class="va">None</span>):</span>
<span id="cb8-100"><a href="#cb8-100" aria-hidden="true" tabindex="-1"></a> <span class="co"># i.e. exactly one side is missing a number</span></span>
<span id="cb8-101"><a href="#cb8-101" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> vertical_gap <span class="op">></span> <span class="fl">12.8</span>:</span>
<span id="cb8-102"><a href="#cb8-102" aria-hidden="true" tabindex="-1"></a> <span class="cf">continue</span></span>
<span id="cb8-103"><a href="#cb8-103" aria-hidden="true" tabindex="-1"></a> required_overlap <span class="op">=</span> <span class="bu">max</span>(required_overlap, <span class="fl">0.95</span>)</span>
<span id="cb8-104"><a href="#cb8-104" aria-hidden="true" tabindex="-1"></a> <span class="co"># Branch 1) If both exist and match => use normal geometry (already in required_overlap)</span></span>
<span id="cb8-105"><a href="#cb8-105" aria-hidden="true" tabindex="-1"></a> <span class="co"># Final merge check</span></span>
<span id="cb8-106"><a href="#cb8-106" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> overlap_ratio <span class="op">>=</span> required_overlap:</span>
<span id="cb8-107"><a href="#cb8-107" aria-hidden="true" tabindex="-1"></a> uf.union(c1.<span class="bu">id</span>, c2.<span class="bu">id</span>)</span>
<span id="cb8-108"><a href="#cb8-108" aria-hidden="true" tabindex="-1"></a> groups <span class="op">=</span> uf.get_groups()</span>
<span id="cb8-109"><a href="#cb8-109" aria-hidden="true" tabindex="-1"></a> merged_formula_clusters <span class="op">=</span> []</span>
<span id="cb8-110"><a href="#cb8-110" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span> group_ids <span class="kw">in</span> groups.values():</span>
<span id="cb8-111"><a href="#cb8-111" aria-hidden="true" tabindex="-1"></a> group_clusters <span class="op">=</span> [id_to_cluster[g] <span class="cf">for</span> g <span class="kw">in</span> group_ids]</span>
<span id="cb8-112"><a href="#cb8-112" aria-hidden="true" tabindex="-1"></a> merged_bbox <span class="op">=</span> <span class="bu">type</span>(group_clusters[<span class="dv">0</span>].bbox)(</span>
<span id="cb8-113"><a href="#cb8-113" aria-hidden="true" tabindex="-1"></a> l<span class="op">=</span><span class="bu">min</span>(c.bbox.l <span class="cf">for</span> c <span class="kw">in</span> group_clusters),</span>
<span id="cb8-114"><a href="#cb8-114" aria-hidden="true" tabindex="-1"></a> t<span class="op">=</span><span class="bu">min</span>(c.bbox.t <span class="cf">for</span> c <span class="kw">in</span> group_clusters),</span>
<span id="cb8-115"><a href="#cb8-115" aria-hidden="true" tabindex="-1"></a> r<span class="op">=</span><span class="bu">max</span>(c.bbox.r <span class="cf">for</span> c <span class="kw">in</span> group_clusters),</span>
<span id="cb8-116"><a href="#cb8-116" aria-hidden="true" tabindex="-1"></a> b<span class="op">=</span><span class="bu">max</span>(c.bbox.b <span class="cf">for</span> c <span class="kw">in</span> group_clusters)</span>
<span id="cb8-117"><a href="#cb8-117" aria-hidden="true" tabindex="-1"></a> )</span>
<span id="cb8-118"><a href="#cb8-118" aria-hidden="true" tabindex="-1"></a> combined_cells <span class="op">=</span> []</span>
<span id="cb8-119"><a href="#cb8-119" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span> c <span class="kw">in</span> group_clusters:</span>
<span id="cb8-120"><a href="#cb8-120" aria-hidden="true" tabindex="-1"></a> combined_cells.extend(c.cells)</span>
<span id="cb8-121"><a href="#cb8-121" aria-hidden="true" tabindex="-1"></a> merged_cluster <span class="op">=</span> group_clusters[<span class="dv">0</span>]</span>
<span id="cb8-122"><a href="#cb8-122" aria-hidden="true" tabindex="-1"></a> merged_cluster.bbox <span class="op">=</span> merged_bbox</span>
<span id="cb8-123"><a href="#cb8-123" aria-hidden="true" tabindex="-1"></a> merged_cluster.cells <span class="op">=</span> <span class="va">self</span>._sort_cells(<span class="va">self</span>._deduplicate_cells(combined_cells))</span>
<span id="cb8-124"><a href="#cb8-124" aria-hidden="true" tabindex="-1"></a> merged_formula_clusters.append(merged_cluster)</span>
<span id="cb8-125"><a href="#cb8-125" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> non_formula_clusters <span class="op">+</span> merged_formula_clusters</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
</details>
</div>
<p>The core of this enhancement is the new *_merge_vertically_adjacent_formulas* method. This algorithm intelligently identifies and merges clusters that likely belong to the same formula. It operates on a set of heuristics designed to distinguish between separate, adjacent equations and multiple lines of a single equation:</p>
<ol type="1">
<li><p><strong>Dynamic Proximity Threshold:</strong> Rather than using a fixed pixel distance, the method first calculates the median height of all detected formulas on a page. It then defines a dynamic vertical proximity threshold based on a factor of this median height, allowing it to adapt to documents with different font sizes and line spacing.</p></li>
<li><p><strong>Formula Number Extraction:</strong> A helper function, _extract_formula_number, uses regular expressions to find equation numbers (e.g., (1), (2a), (A5)) within the text of each formula cluster.</p></li>
<li><p><strong>Contextual Merging Logic:</strong> The algorithm iterates through pairs of vertically adjacent formula clusters and applies a decision-making process:</p>
<ul>
<li>If both clusters have distinct formula numbers (e.g., (1) and (2)), they are correctly identified as separate equations and are not merged.<br>
</li>
<li>If neither cluster has a formula number, they are only merged if the vertical gap between them is minimal and their horizontal overlap is nearly complete, which is characteristic of multi-line equations without a single encompassing number.<br>
</li>
<li>If one cluster has a number and the adjacent one does not, a stricter set of proximity and overlap rules is applied.</li>
</ul></li>
</ol>
<p>Once candidate clusters for merging are identified, a Union-Find data structure is used to efficiently group them. The final step involves creating a new, single bounding box that encompasses all the merged clusters and combining their constituent text cells. This new, unified cluster is then passed to the next stage of the pipeline, ensuring the entire formula is processed as a single unit as demonstrated in <a href="#fig-unfragmented" class="quarto-xref">Figure <span>7.7</span></a>. Additional heuristics, such as the *_filter_tables_containing_page_footer* method, were also added to this module to correct other common layout analysis errors, further improving the overall quality of the parsed document structure.</p>
<div id="fig-unfragmented" class="quarto-float quarto-figure quarto-figure-left anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-unfragmented-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="figures/postprocessed_layout_page_00009_crop_carb.png" class="img-fluid figure-img">
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-unfragmented-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure 7.7: Unfragmented Formulas
</figcaption>
</figure>
</div>
</section>
<section id="misclassification-of-page-layouts" class="level4" data-number="7.1.5.7">
<h4 data-number="7.1.5.7" class="anchored" data-anchor-id="misclassification-of-page-layouts"><span class="header-section-number">7.1.5.7</span> Misclassification of Page Layouts</h4>
<p>This analysis details further enhancements made to the layout post-processing logic to address specific, recurring classification errors observed in certain document types, such as pre-prints or articles with line numbers.</p>
<p>A common failure mode was observed where the layout detection model would misclassify an entire page or large sections of a page as a single, large TABLE cluster. This issue was particularly prevalent in documents that featured line numbers running down the left-hand margin, as seen in <a href="#fig-table-misscl" class="quarto-xref">Figure <span>7.8</span></a>. The presence of this repeating, structured numerical column appeared to mislead the model into perceiving the entire text block as tabular data. This resulted in the loss of all other semantic distinctions for that page (e.g., paragraphs, headers, lists), rendering the content unusable for downstream extraction.</p>
<div id="fig-table-misscl" class="quarto-float quarto-figure quarto-figure-left anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-table-misscl-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="figures/utility_postprocessed_layout_page_00003_before_fix_attempt.png" class="img-fluid figure-img">
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-table-misscl-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure 7.8: Table Missclassification
</figcaption>
</figure>
</div>
</section>
<section id="heuristic-based-re-classification-of-misidentified-clusters" class="level4" data-number="7.1.5.8">
<h4 data-number="7.1.5.8" class="anchored" data-anchor-id="heuristic-based-re-classification-of-misidentified-clusters"><span class="header-section-number">7.1.5.8</span> Heuristic-Based Re-classification of Misidentified Clusters</h4>
<p>To correct this, a new heuristic-based filtering method, *_filter_tables_containing_page_footer*, was engineered and integrated into the main postprocess pipeline. While the method name suggests a focus on page footers, its logic was designed more broadly to identify and re-classify any cluster that was erroneously labeled as a table-like structure when it was, in fact, the main body text of the page.</p>
<div id="filter-large-tables" class="cell" data-execution_count="10">
<details class="code-fold">
<summary>Code</summary>
<div class="sourceCode cell-code" id="cb9"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb9-1"><a href="#cb9-1" aria-hidden="true" tabindex="-1"></a><span class="co"># In scripts/docling/utils/layout_postprocessor.py</span></span>
<span id="cb9-2"><a href="#cb9-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb9-3"><a href="#cb9-3" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> _filter_tables_containing_page_footer(<span class="va">self</span>, clusters: List[Cluster], </span>
<span id="cb9-4"><a href="#cb9-4" aria-hidden="true" tabindex="-1"></a> min_area_ratio: <span class="bu">float</span> <span class="op">=</span> <span class="fl">0.70</span>, </span>
<span id="cb9-5"><a href="#cb9-5" aria-hidden="true" tabindex="-1"></a> min_cells_threshold: <span class="bu">int</span> <span class="op">=</span> <span class="dv">50</span>,</span>
<span id="cb9-6"><a href="#cb9-6" aria-hidden="true" tabindex="-1"></a> min_density_threshold: <span class="bu">float</span> <span class="op">=</span> <span class="fl">0.001</span>) <span class="op">-></span> List[Cluster]:</span>
<span id="cb9-7"><a href="#cb9-7" aria-hidden="true" tabindex="-1"></a> <span class="co">"""</span></span>
<span id="cb9-8"><a href="#cb9-8" aria-hidden="true" tabindex="-1"></a><span class="co"> Enhanced logic to avoid reclassifying legitimate large tables.</span></span>
<span id="cb9-9"><a href="#cb9-9" aria-hidden="true" tabindex="-1"></a><span class="co"> """</span></span>
<span id="cb9-10"><a href="#cb9-10" aria-hidden="true" tabindex="-1"></a> large_labels <span class="op">=</span> {DocItemLabel.TABLE, DocItemLabel.DOCUMENT_INDEX, </span>
<span id="cb9-11"><a href="#cb9-11" aria-hidden="true" tabindex="-1"></a> DocItemLabel.KEY_VALUE_REGION, DocItemLabel.FORM}</span>
<span id="cb9-12"><a href="#cb9-12" aria-hidden="true" tabindex="-1"></a> page_area <span class="op">=</span> <span class="va">self</span>.page_size.width <span class="op">*</span> <span class="va">self</span>.page_size.height</span>
<span id="cb9-13"><a href="#cb9-13" aria-hidden="true" tabindex="-1"></a> reclassified_count <span class="op">=</span> <span class="dv">0</span></span>
<span id="cb9-14"><a href="#cb9-14" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb9-15"><a href="#cb9-15" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span> cluster <span class="kw">in</span> clusters:</span>
<span id="cb9-16"><a href="#cb9-16" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> cluster.label <span class="kw">in</span> large_labels:</span>
<span id="cb9-17"><a href="#cb9-17" aria-hidden="true" tabindex="-1"></a> cluster_area_ratio <span class="op">=</span> cluster.bbox.area() <span class="op">/</span> page_area</span>
<span id="cb9-18"><a href="#cb9-18" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> cluster_area_ratio <span class="op">>=</span> min_area_ratio:</span>
<span id="cb9-19"><a href="#cb9-19" aria-hidden="true" tabindex="-1"></a> all_cells <span class="op">=</span> <span class="va">self</span>._collect_all_cells(cluster)</span>
<span id="cb9-20"><a href="#cb9-20" aria-hidden="true" tabindex="-1"></a> cell_density <span class="op">=</span> <span class="bu">len</span>(all_cells) <span class="op">/</span> cluster.bbox.area()</span>
<span id="cb9-21"><a href="#cb9-21" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb9-22"><a href="#cb9-22" aria-hidden="true" tabindex="-1"></a> <span class="co"># Reclassify if the cluster is large but sparsely populated with text cells</span></span>
<span id="cb9-23"><a href="#cb9-23" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> <span class="bu">len</span>(all_cells) <span class="op"><</span> min_cells_threshold <span class="kw">or</span> cell_density <span class="op"><</span> min_density_threshold:</span>
<span id="cb9-24"><a href="#cb9-24" aria-hidden="true" tabindex="-1"></a> cluster.label <span class="op">=</span> DocItemLabel.TEXT</span>
<span id="cb9-25"><a href="#cb9-25" aria-hidden="true" tabindex="-1"></a> reclassified_count <span class="op">+=</span> <span class="dv">1</span></span>
<span id="cb9-26"><a href="#cb9-26" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> clusters</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
</details>
</div>
<p>The algorithm operates on a set of carefully tuned heuristics to distinguish between a legitimate, large table and a misclassified text block:</p>
<ol type="1">
<li><p><strong>Target Selection:</strong> The function first identifies any cluster labeled as a <em>TABLE</em> (or other large “wrapper” types like <em>FORM</em>) for inspection.</p></li>
<li><p><strong>Area Threshold:</strong> It only considers clusters that are exceptionally large, occupying a significant portion of the page (e.g., <em>min_area_ratio</em> of 70%). This prevents the algorithm from affecting smaller, legitimate tables.</p></li>
<li><p><strong>Cell Count & Density Check:</strong> This is the core heuristic. It recursively collects all the individual text cells within the large cluster and calculates two metrics: the absolute number of cells and the “cell density” (number of cells per unit of bounding box area).</p></li>
<li><p><strong>Re-classification Logic:</strong> If a cluster is very large but contains a low number of text cells or has a very low cell density, it is highly unlikely to be a real table. In such cases, the algorithm re-classifies the cluster’s label from <em>TABLE</em> to <em>TEXT</em>.</p></li>
</ol>
<p>By adding this function to the postprocess pipeline, the system can now correctly identify and relabel these large, sparse, misclassified text blocks, preserving the semantic integrity of the page structure as demonstrated by the corrected output in <a href="#fig-table-correction" class="quarto-xref">Figure <span>7.9</span></a>.</p>
<div id="fig-table-correction" class="quarto-float quarto-figure quarto-figure-left anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-table-correction-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="figures/utility_postprocessed_layout_page_00003_after.png" class="img-fluid figure-img">
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-table-correction-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure 7.9: Table Correction
</figcaption>
</figure>
</div>
</section>
<section id="pre-emptive-filtering-of-page-level-artifacts" class="level4" data-number="7.1.5.9">
<h4 data-number="7.1.5.9" class="anchored" data-anchor-id="pre-emptive-filtering-of-page-level-artifacts"><span class="header-section-number">7.1.5.9</span> Pre-emptive Filtering of Page-Level Artifacts</h4>
<p>In addition to the post-processing of layout clusters, further enhancements were engineered upstream in the <em>docling/backend/docling_parse_v4_backend.py</em> module to improve the initial quality of the data fed to the layout analysis model. This module serves as a low-level backend that interacts directly with the <em>docling-parse</em> C++ library, which is responsible for the initial extraction of raw text cells and their coordinates from the PDF document. It was observed that certain document formats, particularly pre-prints, often contain page-level artifacts like line numbers in the margins. These structured, non-substantive elements were found to frequently mislead the layout model, leading to catastrophic classification errors where an entire page of text would be misidentified as a single, large <em>TABLE</em> cluster.</p>
<p>To mitigate this failure mode, the primary modification was made to the <em>get_text_cells</em> method within the <em>DoclingParseV4PageBackend</em> class. The original implementation of this method simply transformed the coordinate system of all extracted text cells and returned the complete, unfiltered list. The enhanced version introduces a new helper method, *_is_left_margin_line_number*, which applies a series of geometric heuristics to identify and pre-emptively filter these line-number artifacts before they are passed to the layout model.</p>
<p>The heuristic function determines if a given text cell is a line number by evaluating three specific spatial properties. First, it confirms the cell is located within a narrow vertical band on the far-left of the page, defined by a <em>LEFT_MARGIN_THRESHOLD</em> (e.g., the leftmost 8% of the page width). Second, it verifies that the cell’s bounding box width is less than this <em>MAX_WIDTH_THRESHOLD</em>, characteristic of short numerical strings. Finally, it ensures the cell has a reasonable height via <em>MIN_HEIGHT_THRESHOLD</em> to avoid incorrectly filtering other small page markings like footnote symbols. A cell is only flagged and removed if it satisfies all three conditions. By filtering the cell list with this function, the layout model receives a much cleaner representation of the page’s semantic content, significantly reducing classification errors and preserving the integrity of the document structure for all downstream processing.</p>
<p><strong>Original Implementation (get_text_cells)</strong></p>
<div id="text-cells-og" class="cell" data-execution_count="11">
<details class="code-fold">
<summary>Code</summary>
<div class="sourceCode cell-code" id="cb10"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb10-1"><a href="#cb10-1" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> get_text_cells(<span class="va">self</span>) <span class="op">-></span> Iterable[TextCell]:</span>
<span id="cb10-2"><a href="#cb10-2" aria-hidden="true" tabindex="-1"></a> page_size <span class="op">=</span> <span class="va">self</span>.get_size()</span>
<span id="cb10-3"><a href="#cb10-3" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb10-4"><a href="#cb10-4" aria-hidden="true" tabindex="-1"></a> <span class="co"># Applies coordinate transformation but returns all cells</span></span>
<span id="cb10-5"><a href="#cb10-5" aria-hidden="true" tabindex="-1"></a> [tc.to_top_left_origin(page_size.height) <span class="cf">for</span> tc <span class="kw">in</span> <span class="va">self</span>._dpage.textline_cells]</span>
<span id="cb10-6"><a href="#cb10-6" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb10-7"><a href="#cb10-7" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> <span class="va">self</span>._dpage.textline_cells</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
</details>
</div>
<p><strong>Modified Implementation with Heuristic Filtering</strong></p>
<div id="text-cells-mod" class="cell" data-execution_count="12">
<details class="code-fold">