Update documentation (#249)

* update documentation and dependencies page
* Add study on thesis extraction (up to 1Gb PDF files)

---------

Signed-off-by: Luca Foppiano <luca@foppiano.org>

Author: lfoppiano

Dependencies_INSTALL.md

@@ -1,9 +1,18 @@
 # External Libraries
 
-Here we describe procedures to compile the static dependencies.
+> **You do not need this to build pdfalto.** The static libraries and their headers are committed under `libs/` for
+> every supported platform, so a normal build just works — see [Build](Readme.md#build). This document describes how
+> those libraries are produced, and is only relevant when updating a dependency version or porting to a new platform.
 
-The include and the libraries are shipped under `libs`, you can also find the artifact in the bottom of the page of 
-each [build](https://github.com/kermitt2/pdfalto/actions) under [Manual build static libraries](https://github.com/kermitt2/pdfalto/actions/workflows/ci-build-libs.yml).  
+Normally the libraries are built by the
+[`ci-build-libs.yml`](https://github.com/kermitt2/pdfalto/blob/master/.github/workflows/ci-build-libs.yml) workflow
+("Manual build static libraries"), which runs the procedures below across a matrix of Linux and macOS runners and
+publishes the results as artifacts at the bottom of each [run](https://github.com/kermitt2/pdfalto/actions). Updating
+a dependency means bumping its version variable at the top of that workflow, running it, and committing the resulting
+artifacts into `libs/` — the versions listed below should be kept in sync with it.
+
+The manual steps that follow are the fallback for a platform CI does not cover, or for debugging a dependency
+locally. `./install_deps.sh` automates the same procedures.
 
 ## Requirements
 
@@ -13,6 +22,8 @@ each [build](https://github.com/kermitt2/pdfalto/actions) under [Manual build st
 
 ## Current versions:
 
+(these mirror the version variables at the top of `ci-build-libs.yml`)
+
 - LibPNG: 1.6.56
 - zLib: 1.3.2
 - libxml2: 2.15.2
@@ -125,7 +136,7 @@ LibPNG and zLib sources are downloaded and built as part of the dependency build
    make install
     ```
 
-See [ICU Readme](http://source.icu-project.org/repos/icu/trunk/icu4c/readme.html) for futher details.
+See the [ICU4C build guide](https://unicode-org.github.io/icu/userguide/icu4c/build.html) for further details.
 
 7. Copy `output/lib/libicuuc.a` and `output/lib/libicudata.a` under `libs/icu/<OS>/<ARCH>/` and headers from `output/include/` under `libs/icu/include/`.
 

Readme.md

@@ -26,6 +26,8 @@ end-user.
 * compilers : clang > 5 or gcc > 7, c++17 required
 * makefile generator : cmake >= 3.10.0
 * fetching dependencies : wget
+* git : the Xpdf source is a git submodule, so the repository must be cloned (not downloaded as a zip) and the
+  submodule initialised with `git submodule update --init --recursive` — see [Build](#build)
 
 ## Usage
 
@@ -127,19 +129,25 @@ line, the following outputs the text content only:
 
 ## Dependencies
 
-Dependencies can be recompiled by running this [script](https://github.com/kermitt2/pdfalto/blob/master/install_deps.sh)
+**Nothing to do here for a normal build.** The static libraries pdfalto links against (libxml2, freetype, libpng,
+zlib, ICU) are committed to this repository under `libs/`, prebuilt for each supported platform (`linux/64`,
+`linux/arm64`, `mac/64`, `mac/arm64`, and windows). The xpdf language support packages are likewise committed under
+`languages/`. Cloning the repository is enough — go straight to [Build](#build).
 
-> ./install_deps.sh
+These libraries are produced by the
+[`ci-build-libs.yml`](https://github.com/kermitt2/pdfalto/blob/master/.github/workflows/ci-build-libs.yml) workflow,
+which builds every platform in a matrix and is run manually (`workflow_dispatch`) when a dependency needs updating.
+The library versions it pins are declared at the top of that file. Rebuilding them is a maintainer task, not a
+prerequisite for building pdfalto.
 
-The script will download and build the dependencies unders `libs/` and the additional language support packages for xpdf
-under `languages/`.
-
-If necessary, see [compiling dependencies procedures](Dependencies_INSTALL.md) for further details.
+If you do need to rebuild them locally — porting to a new platform, or bisecting a dependency — `./install_deps.sh`
+downloads and builds them into `libs/` and `languages/`; see
+[compiling dependencies procedures](Dependencies_INSTALL.md) for details.
 
 ### Known issues
 
 ([issue 41](https://github.com/kermitt2/pdfalto/issues/41)) might occur while building, in this case you'll need to
-compile the dependencies before building pdflato.
+compile the dependencies before building pdfalto.
 
 ## Build
 
@@ -187,7 +195,9 @@ building, see described workaround.
 
 - map special characters in secondary fonts to their expected unicode
 
-- try to optimize speed and memory
+- reduce the memory still retained per page in the text extraction path: peak memory no longer grows with the number
+  of pages, but some per-page allocations are released later than they could be, so very long documents (tens of
+  thousands of pages) still hold more than they need to
 
 - see the issue tracker for further tasks