Support for images

[christiandmiddel]

There has been a couple of requests coming to support images also.
I have been doing some investigation on this, still not complete.
Findings so far are:

• The german export of the wikipedia raw articles seem to reference 1.5 million images
• References in the raw wikitext seem to be language dependent (EN=File:, DE=Datei:, FR=Fichier)
• Rough estimate for the total size of the 1.5 million images is 2TB
• Certain percentage of the files cannot be downloaded due to copyright restrictions

Some initial download tests have shown, that a somehow robust mechanism has to be implemented because a download in one big chunk is impossible, so a certain storage is required where we keep all the file references and their actual download status.

[christiandmiddel]

The attached kraken mechanism seems to run stable over night, was running 10 hours without an issue. Transported approx. 80GB of data using 50 channels. We received 50.000 files including 1.500 HTTP 404 failures which turns into 3% overal progress

[christiandmiddel]

I finally chose MapDB over Nitrite as Nitrite had issues with iterating the collection and at the same time writing updates to it.

[christiandmiddel]

Just to keep a reminder on this, I have also seen that for any kind of reduction in size that we might want to consider with final image library, the wheel does not have to be reinvented. My linux box has ImageMagick installed that can do:

convert input.jpg -adaptive-resize 50% -quality 75 output.jpg

[christiandmiddel]

More rounds of downloading with 100 channels where showing similar result, 50.000 files download between 10 and 12 hours.