For disussion: Inflight unzip

[skjolber]

Summary:
Create a CSVInputSource which pulls data from an URL to download and parse at the same time.

• Adds commons-compress dependency for low-level ZIP access.
• HTTP cache approach with Ranges:
  • Segments for random access (i.e. read various headers / indexes)
    • Transferred on demand
  • ZIP Entry cache (compressed data)
    • Transferred in background thread

Expected behavior:
Improve overall time for download + parse.

For Entur's aggregated GTFS zip file (500+ MB):

Benchmark                                            (megabytesPerSecond)  Mode  Cnt    Score   Error  Units
GtfsDownloadParseBenchmark.testParseDirectlyFromURL                     5    ss       118,078           s/op
GtfsDownloadParseBenchmark.testParseDirectlyFromURL                    10    ss        61,820           s/op
GtfsDownloadParseBenchmark.testParseDirectlyFromURL                    15    ss        47,379           s/op
GtfsDownloadParseBenchmark.testParseDirectlyFromURL                    20    ss        48,587           s/op
GtfsDownloadParseBenchmark.testParseDirectlyFromURL                    25    ss        47,878           s/op
GtfsDownloadParseBenchmark.testParseDirectlyFromURL                    30    ss        47,813           s/op
GtfsDownloadParseBenchmark.testParseDirectlyFromURL                    40    ss        47,516           s/op
GtfsDownloadParseBenchmark.testParseDirectlyFromURL                    50    ss        47,388           s/op
GtfsDownloadParseBenchmark.testParseDirectlyFromURL                    60    ss        47,911           s/op
GtfsDownloadParseBenchmark.testParseDirectlyFromURL                    80    ss        48,128           s/op
GtfsDownloadParseBenchmark.testParseDirectlyFromURL                   100    ss        47,811           s/op
GtfsDownloadParseBenchmark.testParseDirectlyFromURL                   125    ss        49,662           s/op
GtfsDownloadParseBenchmark.testParseDirectlyFromURL                   150    ss        47,737           s/op
GtfsDownloadParseBenchmark.testParseDirectlyFromURL                   200    ss        48,675           s/op
GtfsDownloadParseBenchmark.testParseDirectlyFromURL                   500    ss        47,992           s/op
GtfsDownloadParseBenchmark.testTransferThenParse                        5    ss       164,664           s/op
GtfsDownloadParseBenchmark.testTransferThenParse                       10    ss       104,925           s/op
GtfsDownloadParseBenchmark.testTransferThenParse                       15    ss        85,917           s/op
GtfsDownloadParseBenchmark.testTransferThenParse                       20    ss        75,728           s/op
GtfsDownloadParseBenchmark.testTransferThenParse                       25    ss        71,392           s/op
GtfsDownloadParseBenchmark.testTransferThenParse                       30    ss        66,986           s/op
GtfsDownloadParseBenchmark.testTransferThenParse                       40    ss        61,982           s/op
GtfsDownloadParseBenchmark.testTransferThenParse                       50    ss        58,279           s/op
GtfsDownloadParseBenchmark.testTransferThenParse                       60    ss        56,672           s/op
GtfsDownloadParseBenchmark.testTransferThenParse                       80    ss        53,689           s/op
GtfsDownloadParseBenchmark.testTransferThenParse                      100    ss        54,637           s/op
GtfsDownloadParseBenchmark.testTransferThenParse                      125    ss        52,002           s/op
GtfsDownloadParseBenchmark.testTransferThenParse                      150    ss        50,476           s/op
GtfsDownloadParseBenchmark.testTransferThenParse                      200    ss        49,912           s/op
GtfsDownloadParseBenchmark.testTransferThenParse                      500    ss        47,637           s/op

So numbers seem to indicate that the performance is better for low to medium bandwidth, the two approaches converges (as expected) on very high bandwidth (> gigabit) connections.

The big question is how users actually get hold of their GTFS feeds (mounted via filesystem or URL), and if via URL, the actual bandwidth.

• Linked all relevant issues

[leonardehrenfried]

Thanks for this.

I don't have any hard numbers how OBA is used (and I don't think we will get them) but here are mine: at home I have a 250MB connection and on my servers I have gigabit. So for me personally it would not make a big difference.

I think if you want to optimize this, then you could do so in the OTP code itself, which has its own HTTP layer. There you can get real world data from users and we have a well established review and feedback process.

Lastly, I would not be against creating a separate module for just this functionality. The separate module would be there to keep the dependencies of the main code low.