support GZIP codec natively via DecompressionStream - #167
Conversation
Decompression becomes async through decompressPage/readPage/readColumn; custom compressors may now also return a Promise. Browser caveat: DecompressionStream only accepts single-member gzip streams per the WHATWG spec, so rare multi-member pages still need a custom decompressor.
|
Interesting, thanks for implementing the PR, Tom. I guess that changing the methods to async will not be accepted as such, though. Let's wait for @platypii review. |
|
yeah consider this ideation over the possabilities |
|
Yea this is interesting, but I am nervous that switching things to async this deep through the pipeline will have performance implications. Will take a look. Thanks @tomlarkworthy! |
|
The async overheads are real but quite small relative to actually executing high CPU tasks like decompression. If you are actually using gzip decompression there is a significant throughput gain because the native one is off the Javascript thread, so you get to unblock the main thread for reading pages, and the decompression can occur in parallel. So if you actually use gzip this is actually better I think, the page does not block. I think using native gzip decompression unblocks high performance browser parquet handling in the browser without the drawbacks of shipping and executing decompression codecs on the main thread. So you can have a small bundle size AND still efficiently do the data transfer AND its actually faster throughput AND it leaves the page's Javascript free to do other things. |
3 participants
Browser have gzip natively, so I was wondering if we can actually use that for native gzip support on parquet. Yes we can! But it does require changing some internal APIs for async. WDYT?
This lets the browser transfer large parquet files effeciently without the bloat of implementing a decompression codec.