I think we've basically implement optimal size use searchable static data. However, browserslist is still the largest data in rspack's rodata due to its scale. Even with pooled str, each string and list still takes up at least 4 bytes.
0x00000000017f105c 83752 r browserslist_data::caniuse::features::FEATURES_STAT_VERSION_INDEX
0x0000000001846010 207776 r browserslist_data::caniuse::region::REGIONS_USAGES
0x0000000001813470 207776 r browserslist_data::caniuse::region::REGIONS_VERSIONS
0x00000000016a5060 1359868 r browserslist_data::caniuse::features::FEATURES_STAT_VERSION_STORE
The caniuse data currently use about 2MB of space, but it can be compressed to less than 200KB by zstd. We have some tricks to optimize it (like inline id, seekable huffman coding). but it is too complicated and hard to compare with zstd.
Using the zstd or other compression algorithm in -data crate is not ideal, as it means that the decompressed data needs to occupy dirty memory and cannot be evicted.
I think a better approach is to use traits to abstract -data crate and allow users to use dynamic data. this is similar to the provider of icu4x.
It will look something like this
use browserslist::{Opts, resolve_with_provider};
use browserslist_data_providers::DataProvider;
let buf = fs::read_to_end("data.zstd")?;
let provider = DataProvider::new(&buf)?;
let distribs = resolve_with_provider(["ie <= 6"], &Opts::default(), &provider).unwrap();
I think we've basically implement optimal size use searchable static data. However,
browserslistis still the largest data in rspack's rodata due to its scale. Even with pooled str, each string and list still takes up at least 4 bytes.The caniuse data currently use about 2MB of space, but it can be compressed to less than 200KB by zstd. We have some tricks to optimize it (like inline id, seekable huffman coding). but it is too complicated and hard to compare with zstd.
Using the zstd or other compression algorithm in
-datacrate is not ideal, as it means that the decompressed data needs to occupy dirty memory and cannot be evicted.I think a better approach is to use traits to abstract
-datacrate and allow users to use dynamic data. this is similar to the provider of icu4x.It will look something like this