Skip to content

Support dynamic browserslist-data #38

Description

@quininer

I think we've basically implement optimal size use searchable static data. However, browserslist is still the largest data in rspack's rodata due to its scale. Even with pooled str, each string and list still takes up at least 4 bytes.

0x00000000017f105c      83752 r browserslist_data::caniuse::features::FEATURES_STAT_VERSION_INDEX
0x0000000001846010     207776 r browserslist_data::caniuse::region::REGIONS_USAGES
0x0000000001813470     207776 r browserslist_data::caniuse::region::REGIONS_VERSIONS
0x00000000016a5060    1359868 r browserslist_data::caniuse::features::FEATURES_STAT_VERSION_STORE

The caniuse data currently use about 2MB of space, but it can be compressed to less than 200KB by zstd. We have some tricks to optimize it (like inline id, seekable huffman coding). but it is too complicated and hard to compare with zstd.

Using the zstd or other compression algorithm in -data crate is not ideal, as it means that the decompressed data needs to occupy dirty memory and cannot be evicted.
I think a better approach is to use traits to abstract -data crate and allow users to use dynamic data. this is similar to the provider of icu4x.

It will look something like this

use browserslist::{Opts, resolve_with_provider};
use browserslist_data_providers::DataProvider;

let buf = fs::read_to_end("data.zstd")?;
let provider = DataProvider::new(&buf)?;
let distribs = resolve_with_provider(["ie <= 6"], &Opts::default(), &provider).unwrap();

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions