System Info
System:
OS: Linux 6.8 Ubuntu 22.04.5 LTS 22.04.5 LTS (Jammy Jellyfish)
CPU: (8) x64 Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
Memory: 40.61 GB / 61.80 GB
Container: Yes
Shell: 5.1.16 - /usr/bin/bash
Binaries:
Node: 22.22.0 - ~/.nvm/versions/node/v22.22.0/bin/node
Yarn: 4.12.0 - ~/src/server/node_modules/.bin/yarn
npm: 10.9.4 - ~/.nvm/versions/node/v22.22.0/bin/npm
Watchman: 20240414.112832.0 - /usr/local/bin/watchman
npmPackages:
@rspack/cli: ^2.0.3 => 2.0.0
@rspack/core: ^2.0.3 => 2.0.0
@rspack/plugin-react-refresh: ^2.0.0 => 2.0.0
Details
I'm attempting to optimize a large rspack build, and earlier in the process I was taking rspack's numbers (e.g. "Rspack 2.0.3 compiled successfully in 13.18 s") at face value. However, in profiling the larger process, where I'm invoking rspack within bazel, I found that bazel believed rspack took several seconds more time. I have since verified that bazel is correct by adding a time around the rspack invocation: rspack seems to take several seconds longer than it reports. Having dove into profiling with node's --cpu-prof, I've determined the initial chunk of this time to be loading modules & evaluating the config, and optimized that down to 300ms so far; when the cache is off I have about another 600ms unaccounted for, but with cache: { type: 'persistent'} in my config that skyrockets to about 3.4s total difference or about 3.1s unaccounted for - so about 2.5s due to the cache being enabled.
To be clear: the cache is still a speedup in scenarios where it probably should be, but the printed stats oversell it's benefit - my guess is by not accounting for time to write the cache back to disk when the build is done.
results from timing my build (run via @rspack/cli)
real 0m16.644s
user 0m41.412s
sys 0m14.440s
logs from rspack itself
LOG from rspack.persistentCache
<i> persistent cache enabled
<i> build dependencies are valid (9 tracked)
<i> meta persistent cache recovery succeeded
<i> snapshot restored with detected changed dependencies:
<i> modified paths (1):
<i> - /mnt/cache/bazel/output/7f9e2c199699ecc9923b138ecc9eb7a1/execroot/__main__/bazel-out/k8-fastbuild/bin/js/core/uri.esnext.js
<i> make persistent cache recovery succeeded
<i> minimize persistent cache recovery succeeded
Rspack 2.0.3 compiled successfully in 13.18 s
note the significant difference in real vs rspack's claimed time.
I tried adding stats.timings: true to my config but that didn't result in any more logging output.
My ask here is twofold:
a) add some logging that shows where rspack's time is going when reading/writing the cache
b) optimize cache reads & writes for large builds. Perhaps writes could be done in parallel with the end of the build? or perhaps a better file format would reduce serialization/deserialization overhead?
Reproduce link
No response
Reproduce Steps
time npm run rspack on a repo with a larger rspack build, with persistent cache enabled. (I can't share my codebase with you so I don't have an easy repro)
System Info
System:
OS: Linux 6.8 Ubuntu 22.04.5 LTS 22.04.5 LTS (Jammy Jellyfish)
CPU: (8) x64 Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
Memory: 40.61 GB / 61.80 GB
Container: Yes
Shell: 5.1.16 - /usr/bin/bash
Binaries:
Node: 22.22.0 - ~/.nvm/versions/node/v22.22.0/bin/node
Yarn: 4.12.0 - ~/src/server/node_modules/.bin/yarn
npm: 10.9.4 - ~/.nvm/versions/node/v22.22.0/bin/npm
Watchman: 20240414.112832.0 - /usr/local/bin/watchman
npmPackages:
@rspack/cli: ^2.0.3 => 2.0.0
@rspack/core: ^2.0.3 => 2.0.0
@rspack/plugin-react-refresh: ^2.0.0 => 2.0.0
Details
I'm attempting to optimize a large rspack build, and earlier in the process I was taking rspack's numbers (e.g. "Rspack 2.0.3 compiled successfully in 13.18 s") at face value. However, in profiling the larger process, where I'm invoking rspack within bazel, I found that bazel believed rspack took several seconds more time. I have since verified that bazel is correct by adding a
timearound the rspack invocation: rspack seems to take several seconds longer than it reports. Having dove into profiling with node's--cpu-prof, I've determined the initial chunk of this time to be loading modules & evaluating the config, and optimized that down to 300ms so far; when the cache is off I have about another 600ms unaccounted for, but withcache: { type: 'persistent'}in my config that skyrockets to about 3.4s total difference or about 3.1s unaccounted for - so about 2.5s due to the cache being enabled.To be clear: the cache is still a speedup in scenarios where it probably should be, but the printed stats oversell it's benefit - my guess is by not accounting for time to write the cache back to disk when the build is done.
results from timing my build (run via @rspack/cli)
logs from rspack itself
note the significant difference in real vs rspack's claimed time.
I tried adding stats.timings: true to my config but that didn't result in any more logging output.
My ask here is twofold:
a) add some logging that shows where rspack's time is going when reading/writing the cache
b) optimize cache reads & writes for large builds. Perhaps writes could be done in parallel with the end of the build? or perhaps a better file format would reduce serialization/deserialization overhead?
Reproduce link
No response
Reproduce Steps
time npm run rspackon a repo with a larger rspack build, with persistent cache enabled. (I can't share my codebase with you so I don't have an easy repro)