Skip to content

Commit d6e9a01

Browse files
JulianCloudNTHfacebook-github-bot
authored andcommitted
CI: cache SwiftShader prebuilt to skip per-run from-source build
Summary: Try a pinned-rev SwiftShader prebuilt from gha-artifacts before building from source (mirrors the torch-wheel download-or-build in .ci/scripts/utils.sh); on a cache miss, fall back to the existing from-source build. The key embeds SWIFTSHADER_REV, so a rev bump auto-invalidates (re-seed S3 once per bump). Follow-up to the Dawn CI diff, addressing review feedback to cache the build. Differential Revision: D108185797
1 parent f0dff03 commit d6e9a01

1 file changed

Lines changed: 26 additions & 12 deletions

File tree

.ci/scripts/setup-webgpu-linux-deps.sh

Lines changed: 26 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,9 @@
1010
# * Dawn : Google's official nightly prebuilt, downloaded directly from
1111
# github.com/google/dawn/releases (pinned tag+rev+sha256) -- the same
1212
# "fetch a pinned upstream prebuilt" pattern used for other CI deps.
13-
# * SwiftShader : built from source at a pinned rev compatible with the Dawn
14-
# above (the ossci prebuilt is from 2020, too old for current Dawn). No S3.
13+
# * SwiftShader : a cached prebuilt (gha-artifacts, keyed by rev) when present,
14+
# else built from source at a pinned rev compatible with the Dawn above
15+
# (the ossci prebuilt is from 2020, too old for current Dawn).
1516
# Dawn (Chrome's WebGPU impl; its WGSL compiler Tint is the spec reference) on
1617
# SwiftShader gives a headless, deterministic, spec-faithful CLI backend.
1718
#
@@ -75,17 +76,30 @@ fi
7576
# current Dawn; build a matching modern SwiftShader instead. Self-contained
7677
# cmake build (vendored LLVM); the ICD lands under build/<OS>/.
7778
if [[ ! -d "${_ss_dir}/build" ]]; then
78-
if [[ ! -d "${_ss_dir}/.git" ]]; then
79-
git clone https://github.com/google/swiftshader "${_ss_dir}"
79+
# Try a cached prebuilt (keyed by rev) before the expensive from-source build;
80+
# mirrors the gha-artifacts download-or-build pattern in .ci/scripts/utils.sh.
81+
_ss_key="swiftshader-${SWIFTSHADER_REV}-ubuntu-latest"
82+
_ss_cache_url="https://gha-artifacts.s3.us-east-1.amazonaws.com/cached_artifacts/pytorch/executorch/webgpu/${_ss_key}.tar.gz"
83+
if curl --silent --show-error --location --fail --retry 3 --retry-all-errors \
84+
--output "/tmp/${_ss_key}.tar.gz" "${_ss_cache_url}"; then
85+
# Cache hit: restore the prebuilt build/ tree (ICD + libs).
86+
mkdir -p "${_ss_dir}/build"
87+
tar -C "${_ss_dir}/build" -xzf "/tmp/${_ss_key}.tar.gz"
88+
else
89+
# Cache miss: build from source. After a rev bump, seed the key in S3
90+
# (tar -C build/ . -> the cache URL above) to skip this next run.
91+
if [[ ! -d "${_ss_dir}/.git" ]]; then
92+
git clone https://github.com/google/swiftshader "${_ss_dir}"
93+
fi
94+
git -C "${_ss_dir}" checkout "${SWIFTSHADER_REV}"
95+
# vk_swiftshader's deps are vendored in-tree; tolerate unreachable
96+
# disabled-feature submodules (angle, test-only) failing to fetch.
97+
git -C "${_ss_dir}" submodule update --init --recursive || true
98+
cmake -S "${_ss_dir}" -B "${_ss_dir}/build" -DCMAKE_BUILD_TYPE=Release \
99+
-DSWIFTSHADER_BUILD_TESTS=OFF -DSWIFTSHADER_BUILD_PVR=OFF \
100+
-DSWIFTSHADER_BUILD_BENCHMARKS=OFF
101+
cmake --build "${_ss_dir}/build" --parallel "$(nproc)" --target vk_swiftshader
80102
fi
81-
git -C "${_ss_dir}" checkout "${SWIFTSHADER_REV}"
82-
# vk_swiftshader's deps are vendored in-tree; tolerate unreachable
83-
# disabled-feature submodules (angle, test-only) failing to fetch.
84-
git -C "${_ss_dir}" submodule update --init --recursive || true
85-
cmake -S "${_ss_dir}" -B "${_ss_dir}/build" -DCMAKE_BUILD_TYPE=Release \
86-
-DSWIFTSHADER_BUILD_TESTS=OFF -DSWIFTSHADER_BUILD_PVR=OFF \
87-
-DSWIFTSHADER_BUILD_BENCHMARKS=OFF
88-
cmake --build "${_ss_dir}/build" --parallel "$(nproc)" --target vk_swiftshader
89103
fi
90104
_ss_icd="$(find "${_ss_dir}/build" -name vk_swiftshader_icd.json 2>/dev/null | head -1)"
91105
[[ -n "${_ss_icd}" ]] || { echo "ERROR: SwiftShader ICD not found after build" >&2; exit 1; }

0 commit comments

Comments
 (0)