Commit 48affea
authored
[envpool] Upgrade JAX, physics assets, Qt, and LLVM tooling (#430)
## Description
Upgrade the compatible JAX, physics-asset, oracle, Qt, and LLVM
dependencies on top of main (`27a47a1`). Keep source builds and
installed release wheels on the same model data by requiring the newly
published `envpool-assets*` 0.4.x line.
| Dependency | Before | After |
| --- | --- | --- |
| JAX / jaxlib | 0.9.2 | 0.11.1 |
| OpenXLA FFI headers | `187a5eb` | `dcf304b`, matching JAX 0.11.1 |
| trimesh | 4.12.2 | 5.0.0 |
| qhull | `62ccc56` | `d1c2fc0`, matching the pinned MuJoCo source |
| Bazel protobuf | 36.0 | 36.0.bcr.1 (BCR packaging revision, not a new
upstream runtime) |
| object_sim | 0.1.0 | 0.1.1 |
| YCB_sim | `46edd9c` | `57546b8` |
| mujoco_menagerie | `1b86ece` | `da76818` |
| Jumanji oracle dependencies | JAX 0.4.35, NumPy 1.26.4, SciPy 1.17.1 |
JAX 0.11.1, NumPy 2.5.2, SciPy 1.18.0; refresh the transitive lock |
| MyoSuite oracle dependencies | NumPy 1.26.4, Gymnasium 1.2.3, SciPy
1.17.1, protobuf 7.34.1 | NumPy 2.5.2, Gymnasium 1.3.0, SciPy 1.18.0,
protobuf 7.35.1; refresh the transitive lock |
| clang-format / clang-tidy | LLVM 20 configuration; tidy wrapper could
select 18 | Pinned 22.1.8 tools and explicit tidy executable |
| Qt, macOS / Windows | Qt 5 | 6.11.1 |
| Qt, Ubuntu source CI | Qt 5 | Distro Qt 6.4.2 |
| Qt, Linux release wheels | External Qt 5.15.3 | Bundled Qt 6.11.1,
built inside manylinux_2_28 |
| Three envpool-assets packages | 0.3.x | 0.4.x |
The three asset wheels were [built, checked, smoke-tested, and
published](https://github.com/Trinkle23897/envpool-assets/actions/runs/33047285626)
from commit `4d8a4eae4874ee0109ed045b4e98923a0ee97606`. Downloading them
from PyPI and comparing every uncompressed entry confirmed all 3,581
files match the locally tested candidates. Each wheel remains below the
100 MB limit.
Compatibility fixes and cleanup:
- Patch the test-only Playground oracle for the current JAX `clip` API
and the test-only MyoSuite oracle for NumPy 2 scalar conversion. Extend
the existing scalar CPU-dispatch mask for NumPy 2.5's x86-64 baseline
names; do not widen alignment tolerances.
- Keep CVRP capacity in upstream integer demand units and normalize only
observations. Respect the configured capacity for default demands,
including capacities 3, 20, and 30.
- Update the Jumanji render cache only once per received transition,
fixing double-counted synchronous scores. Preserve fractional
SearchAndRescue target counts and expose per-searcher rewards alongside
EnvPool's scalar sum.
- Support Qt 5/6 headers, frameworks, MSVC import libraries, DLL
packaging, and Linux missing-runtime errors. Build a minimal shared Qt
6.11.1 raster runtime inside the AlmaLinux 8 / manylinux_2_28 release
images; bundle Core/Gui, licenses, and source/build information without
raising the glibc baseline. Cache the SDK per architecture and validate
the wheel after moving the SDK out of its install path. Use
version-specific Procgen pixel baselines, retaining the same seed,
rollout, reward assertions, and tolerances; a Qt 5 control reproduces
the previous baseline.
- Pin the upstream aqt fix needed to install Qt 6.11 on Windows. Include
Qt license texts, source/checksum information, the SDK SBOM, and build
configuration in Windows wheels, reusing the existing Qt source pin and
wheel-notice collector. Restore MSYS argument conversion only in the
notice-collection subprocess so native Windows curl can open Bash paths
without changing Bazel's environment.
- Fix the clang-tidy wrapper variable collision and address LLVM 22
findings without changing runtime algorithms.
- Remove the unused Qt BUILD file and stale Acme example requirements.
Acme is an optional user installation; there is no Bazel Acme
dependency.
Versions deliberately retained:
| Dependency | Retained version / reason |
| --- | --- |
| MuJoCo / MJX, dm-control | 3.11.0 / 3.11.0, 1.0.44. Newer versions
were unavailable in the configured package index during validation; keep
the source and Python oracle versions coherent. |
| SWIG, main SciPy, main Python protobuf | 4.4.1, 1.18.0, 7.35.1. No
usable newer versions in that configured index during validation. Oracle
SciPy/protobuf do advance as listed above. |
| myo_sim | `33f3ded`, the MyoSuite 2.12.2 gitlink. Newer myo_sim
removes legacy XML fragments and requires a separate MjSpec migration. |
| Jumanji / MyoSuite oracle source | Keep Jumanji 1.1.2 and MyoSuite
2.12.2; upgrade their dependency stacks. |
| SDL | Keep SDL2-compat 2.32.70, SDL3 3.4.14, and SDL2_ttf 2.24.0.
Moving SDL_ttf to 3.x requires a separate GFootball SDL3 port. |
## Motivation and Context
The dependency updates expose real NumPy/JAX, rendering, and packaging
incompatibilities. This PR fixes those paths and keeps release-wheel
assets aligned with the Bazel source pins. Asset version isolation
prevents older EnvPool releases, which require assets below 0.4, from
automatically receiving the changed physics models.
macOS wheels now require **macOS 13+** because Qt 6.11 requires it. As
in the existing packaging design, they use external Homebrew Qt
frameworks rather than bundling them: run `brew install qtbase` before
importing EnvPool. Windows and Linux manylinux wheels bundle Qt. Linux
retains the glibc 2.28 baseline; the source-built Qt runtime does not
depend on ICU or window-system libraries. Native Linux ARM64 source/test
builds still use **Clang 20** to match the official MuJoCo wheel;
manylinux release builds use the image's GCC toolchain; LLVM 22 here is
the lint/format tooling upgrade, not an ARM compiler change.
The docs explicitly describe a pre-existing Jumanji limitation: some
oracle fixtures replay short stored state/reward trajectories. Passing
those checks establishes replay/render compatibility, not
arbitrary-action semantic alignment. This PR does not claim to finish
that separate implementation work.
- [ ] I have raised an issue to propose this change
([required](https://envpool.readthedocs.io/en/latest/pages/contributing.html)
for new features and bug fixes). No separate issue; this is the
requested dependency-maintenance batch.
## Types of changes
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds core functionality)
- [ ] New environment (non-breaking change which adds 3rd-party
environment)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation (update in the documentation)
- [x] Example (update in the folder of example)
## Implemented Tasks
- [x] Refresh compatible pins and publish the matching asset wheels.
- [x] Fix compatibility and correctness issues; extend existing behavior
tests without adding skips or widening tolerances.
- [x] Complete the original four-platform native test matrix.
- [x] Rebuild and validate all 12 platform/Python release combinations
after the Qt 6 manylinux change.
- [x] Include Qt notices in Windows wheels and repeat all three Windows
release combinations after that fix.
- [ ] Get the GitHub Actions checks green; currently blocked by the
account billing lock before any job steps execute.
Validation:
The native environment/oracle suites were run uncached at
`4d8a4eae4874ee0109ed045b4e98923a0ee97606`. The Qt 6 packaging follow-up
at `5c19371368f3a6f680e24c01bb4ac014f2f68833` does not change
environment implementations or tolerances. All 12 release wheels were
rebuilt and their expanded release tests passed at that follow-up.
Commit `365b4aa72dc414f7a5a6194b3a3b0bbaadf0fec2` records the Qt
repository digest generated during those builds. The Windows-notice
follow-up at `692b82253200713cedcaa19504a9f6fdff0123bb` then rebuilt and
retested all three Windows releases. Their previous uncompressed payload
entries, including all 17 native modules and both Qt DLLs, are
byte-for-byte unchanged except for the updated RECORD; the only added
payload is 45 license/notice/build-information files per wheel. The
macOS and Linux release results remain from `5c19371`; the full native
suites remain from `4d8a4ea`. Final-head `bazel mod deps
--lockfile_mode=error` passed.
| Platform | Full native test suite at `4d8a4ea` | Python 3.12 / 3.13 /
3.14 release rebuild, install, pip check, and release test |
| --- | --- | --- |
| macOS arm64, Qt 6.11.1 | 98/98 passed, uncached | All three passed at
`5c19371` |
| Windows x64, Qt 6.11.1 | 98/98 passed, uncached | All three passed
again at `692b822`, including Qt notices; SDK search paths removed and
loaded DLL origins checked |
| Linux x64 devbox, source Qt 6.4.2 / release Qt 6.11.1 | 98/98 passed,
uncached; coverage report generated | All three native manylinux_2_28
runs passed at `5c19371`, with the Qt SDK moved away |
| Native Linux arm64 VM, source Qt 6.4.2 / release Qt 6.11.1 | 98/98
passed, uncached | All three native manylinux_2_28 runs passed at
`5c19371`, with the Qt SDK moved away |
Qt release verification:
- Command: `make pypi-wheel`, install the resulting wheel, `python -m
pip check`, and `make release-test`, under each Python version.
- Linux explicitly uses `make PROCGEN_QT_RUNTIME=bundled release-test`
after moving `/opt/envpool-qt6` out of its install path. All six runs
confirmed Qt 6.11.1 Core/Gui were loaded from the installed wheel's
`envpool.libs`, not a system SDK. No Qt 5 or ICU library is bundled.
- The existing Procgen 1,001-step pixel/reward baseline, 1,000-step
channel-order comparison, all-task determinism checks, and all-task
multi-step render suite now run against installed release wheels. No
baseline or tolerance changed in this follow-up. The optional upstream
Python Procgen oracle is not a release dependency; these checks do not
claim new upstream-oracle coverage.
- Linux x64 wheels are 59.85 MB; ARM64 wheels are 59.14-59.15 MB, below
the unchanged 100 MB cap. All six passed CRC, SHA256/RECORD,
platform-tag, 17-native-module, and Qt library/license payload checks.
- Windows release tests clear Qt SDK variables/search paths and use
`GetModuleFileNameW` to check that Qt6 Core/Gui DLLs come from the
installed wheel. The latest remote driver completed all eleven
license-preparation/build/install/test/final-check stages with exit code
0. The three wheels are 53.35-55.48 MB and passed CRC, remote SHA256,
complete RECORD, and Qt license/SBOM payload checks.
Lint and build validation:
- The CI's `make ruff py-format cpplint clang-format buildifier
addlicense mypy docstyle spelling` passed locally for the Qt changes;
Linux also passed the full `make lint addlicense`, including clang-tidy
(59 targets, successful BEP). `actionlint`, shell syntax checks, and
focused Python type checks passed. The Windows-notice follow-up also
passed `actionlint`, Bash syntax checks, and macOS `make docstyle
spelling addlicense`.
- The prior full-suite commands were `make BAZEL_TEST_TARGETS=//...
bazel-test` on macOS, Windows, and native ARM64; Linux x64 used `make
BAZEL_TEST_TARGETS=//... bazel-coverage`, `scripts/coverage_summary.py`,
and `genhtml`. Tests used `--nocache_test_results`. The Linux report
covers 534 files, with 90.78% line coverage and 55.21% branch coverage.
- Native XLA FFI is Linux-only: the existing MiniGrid JIT step and Atari
100-step JIT/correctness comparisons passed on x64 and ARM64. macOS and
Windows test the existing unsupported-XLA error contract; their passing
suites do not imply native XLA support.
- The separate `make bazel-build` development-wheel builds passed on
macOS and Linux x64 in the original matrix. macOS wheels retain their
documented external Homebrew Qt requirement.
[Hosted checks](https://github.com/sail-sg/envpool/pull/430/checks)
remain separate from these local results. The final checks on `692b822`
confirm the account billing lock still prevents all ten runnable jobs
from starting any steps; the earlier lint retry also failed before
execution. Hosted failures are not claimed as passes; the account must
be unlocked before those jobs can be rerun.
The added CVRP capacity cases were verified red/green: capacities 3 and
30 fail against the pre-fix wheel and pass after the fix; capacity 20
remains green. Existing 2048 and SearchAndRescue tests now cover the
corrected render-cache and reward behavior. No static
configuration/source-text tests were added.
## Checklist
- [x] I have read the
[CONTRIBUTION](https://envpool.readthedocs.io/en/latest/pages/contributing.html)
guide (**required**)
- [x] My change requires a change to the documentation.
- [x] I have updated the tests accordingly (*required for a bug fix or a
new feature*).
- [x] I have updated the documentation accordingly.
- [x] I have reformatted the code using `make format` (**required**)
- [x] I have checked the code using `make lint` (**required**)
- [x] I have ensured `make bazel-test` pass. (**required**)
## New Environment Checklist
Not applicable: this PR does not add an environment family or upstream
task family.
- [ ] Runtime logic is native C++ and does not bridge to the official
Python environment.
- [ ] All intended upstream task IDs/scenarios are registered,
documented, and covered by tests.
- [ ] The upstream oracle/version is pinned, and tests check EnvPool
registration/configs against it.
- [ ] Determinism tests cover reset plus multi-step rollouts for every
registered ID, including render frames when rendering is supported.
- [ ] Oracle alignment tests compare step-level observations, rewards,
done/truncation, info, and renders after at most one reset-time state
sync.
- [ ] Render tests cover reset, multi-step, batched render/env-id
selection, and docs include EnvPool-vs-official images when an official
renderer exists.
- [ ] `envpool/make_test.py`, release packaging, docs, and README
support lists are updated.1 parent 27a47a1 commit 48affea
61 files changed
Lines changed: 1041 additions & 1513 deletions
File tree
- .github/workflows
- docker
- docs
- content
- env
- envpool
- box2d
- core
- jumanji
- mujoco
- metaworld
- myosuite
- procgen
- examples/acme_examples
- scripts
- third_party
- clang_tidy
- jumanji
- mujoco_playground
- myosuite
- pip_requirements
- qt
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
27 | | - | |
28 | | - | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
32 | 33 | | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
33 | 37 | | |
34 | 38 | | |
35 | 39 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
| 16 | + | |
15 | 17 | | |
16 | 18 | | |
17 | 19 | | |
| |||
24 | 26 | | |
25 | 27 | | |
26 | 28 | | |
| 29 | + | |
| 30 | + | |
27 | 31 | | |
28 | 32 | | |
| 33 | + | |
29 | 34 | | |
30 | 35 | | |
31 | 36 | | |
32 | 37 | | |
33 | 38 | | |
| 39 | + | |
34 | 40 | | |
35 | 41 | | |
| 42 | + | |
36 | 43 | | |
37 | 44 | | |
| 45 | + | |
| 46 | + | |
38 | 47 | | |
39 | 48 | | |
40 | 49 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
44 | | - | |
| 44 | + | |
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
39 | | - | |
| 39 | + | |
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
56 | 56 | | |
57 | 57 | | |
58 | 58 | | |
59 | | - | |
| 59 | + | |
60 | 60 | | |
61 | | - | |
| 61 | + | |
62 | 62 | | |
63 | 63 | | |
64 | 64 | | |
| |||
84 | 84 | | |
85 | 85 | | |
86 | 86 | | |
| 87 | + | |
| 88 | + | |
87 | 89 | | |
88 | 90 | | |
89 | 91 | | |
| |||
109 | 111 | | |
110 | 112 | | |
111 | 113 | | |
112 | | - | |
| 114 | + | |
113 | 115 | | |
114 | | - | |
115 | 116 | | |
116 | 117 | | |
117 | 118 | | |
| |||
126 | 127 | | |
127 | 128 | | |
128 | 129 | | |
129 | | - | |
| 130 | + | |
130 | 131 | | |
131 | 132 | | |
132 | | - | |
| 133 | + | |
133 | 134 | | |
134 | 135 | | |
135 | 136 | | |
136 | 137 | | |
137 | 138 | | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
138 | 149 | | |
139 | 150 | | |
140 | 151 | | |
| |||
144 | 155 | | |
145 | 156 | | |
146 | 157 | | |
147 | | - | |
148 | | - | |
149 | | - | |
150 | | - | |
| 158 | + | |
| 159 | + | |
151 | 160 | | |
152 | | - | |
153 | | - | |
154 | | - | |
155 | | - | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
156 | 164 | | |
157 | 165 | | |
158 | 166 | | |
| |||
219 | 227 | | |
220 | 228 | | |
221 | 229 | | |
222 | | - | |
| 230 | + | |
223 | 231 | | |
224 | 232 | | |
225 | | - | |
226 | | - | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
227 | 238 | | |
228 | 239 | | |
229 | 240 | | |
230 | 241 | | |
231 | 242 | | |
232 | 243 | | |
233 | 244 | | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
234 | 251 | | |
235 | 252 | | |
236 | 253 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
49 | | - | |
| 49 | + | |
50 | 50 | | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | 51 | | |
55 | 52 | | |
56 | 53 | | |
| |||
167 | 164 | | |
168 | 165 | | |
169 | 166 | | |
170 | | - | |
| 167 | + | |
171 | 168 | | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | 169 | | |
176 | 170 | | |
177 | 171 | | |
| |||
243 | 237 | | |
244 | 238 | | |
245 | 239 | | |
246 | | - | |
| 240 | + | |
247 | 241 | | |
248 | 242 | | |
249 | 243 | | |
250 | 244 | | |
251 | 245 | | |
252 | 246 | | |
253 | 247 | | |
254 | | - | |
| 248 | + | |
255 | 249 | | |
256 | 250 | | |
257 | 251 | | |
| |||
348 | 342 | | |
349 | 343 | | |
350 | 344 | | |
351 | | - | |
| 345 | + | |
352 | 346 | | |
353 | 347 | | |
354 | | - | |
355 | | - | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
356 | 353 | | |
357 | 354 | | |
358 | 355 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| |||
0 commit comments