[envpool] Fix native seed randomization and extend deterministic checks - #434
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0b9326d704
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ) | ||
|
|
||
| py_test( | ||
| name = "seed_test", |
There was a problem hiding this comment.
Keep known-failing checks out of the default test graph
The checked Linux workflow runs BAZEL_TEST_TARGETS="//...", and Windows uses the Makefile's identical default, so this target and //envpool/mujoco:myosuite_seed_test run in every normal CI build. The commit's validation reports 147 assertion failures and two VizDoom segfaults, while neither target is tagged out or uses passing expectations; consequently this change makes full-platform CI unconditionally red. Fix the reported implementations before enabling these targets in the default graph, or keep the diagnostic probes outside it until they pass.
AGENTS.md reference: AGENTS.md:L17-L17
Useful? React with 👍 / 👎.
Description
A reset that ignores its seed still passes a same-seed equality test. Several native environments had that behavior, and existing deterministic tests did not check the default reset/rollout paths across their whole families. This PR extends the existing family tests to require meaningful state variation where the official task is stochastic, and fixes the implementations those checks exposed.
terminatedandtruncatedwhen a natural ending coincides with the time limit.The tests stay in their existing environment families: there is no root seed-test suite or separate MyoSuite seed target. Shared behavioral assertions live in
envpool/python/seed_test_utils.py; that helper is also included in the wheel because the installed-wheel tests use it. Environment simulation remains native C++, with official Python packages used only by test/oracle and build-time metadata tooling.The assertions compare physical state or real multi-step trajectories, excluding seed/RNG metadata and counters. They check equal-seed reproducibility, different seeds, successive resets, and vector slots where appropriate. Deliberately fixed tasks remain fixed. No new skips, xfails, recorded-output fixtures, or relaxed alignment tolerances were introduced to pass these checks.
Motivation and Context
Closes #432. Both reported tasks reproduced on main
46b1e2f769320b2e1d579709f5979d32dc48e9ca. The old MyoSuite same-seed test passed all 398 configurations; adding state-variation assertions to that same deterministic method exposed 116 failures. The test changes before the native fixes are preserved in the commit history.The original audit covered 1,119 distinct configurations across 19 families and found 147 seed/reset failures plus two ViZDoom initialization crashes. Highway previously checked seed sensitivity for only two configurations. Jumanji lacked a family-wide default-path check, and some existing oracle tests inject future recorded outputs. Reset-time oracle synchronization can also overwrite the randomized state being checked. The new seed assertions use neither synchronization nor recorded outputs.
The corrected registry has the same 1,704 IDs and 1,120 distinct configurations: fixing the multi-agent termination settings makes a previously identical Highway configuration distinct.
This addresses the existing report in #432; no additional issue was opened.
Types of changes
Implemented Tasks
Validation
Final source commit:
094e53ca2feeefd055625ef93a5c2a3d4b985949. Native platform validation follows the same toolchains and release procedure as the dependency-upgrade validation. Linux full lint, including clang-tidy and the aggregatemake linttarget, passed.Each full test invocation selects
//..., disables test-result caching, and checks that every configured test target completed successfully. This includes the existing oracle/render targets and all MyoSuite task shards. The affected families cover 398 MyoSuite, 31 Highway, 26 Jumanji, and 21 ViZDoom distinct configurations. ViZDoom's existing 29 test cases run in four shards.The full-suite commands, omitting machine-specific cache and toolchain paths, are:
Each release check builds the wheel with the corresponding Python version, installs it, runs
pip check, and executesmake release-test. The installed-wheel smoke check rejects source-tree imports. Linux wheels retain themanylinux_2_28baseline and passauditwheel show. Linux and Windows runtime checks run without the Qt SDK search paths; Linux also hides the SDK directory. macOS keeps its existing system Qt requirement.Existing Jumanji oracle/render tests pass, but some legacy oracle cases replay recorded future outputs; they are not independent proof of native step semantics. The new seed checks and affected rule tests exercise real native paths. MyoSuite oracle/render coverage is the existing coverage plus the two reported Finger rollout cases, not a claim of complete upstream alignment for all 398 tasks.
Hosted workflow runs on the previously published revision were blocked by GitHub's billing/spending limit before job steps executed. The results above come from native runners; they do not claim hosted workflow passes.
Checklist
make format(required)make lint(required)make bazel-testpass. (required)New Environment Checklist
Not applicable: this PR adds no environment family or upstream task family.
envpool/make_test.py, release packaging, docs, and README support lists are updated.