select: keep the solver's Newton carry varying under shard_map - #18
Draft
E-Rum wants to merge 1 commit into
Draft
Conversation
E-Rum
force-pushed
the
fix/solver-adaptive-cutoffs-shard-map
branch
from
July 28, 2026 16:26
1a73833 to
f72f4cc
Compare
The solver adaptive-cutoff path seeded its fori_loop bracket from
constants (jnp.zeros / jnp.full), while the loop body's outputs derive
from r_ij. Under shard_map r_ij varies over the data-parallel mesh axis,
so carry-in and carry-out disagreed on manual axis types and scan's
carry type check rejected the loop at trace time:
TypeError: scan body function carry input and carry output must have
equal types [...] float32[1024] vs float32[1024]{V:dp}
Any data-parallel run using adaptive_cutoff_method="solver" failed to
trace; single-device runs were unaffected, since callers only apply
shard_map when there is more than one device.
pcast the bracket to varying, reading the axes off the data so the fix
is agnostic to the caller's mesh axis names and a no-op outside
shard_map. Runtime behaviour is unchanged: each shard already solves on
its own atoms, and the annotation carries no computation.
Adds tests/test_shard_map.py, which runs both adaptive-cutoff methods
under shard_map on forced CPU devices and checks values and gradients
against the single-device result -- multi-device coverage the suite
lacked. The device count is forced in conftest, before any test module
imports jax.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
E-Rum
force-pushed
the
fix/solver-adaptive-cutoffs-shard-map
branch
from
July 28, 2026 16:28
f72f4cc to
d6aa133
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
get_adaptive_cutoffs_solverseeds itsfori_loopbracket from constants:but the loop body's outputs derive from
r_ij. Undershard_map,r_ijvaries over the data-parallel mesh axis, so carry-in and carry-out disagree on manual axis types andscan's carry type check rejects the loop at trace time:Every data-parallel run with
adaptive_cutoff_method="solver"fails to trace. Single-device runs are unaffected, because callers (e.g. iris) only wrap the step inshard_mapwhen there is more than one device — so this does not show up in a single-GPU smoketest. Thegridmethod has no loop and was never affected.Fix
pcastthe bracket to varying, reading the axes off the data rather than naming them, so it is agnostic to the caller's mesh axis names and a no-op (empty axes) outsideshard_map. The third carry element,0.5 * r_hi, inherits varying-ness fromr_hi.Runtime behaviour is unchanged: each shard already solves on its own atoms, and the annotation carries no computation.
Note this reads
jax.typeof(x).manual_axis_type.varying, which is the JAX >= 0.11 spelling (it was.vmain 0.9/0.10).pyproject.tomlcurrently floors jax at 0.4.30; if older versions are still meant to be supported, the floor should be raised rather than this made conditional — on those releases the attribute is absent and the bug is unfixable in the same way.Tests
tests/test_shard_map.pyruns both adaptive-cutoff methods undershard_mapand checks values and gradients against the single-device result — multi-device coverage the suite currently lacks. It needs no GPU:conftest.pyforces four CPU devices viaXLA_FLAGS, set there because XLA reads it at backend init, before any test module imports jax.[solver]fails with theTypeErrorabove,[grid]passes.12 passed, 36 skippedon jax 0.11.0.shard_map.Verified on hardware (jax 0.11.0, RTX PRO 6000 Blackwell): a 4-way sharded PET training run that previously died at trace time now compiles and proceeds past optimizer/loss setup. A real multi-GPU job is queued to confirm the collectives path end-to-end — the MIG slices used for the quick test cannot form an NCCL communicator, which is a MIG limitation unrelated to this change.
🤖 Generated with Claude Code