Conversation
torch 2.13 hard-removed the named tensor feature (pytorch/pytorch#173895), so Tensor.align_as, align_to, refine_names and rename no longer exist. The auto registration table in default_torch_ops.py dereferences them while building a module-level dict literal, which makes `import thunder` fail outright there with `AttributeError: type object 'Tensor' has no attribute 'align_as'`. Gate those four entries on a new `_TORCH_GREATER_EQUAL_2_13` constant, and add thunder/constants.py as a home for such checks, following litgpt/constants.py.
torch/testing/_internal/common_utils.py reads the deprecated torch._dynamo.config.inline_inbuilt_nn_modules as a default argument, so the FutureWarning fires when the module is imported. With `error::FutureWarning` set, that fails collection of all the distributed tests. Ignore that one message; every other FutureWarning stays fatal.
torchaudio is not imported anywhere in the repo, but CI's adjust-torch-versions.py
pins it in lockstep with torch. torchaudio stopped tracking torch releases after
2.11 ('compatible with future versions of torch'), so on torch 2.14 CI resolves
torchaudio==2.14.0, which does not exist, and every job fails at dependency
install before running a single test.
… into fix/torch-213-named-tensor-import
has_names went away in the same named-tensor removal as align_as/align_to/ refine_names/rename, so import still failed on torch 2.13 without it.
bhimrazy
requested review from
k223kim,
lianakoleva and
tejapulagam
as code owners
September 15, 2026 16:44
MANIFEST.in's `exclude *.toml` dropped pyproject.toml from the sdist. uv builds the wheel from the sdist, so setuptools fell back to legacy setup.py-only metadata there: the wheel came out named `thunder` (an unrelated PyPI project) instead of `lightning-thunder`, and lost its summary, author, license and dependencies. uv now rejects this outright: The source distribution declares name lightning-thunder, but the wheel declares name thunder The published 0.2.6 sdist has the same gap, so installing it from source builds a mis-named, dependency-less wheel.
torch 2.13 reimplemented Function.apply in C, so `Function.apply.__func__` no longer exists and `import thunder` died at module scope. Swapping in the bound `Function.apply` would not work: it is a builtin_function_or_method, and `MyFunc.apply` is a fresh object that is neither equal nor hash-equal to it, so the identity-keyed lookaside map would never match a user's subclass and every autograd.Function would silently stop being traced. The classmethod_descriptor on Function is the one object all subclasses share, so that is the key. The interpreter also has to reach it: its builtin-method unwrapping skipped the case where __self__ is a class, leaving dispatch on the unusable bound object. Handle builtin classmethods there by walking the class's own mro, which also keeps the real provenance of __self__.
torch replaced args_tensor_mask with a required saved_for_backward_idx (plus an optional dirty_idx) on the autograd_function_apply higher-order op, so test_autograd_function_apply failed with KeyError: 'saved_for_backward_idx' on torch >= 2.12. The test helper now supplies it, and thunder's own symbol accepts both new kwargs so the plain-callable path keeps working. Neither index changes what thunder traces: they only drive ctx.mark_dirty/mark_non_differentiable, which thunder does not model -- same as the non_differentiable_idx it already ignores.
thunder's forward-mode AD imports torch._decomp.decompositions_for_jvp, which torch.jit.scripts its decompositions on import. With error::FutureWarning that sank 133 tests in the grads suite on nightly. The filter matches on a prefix so it covers both wordings torch emits: the plain deprecation and the 'not supported in Python 3.14+' variant.
torch 2.14 rejects group_norm inputs with an empty channel or spatial extent
("Expected number of channels to be greater than 0", "Expected HxW to be greater
than 0"); 2.13 and earlier accepted them. The sample generator builds shapes from
(0, 2) in every dim, so it fed the torch reference inputs it now refuses. An empty
batch is still accepted, so only channels and the spatial dims are filtered.
Checked against 2.12.1, 2.13.0, 2.14.0 and 2.15.dev: the boundary is 2.14, not
2.13, hence the new constant.
The notebooks job installs with -U, which wants to replace PyJWT 2.7.0. That copy came from the image's debian packages, so it has no RECORD file and uv refuses: error: uninstall-no-record-file Cannot uninstall PyJWT 2.7.0 --ignore-installed leaves it in place. The image shipping PyJWT via apt rather than pip is the underlying issue.
bhimrazy
changed the title
--ignore-installed is global and takes no argument, so on the -U line it made pip ignore every installed package, not just PyJWT. It pulled a stock torch 2.14.0 over the image's 2.12.0a0+git044db10 build, and nvfuser (compiled against that torch, and version-stamped nvfuser_cu128_torch28) then failed to load: AssertionError: nvFuser is missing! xentropy_cuda ... undefined symbol: _ZN3c104cuda29c10_cuda_check_implementation... Installing PyJWT on its own line gives it a RECORD, so the -U install that follows can manage it normally and leaves the image's matched torch/nvfuser set alone.
The install resolved to TE 2.19, whose source build needs torch/csrc/distributed/c10d/symm_mem/nccl_dev_cap.hpp -- a header torch 2.8.0 does not ship -- so the wheel build failed. Building from source is unavoidable here: TE publishes torch2.8.0 wheels only for cu129 and this image is cu128, so the URL it guesses always 404s. That makes the source build the thing that has to work, and ep.cpp first includes that header in 2.17, so 2.16 is the last version that compiles against torch 2.8.
Three gaps that each mask the next, so all three are needed before any test moves: - MappingKeysView had no __len__, while torch fx does len(user.users.keys()) (torch/fx/graph.py). Added alongside isdisjoint as a plain interpreted method; __iter__/__reversed__ stay lookasides because they build wrapped iterators. - wrap_args_from_list indexed and len()'d its argument, but the CALL_FUNCTION_EX handler two lines above only asserts Iterable -- f(*gen) is legal Python. It now falls back to interpreted iter/next for non-Sequences, the same way UNPACK_SEQUENCE already handles 'an Iterable, not necessarily a sequence'. That path can raise, so the caller propagates EXCEPTION_RAISED. - Sequence.insert was a NotImplementedError stub. Implemented following append, with a LIST_INSERT provenance instruction beside LIST_APPEND/LIST_EXTEND. Verified against eager: f(*gen), f(*d.keys()), f(*list), f(*tuple), and insert at middle/front/past-end/negative indices.
Naming PyJWT explicitly only moved the failure along: with it handled, the -U install hit the next apt-installed package the same way. error: uninstall-no-record-file Cannot uninstall cryptography 41.0.7 The image installs a set of Python packages through apt; none has a RECORD file, so pip cannot uninstall any of them to make way for an upgrade. Rather than name them one at a time, find every distribution with no RECORD and install a pip-managed copy at the same version, so the install that follows can manage them normally and versions do not otherwise change. Custom builds are skipped by their local version segment (torch 2.12.0a0+git044db10, nvfuser_cu128_torch28): they are not on PyPI, and replacing the image's torch is what broke nvfuser earlier.
torch.fx.GraphModule.recompile() writes `_code`/`_lineno_map` (and `_in_spec`/`_out_spec` for pytree codegen) straight into the module's __dict__. Thunder's attribute-modification tracking mistook these codegen-cache internals for real module state and tried to pack them into the epilogue trace, asserting when the value was a plain str/dict instead of a Proxy. Fixes test_higher_order_inplace_alias_update failing intermittently in CI depending on test order/dynamo cache state.
bhimrazy
force-pushed
the
resolve-ci
branch
from
September 16, 2026 02:53
58b8612 to
be879f4
Compare
Taking every RECORD-less distribution was too broad. The image's apt layer also ships C extensions with no PyPI wheel, so pip tried to build them from source and died on missing system headers: Preparing metadata (pyproject.toml) did not run successfully. meson.build:107:11: ERROR: Dependency "dbus-1" not found (tried pkg-config) Only the packages the upgrade wants to replace need a pip-managed copy, so ask pip for its plan up front with --dry-run --report and intersect that with the RECORD-less set. That covers cryptography and PyJWT, the two that actually failed, and leaves dbus-python and PyGObject alone. Names are compared canonicalised, since the report and the installed metadata disagree on case and separators (PyJWT vs pyjwt).
Both show up as test_higher_order_inplace_alias_update, which fails on every
core job -- macOS, Ubuntu and Windows, on torch latest and nightly alike.
The first is a crash. The object.__setattr__ lookaside makes an AnyProxy during
interpretation and parks it on provenance.proxy, which is the same slot the
prologue unpacker uses for its own cache. The unpacker short-circuits on it and
so emits no bsym and registers no param_ordering entry, and unpacking a guard
that reaches through a GraphModule then dies:
KeyError: 5023200512 (jit_ext.py, from_load_attr)
NotImplementedError: Exception occurred unpacking object from ProvenanceRecord(
i5 = LOAD_ATTR(i4, 'nodes')
i6 = LOAD_ATTR(i5, 'direction')
)
Mark those proxies at the point they are made and unpack them properly. Keying
off "missing from param_ordering" instead looks equivalent but is not: it also
catches legitimately cached proxies and regresses test_aliased_input.
The second is a wrong answer the crash was hiding. Since torch 2.14 the fwd
subgraph returns its outputs as a tuple:
return ((y,), (l_x_,))
so the caller does autograd_function_apply[0]. python_return(*sequencify(output))
unpacks a one-element tuple back to a bare tensor, which turns that subscript
into an index into the tensor -- slice to [1], squeeze to [], and thunderfx
quietly returns a scalar where eager and torch.compile both give f32[2]. Return
the structure the graph produced. Nothing changes for bare or multi-element
outputs, since return y1, y2 and return (y1, y2) are the same thing.
The dynamo splitter flattens a node's example_value to see whether anything in
it requires grad. On torch nightly that value can now be a FakeScriptObject,
the fake-mode stand-in for a ScriptObject, which tree_flatten rejects:
TypeError: tree_flatten of type
<class 'torch._library.fake_class_registry.FakeScriptObject'> is not supported.
taking out four DTensor and MoE tests in the distributed job. It is an opaque
object rather than a container, so list it alongside FunctionCtx: it flattens
to itself, and get_requires_grad then correctly finds no tensor in it.
bitsandbytes' cuda backend still calls torch._check_is_size, which torch nightly now deprecates, and filterwarnings promotes it to an error: FutureWarning: _check_is_size will be removed in a future PyTorch release along with guard_size_oblivious. Use _check(i >= 0) instead. That fails test_nvfuser_cse through thunder's quantization transform. Nothing on our side calls it, so ignore it the way we already ignore torch.jit.script's.
TE 2.9 moved global_amax_buffer, global_scale_buffer and global_amax_history_buffer off FP8GlobalStateManager onto a FP8GlobalState dataclass held at .quantization_state, with no alias left behind: AttributeError: type object 'FP8GlobalStateManager' has no attribute 'global_amax_buffer' The CI pin now resolves TE 2.16, so the test hit it. Reach through quantization_state when it is there and fall back to the class otherwise, so this keeps working either side of 2.9.
The splitter resolves a call_method node against torch.Tensor and asserts if it finds nothing. DTensor methods are not on torch.Tensor, and newer torch emits them as plain call_method nodes rather than decomposing them into the call_function prims the code below still describes, so the whole compile dies: AssertionError: Failed to find method redistribute Fall back to DTensor, which puts redistribute and to_local back in reach. Thunder already has symbols for both, so they resolve as supported rather than merely stopping the crash. Unknown methods still trip the assert, and the lookup stays safe when torch.distributed is unavailable, since DTensor is bound to NoneType there.
cross_entropy under nvfuser is the slowest test in the suite and has now timed out twice on the same dtype: 240.24s call test_core_vs_torch_consistency_cross_entropy_nvfuser_cuda_..float16 185.17s call ..float64 Sitting 0.24s over the limit with the next dtype 55s behind is a cap set too low, not a flaky test: ops was the only job here capped at 240 while the rest use 360, and it is the one running the slowest test with 9 workers sharing a single GPU.
The wrapper expanded both operands before handing them to the prim. On a DTensor that puts an aten.expand in front of the op, and sharding propagation either rejects it outright on a sharded singleton dim: Sharding propagation failed for aten.expand.default(Spec(f32[4, 1](...))) or resolves it to a placement the op's own rule would not have chosen, which is what test_dtensor_opinfo_add_torch caught: AssertionError: DTensor placements do not match: (Shard(dim=0),) != (Shard(dim=1),) Nothing needed the explicit broadcast: these ops broadcast natively in the meta, through torch.add/torch.mul on fake tensors, and in execution through the same functions. Sweeping add and mul over shape and placement combinations on a single-rank CPU mesh, 114 cases in total: before 86 correct, 4 placement mismatches, 24 spurious failures after 114 correct, 0 placement mismatches, 0 spurious failures
torch 2.12 reimplemented the DTensor placements in C++, so Shard, Replicate and
Partial now expose __init__ as a builtin bound to a pybind11 function_record
rather than to the instance being built. _type_call_lookaside took it for an
ordinary bound method, overwrote its __self__ and called it, which lands in the
record's own tp_init. That slot throws a bare C++ exception rather than raising
through Python, so it never reaches the except clause and takes the process out:
terminate called after throwing an instance of 'std::runtime_error'
what(): UNEXPECTED CALL OF function_record_PyTypeObject_methods::tp_init_impl
SIGABRT(6)
Build these in a single opaque call, the way the interpreter already handles
opaque callables, keeping the OPAQUE provenance record.
The test that caught this, test_dtensor_from_local_symbolic_values, reproduces
on a single-rank CPU mesh, and afterwards reports the one cache miss and one
cache hit it asserts.
Ordinary types keep the existing path: only a pybind11 __init__ is a
builtin_function_or_method, where dict, list, torch.device, torch.Size,
torch.Tensor and plain Python classes all give a wrapper_descriptor or function.
The DTensor tests build their reference with plain torch.compile, and on this
image inductor's static Triton launcher dies there:
File "torch/_inductor/runtime/static_triton_launcher.py", line 291, in run
self.C_impl._launch_kernel(
RuntimeError: CUDA driver error: invalid argument
The stack is entirely torch, from torch.compile through aot_autograd into
inductor, so thunder is never reached; when a rank goes down this way its peer
is left waiting on a collective until the 300s timeout. Fall back to triton's
own launcher until the image's torch and triton pairing is fixed.
torch._library.opaque_object keeps its custom-class registry in a WeakKeyDictionary and the DTensor placements are registered in it, so torch's FX codegen looking one up puts a weakref into the provenance chain. should_register_for_prologue assumed every callable constant was a function and died on .__name__. Such a callable is neither __getitem__ nor GetSetDescriptorType.__get__, which are the only two a prologue can unpack, so a missing __name__ just takes the existing reject branch.
Switching to triton's own launcher only changed the message: the same three DTensor mul tests fail at the same point, now with "Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?)" instead of "CUDA driver error: invalid argument". The launcher was not the cause, so drop the override rather than carry an env var that claims a fix.
Newer torch puts a DeviceMesh into the graph as a FakeScriptObject rather than the mesh itself. Three places in the splitter rejected it: make_input_proxy raised, so every node taking a mesh became a split reason, and the placeholder round-trip raised outright. Treat it as the opaque value it is. It is its own metadata, and the object it wraps is what the ops and the proxy check want, so hand that over when an input is needed.
Accepting a FakeScriptObject as its own example-input metadata let a graph taking a DeviceMesh reach the reproducer writer, where arg_like raised TypeError: it only knew how to write tensors, numbers and sequences. A DeviceMesh has no source form to write down -- it needs a live process group -- so there is nothing arg_like could emit that would run. Write None for it, and for an input that could not be inferred, which is what the warning above the input list already promises. The script is then one the reader can finish rather than one that cannot be generated at all. That warning was being built and then dropped: the non-serialized branch assigned over input_str instead of appending to it, so the holes arrived unexplained. Append, and raise the warning for a FakeScriptObject too.
test_dtensor_basic_op built its reference with torch.compile, which put an inductor Triton launch in front of a test that is about thunder's lowering of the op over a DTensor. On the nightly image that launch fails: torch/_inductor/runtime/static_triton_launcher.py:291 RuntimeError: CUDA driver error: invalid argument The stack is entirely torch and thunder is never reached. The three failures are exactly the three entries of functions_to_test, and they are the only DTensor tests that compile their reference; the opinfo tests compare against eager and pass. Eager is the better oracle here in any case. On a two-rank gloo mesh the compiled and eager references agree on values, gradients, placements and local strides for all three functions, so what is given up is coverage of inductor, not of the op.
set_device ran after init_process_group, so the group was created before the rank owned a device. NCCL then infers the rank to GPU mapping from the first collective and warns that the device "is currently unknown", noting that the guess can cause a hang. Set the device first and name it with device_id. The parameter is documented as far back as torch 2.7, comfortably inside the 2.7.1 floor in requirements/base.txt.
A rank stuck in NCCL takes its test down with "Process N terminated or timed out" and no retrievable traceback, which says little about the code under test. One such hang, in test_reduce_scatter_executor_torch_dim0_inplace_False, failed an otherwise green run. Retry only that signature. --only-rerun matches against the exception text, so an assertion failure still fails on its first attempt. The retry is best effort rather than a clean second attempt: _join_processes sends SIGTERM and raises without joining, so a rank stuck in an uninterruptible CUDA call may still hold its GPU when the next attempt starts on the same device.
bhimrazy
force-pushed
the
resolve-ci
branch
from
September 18, 2026 07:49
594238d to
9056d2b
Compare
Six tests crash their xdist worker on windows-latest, intermittently across runs but always the same six: Windows fatal exception: code 0xc000001d [gw1] node down: Not properly terminated 0xc000001d is an illegal instruction, not a stack overflow; faulthandler prints "stack overflow" by name for that one and printed the raw code here. Nothing in the interpreter can raise it either, since it walks existing code objects rather than building new ones, so the fault is in native code. torch's Windows CPU wheels carry AVX-512 instructions inside kernels compiled for the AVX2 target, and they fault on a runner whose CPU stops at AVX2, which is why this comes and goes with the machine the job lands on. The reported fault sits in the bfloat16 vector helpers, and that matches what fails here exactly: test_nanogpt and test_litgpt run their model in bfloat16, while test_nanogpt_complete, a larger model on the same ops in float32, passes. Ask for the unvectorized kernels on Windows. Measured on a bfloat16 matmul and gelu, the capability makes no appreciable difference, since matmul goes to MKL rather than these kernels. Also print the capability, so a green run can be told apart from one that happened to land on a CPU the bad kernels do not fault on. Ref: pytorch/pytorch#145702
Six tests intermittently kill their xdist worker on windows-latest: Windows fatal exception: code 0xc000001d [gw1] node down: Not properly terminated Always the same six, on about half the runs, on released and nightly torch alike. The branch has also passed with the same torch on other runs, so this follows the machine the job lands on rather than anything in the tree. The fault is in eager torch, not in our code. For test_litgpt the innermost frames are torch's Linear.forward, reached from the plain reference call before the jitted function is ever invoked; the interpreter variants fault in the same place through the opaque-call path. What separates these six from the tests that pass is bfloat16: test_litgpt_variants builds the same gpt-neox-like model in float32 and passes in the same runs that crash test_litgpt. The previous attempt set ATEN_CPU_CAPABILITY=default on Windows, on the theory that the wheels carry AVX-512 instructions inside kernels compiled for the AVX2 target. A diagnostic step confirmed the capability applied and all six still crashed, so the fault is not in ATen's ISA-dispatched kernels. That does not clear the wider codegen issue in pytorch/pytorch#145702: bfloat16 matmuls go to oneDNN or MKL, which pick their own instruction set and ignore that variable. Those paths are untested, so the workflow change is reverted rather than extended. Skip the six on Windows until this is understood. Nothing in process can retry them: the worker dies outright and pytest only sees a dead channel, so rerunfailures and xfail both have nothing to act on.
bhimrazy
force-pushed
the
resolve-ci
branch
from
September 18, 2026 12:29
26b9224 to
b8cee09
Compare
test_native_fsdp timed out after 900s on the pt_2.8.0-dev job, with both ranks reporting: using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device. That is the warning an earlier commit here removed from DistributedParallelTestCase._run, but these tests do not go through it. test_native_ddp and test_native_fsdp spawn their ranks through a multiprocessing pool and set up their own group in init_per_process_distributed, which called init_process_group with no device bound and no device_id. One caller sets the device afterwards, which is already too late; the rest never set it. Bind the device and pass device_id there as well, so both spawn paths agree. The CPU backend keeps device_id=None. This does not prove the timeout was that mapping, and the test can still hang for another reason. It does remove the condition torch names as a cause of exactly this.
lianakoleva
reviewed
Sep 18, 2026
Comment on lines
+75
to
+77
| # cross_entropy under nvfuser is the slowest test here and landed right on the old 240s cap | ||
| # (240.24s, vs 185s for its next-slowest dtype) with 9 workers sharing the one GPU. 360s | ||
| # matches every other job in this file. |
Contributor
There was a problem hiding this comment.
maybe we can put this in the PR description, not needed here
Suggested change
| # cross_entropy under nvfuser is the slowest test here and landed right on the old 240s cap | |
| # (240.24s, vs 185s for its next-slowest dtype) with 9 workers sharing the one GPU. 360s | |
| # matches every other job in this file. |
lianakoleva
reviewed
Sep 18, 2026
Comment on lines
+2486
to
+2488
| def __len__(self): | ||
| return len(self.mapping) | ||
|
|
Contributor
There was a problem hiding this comment.
is this code used anywhere? i don't see
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Combines three separate fixes into one branch so CI can be evaluated together, ahead of splitting back into #2829, #2830, #2831 individually:
torchaudiofromrequirements/devel.txt— the pin (adjust-torch-versions.pytracking torch) is unsatisfiable at torch 2.14 (torchaudio stopped releasing in lockstep after 2.11), and nothing in the repo imports it.torch._dynamo.config.inline_inbuilt_nn_modulesFutureWarning, which fires at import time fromtorch/testing/_internal/common_utils.pyand breaks collection of all distributed tests.import thunderon PyTorch 2.13: named tensors were removed upstream, soTensor.align_as,align_to,has_names,refine_namesandrenameare gone. Gates those five behind a new_TORCH_GREATER_EQUAL_2_13constant.Why combined
Each fix individually only exposes the next failure in the chain, making it hard to tell from CI alone whether a given fix works. This branch merges all three so we can see the full picture, then will be split back into the standalone PRs once confirmed.