Skip to content

[bug]: Generation fails on ROCm 7.1 with torch 2.12.x (torch 2.11.0 works) #9410

Description

@lstein

Splitting this out of #9328, which was closed once the launcher's pins.json was corrected to point at the rocm7.1 wheel index. A second, unrelated problem was reported in that thread and has never had its own tracking issue, so there is currently no open record of it and no removal condition for the workaround we document.

Summary

On ROCm 7.1, generation fails with the torch version the rocm7.1 index currently resolves by default (2.12.x). Pinning torch==2.11.0 on the same machine and the same InvokeAI version restores working generation.

Evidence so far

From the original report in #9328 (@fishd72, RX 9070XT, Linux, InvokeAI 6.13.5rc1):

installing the default version of Torch for ROCm 7.1 will currently install 2.12.1 and there appears to be a bug somewhere else in Invoke that prevents this from working. Fixing the version at Torch 2.11.0 works for me and at least one other AMD user.

The same reporter notes that ComfyUI on the same system runs ROCm 7.1 with torch 2.12.1 successfully, which suggests the incompatibility is in how Invoke uses torch rather than in the ROCm wheels themselves.

At least one other AMD user reproduced the same 2.11.0-works / 2.12.x-fails result.

What is not established

  • Whether torch 2.12.x has any equivalent problem on other backends. No CUDA, CPU, Windows, ARM64, or macOS MPS smoke tests have been run against torch 2.12. All evidence to date is ROCm-only, and this issue makes no claim beyond that.
  • The actual failure mode. We do not yet have a traceback, a failing op, or a minimal repro — only "generation does not work".

How to reproduce

  1. Install InvokeAI on a Linux + AMD (ROCm) system, letting --torch-backend=rocm7.1 resolve torch freely (currently 2.12.x):
    uv pip install invokeai==<VERSION> --python 3.12 --python-preference only-managed --torch-backend=rocm7.1 --force-reinstall
  2. Generate an image.
  3. Reinstall with "torch<2.12" added to the same command and generate again — generation succeeds.

What we need

  • A traceback / console log from a failing 2.12.x run.
  • Confirmation of which torch 2.12.x versions are affected (2.12.0 as well as 2.12.1?).
  • Ideally a minimal repro (which op or model stage fails) so this can be reported upstream if it turns out to be a ROCm/torch bug rather than an Invoke one.

Resolution criteria

Close this when generation works on ROCm 7.1 with the torch version that index resolves by default — either because the underlying bug is fixed in Invoke, or because a fixed torch/ROCm release lands. At that point, remove the torch<2.12 caution from the ROCm tab of the manual installation docs, which links here as its removal condition.

cc @fishd72 — if you still have the failing setup handy, a console log from a 2.12.x run would help a lot.

Metadata

Metadata

Assignees

Labels

amdbugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions