ONNX Runtime

ONNX Runtime

SD.Next includes support for ONNX Runtime.

How To Use

--use-directml is currently not available because torch-directml is not released for the latest PyTorch. This does not prevent use of DmlExecutionProvider.

Set Diffusers pipeline to ONNX Stable Diffusion on the System tab.

Performance

The performance depends on the execution provider.

Execution Providers

Currently, CUDAExecutionProvider and DmlExecutionProvider are supported.

Provider	ONNX	Olive	GPU	CPU
CPUExecutionProvider	✅	❌	❌	✅
DmlExecutionProvider	✅	✅	✅	❌
CUDAExecutionProvider	✅	✅	✅	❌
ROCMExecutionProvider	✅	🚧	✅	❌
OpenVINOExecutionProvider	✅	✅	✅	✅

CPUExecutionProvider

Not recommended.

Enabled by default.

DmlExecutionProvider

You can select DmlExecutionProvider by installing onnxruntime-directml.

DirectX 12 is required (Windows or WSL).

CUDAExecutionProvider

You can select CUDAExecutionProvider by installing onnxruntime-gpu (it may already be installed).

🚧 ROCMExecutionProvider

Olive for ROCm is working in progress.

🚧 OpenVINOExecutionProvider

Under development.

Supported Features

• Models from huggingface
• Hires and second pass (without sdxl refiner)
• .safetensors VAE

Known issues

• SD Inpaint may not work.
• SD Upscale pipeline is not tested.
• SDXL Refiner does not work. (due to onnxruntime's issue)

FAQ

I'm getting OnnxStableDiffusionPipeline.__init__() missing 4 required positional arguments: 'vae_encoder', 'vae_decoder', 'text_encoder', and 'unet'

This is usually caused by a broken model cache generated by a failed conversion or Olive run. Remove the affected cache in models/ONNX/cache. You can also manage cache from the ONNX tab in the UI (enable it in settings if hidden).

Olive

Olive is a hardware-aware model optimization tool that combines compression, optimization, and compilation (from PyPI).

Olive Setup

As Olive optimizes the models in ONNX format, you should set up ONNX Runtime first.

1. Go to System tab → Compute Settings.
2. Select Model, Text Encoder and VAE in Compile Model.
3. Set Model compile backend to olive-ai.

Olive-specific settings are under Olive in Compute Settings.

How to switch to Olive from torch-directml

Run these commands using PowerShell.

.\venv\Scripts\activate
pip uninstall torch-directml
pip install torch torchvision --upgrade
pip install onnxruntime-directml
.\webui.bat

From checkpoint

Model optimization occurs automatically before generation.

Supported model inputs include .safetensors, .ckpt, and Diffusers pretrained models. Optimization time depends on your system and execution provider.

The optimized models are automatically cached and used later to create images of the same size (height and width).

From Huggingface

If your system does not have enough memory for local optimization, or you want to skip local optimization, download an optimized model from Hugging Face.

Go to Models → Huggingface tab and download optimized model.

Advanced Usage

Customize Olive workflow

TBA

Olive Performance

Property	Value
Prompt	a castle, best quality
Negative Prompt	worst quality
Sampler	Euler
Sampling Steps	20
Device	RX 7900 XTX 24GB
Version	olive-ai(0.4.0) onnxruntime-directml(1.16.3) ROCm(5.6) torch(olive: 2.1.2, rocm: 2.1.0)
Model	runwayml/stable-diffusion-v1-5 (ROCm), lshqqytiger/stable-diffusion-v1-5-olive (Olive)
Precision	fp16
Token Merging	Olive(0, not supported) ROCm(0.5)

Olive with DmlExecutionProvider	ROCm

Pros and Cons

Pros

• The generation is faster.
• Uses less graphics memory.

Cons

• Optimization is required for every models and image sizes.
• Some features are unavailable.

Olive FAQ

My execution provider does not show up in my settings

After activating python venv, run this command and try again:

(venv) $ pip uninstall onnxruntime onnxruntime-... -y