CUDA EP ReduceMax/ReduceMin return finite limits instead of +/-inf for all-infinite inputs

[lackjava-creator]

Describe the issue

Describe the issue

For floating-point reductions, CUDAExecutionProvider returns finite float limits when all reduced values are infinite, while CPUExecutionProvider and TensorRTExecutionProvider return IEEE +/-inf.

Minimal cases:

• ReduceMax([-inf, -inf]): CPU EP returns -inf, TensorRT EP returns -inf, but CUDA EP returns -3.4028235e+38.
• ReduceMin([+inf, +inf]): CPU EP returns +inf, TensorRT EP returns +inf, but CUDA EP returns +3.4028235e+38.

This looks like a numerical correctness issue in the CUDA reduction path for valid IEEE floating-point inputs. The result is stable across repeated runs.

I reproduced the issue with:

• onnxruntime-gpu 1.17.1
• onnxruntime-gpu 1.18.1
• Python API
• Linux x86_64
• CUDA 11.8 / cuDNN 8.6
• NVIDIA GeForce RTX 3080 Ti

I also tried checking onnxruntime-gpu 1.23.2, but that wheel requires CUDA 12 / cuDNN 9 and falls back to CPU on this machine, so I could not validate the latest CUDA EP behavior here.

To reproduce

To reproduce

Run the following script on a machine with CUDAExecutionProvider available:

import numpy as np
import onnx
import onnxruntime as ort
from onnx import TensorProto, helper

print("onnxruntime", ort.__version__)
print("available providers", ort.get_available_providers())


def make_model(op):
    model = helper.make_model(
        helper.make_graph(
            [helper.make_node(op, ["x"], ["z"], axes=[0], keepdims=0)],
            op,
            [helper.make_tensor_value_info("x", TensorProto.FLOAT, [2])],
            [helper.make_tensor_value_info("z", TensorProto.FLOAT, [])],
        ),
        opset_imports=[helper.make_operatorsetid("", 17)],
    )
    model.ir_version = 8
    onnx.checker.check_model(model)
    return model


def run(model, feed, providers, opt):
    so = ort.SessionOptions()
    so.graph_optimization_level = opt
    sess = ort.InferenceSession(model.SerializeToString(), so, providers=providers)
    return sess.get_providers(), sess.run(None, feed)[0]

cases = [
    ("ReduceMax", np.array([-np.inf, -np.inf], dtype=np.float32)),
    ("ReduceMin", np.array([ np.inf,  np.inf], dtype=np.float32)),
]

for op, x in cases:
    model = make_model(op)
    feed = {"x": x}

    cpu_providers, cpu_out = run(
        model,
        feed,
        ["CPUExecutionProvider"],
        ort.GraphOptimizationLevel.ORT_DISABLE_ALL,
    )
    cuda_providers, cuda_out = run(
        model,
        feed,
        ["CUDAExecutionProvider", "CPUExecutionProvider"],
        ort.GraphOptimizationLevel.ORT_ENABLE_ALL,
    )
    trt_providers, trt_out = run(
        model,
        feed,
        ["TensorrtExecutionProvider", "CUDAExecutionProvider", "CPUExecutionProvider"],
        ort.GraphOptimizationLevel.ORT_ENABLE_ALL,
    )

    print(op)
    print("  CPU providers:", cpu_providers, "output:", cpu_out)
    print("  CUDA providers:", cuda_providers, "output:", cuda_out)
    print("  TensorRT providers:", trt_providers, "output:", trt_out)

Observed output with onnxruntime-gpu 1.17.1:

ReduceMax
  CPU providers: ['CPUExecutionProvider'] output: -inf
  CUDA providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'] output: -3.4028235e+38
  TensorRT providers: ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'] output: -inf
ReduceMin
  CPU providers: ['CPUExecutionProvider'] output: inf
  CUDA providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'] output: 3.4028235e+38
  TensorRT providers: ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'] output: inf

I repeated the minimal repro several times and got the same result. I also reproduced the same CPU/CUDA mismatch with onnxruntime-gpu 1.18.1.

Urgency

Not urgent, but this is a numerical correctness issue for CUDAExecutionProvider on IEEE infinity edge cases.

Platform

Linux

OS Version

Ubuntu 20.04.5 LTS (x86_64)

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

onnxruntime-gpu 1.17.1 and 1.18.1

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

CUDA 11.8, cuDNN 8.6.0, NVIDIA driver 580.105.08, NVIDIA GeForce RTX 3080 Ti