TensorRT EP engine cache reuses engine for same graph name with different static input shape

[lackjava-creator]

Describe the issue

When TensorrtExecutionProvider engine caching is enabled, two different ONNX models with the same graph name and the same structure but different static input shapes can collide in the TensorRT engine cache.

In the minimal reproducer below:

• Model A is Add(x, y) with static input shape [3, 3] and graph name main_graph.
• Model B is Add(x, y) with static input shape [4, 4] and the same graph name main_graph.
• Model B passes when run alone.
• Running A first and then B with the same trt_engine_cache_path makes B fail with TensorRT setInputShape() static dimension mismatch.
• Disabling engine cache, changing the graph names, or using different trt_engine_cache_prefix values makes both models pass.

This looks like the default TensorRT EP engine cache key/prefix does not distinguish static input shapes when graph names collide, so the engine generated for the 3x3 model is reused for the 4x4 model.

I found #22179, but this case does not use trt_dump_ep_context_model or weight-stripped engines. It reproduces with a single Add node and trt_engine_cache_enable=True.

To reproduce

import json
import tempfile

import numpy as np
import onnx
import onnxruntime as ort
from onnx import TensorProto, helper


def make_add_model(n: int, graph_name: str = "main_graph") -> onnx.ModelProto:
    x = helper.make_tensor_value_info("x", TensorProto.FLOAT, [n, n])
    y = helper.make_tensor_value_info("y", TensorProto.FLOAT, [n, n])
    z = helper.make_tensor_value_info("z", TensorProto.FLOAT, [n, n])
    graph = helper.make_graph(
        [helper.make_node("Add", ["x", "y"], ["z"])],
        graph_name,
        [x, y],
        [z],
    )
    model = helper.make_model(graph, opset_imports=[helper.make_operatorsetid("", 17)])
    model.ir_version = 8
    return model


def run_one(label: str, n: int, cache_dir: str, *, cache=True, graph_name="main_graph", prefix=None):
    opts = {}
    if cache:
        opts = {"trt_engine_cache_enable": True, "trt_engine_cache_path": cache_dir}
        if prefix is not None:
            opts["trt_engine_cache_prefix"] = prefix
    providers = [
        ("TensorrtExecutionProvider", opts),
        "CUDAExecutionProvider",
        "CPUExecutionProvider",
    ]
    so = ort.SessionOptions()
    so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
    so.log_severity_level = 4
    sess = ort.InferenceSession(
        make_add_model(n, graph_name).SerializeToString(), so, providers=providers
    )
    feed = {
        "x": np.ones((n, n), dtype=np.float32),
        "y": np.ones((n, n), dtype=np.float32),
    }
    out = sess.run(None, feed)
    return [list(np.asarray(v).shape) for v in out]


def run_sequence(name, *, cache=True, graph_names=("main_graph", "main_graph"), prefixes=(None, None)):
    cache_dir = tempfile.mkdtemp(prefix=f"ort_trt_cache_{name}_")
    result = []
    for label, n, graph_name, prefix in [
        ("A_3x3", 3, graph_names[0], prefixes[0]),
        ("B_4x4", 4, graph_names[1], prefixes[1]),
    ]:
        try:
            result.append(
                {
                    "label": label,
                    "status": "PASS",
                    "shapes": run_one(
                        label,
                        n,
                        cache_dir,
                        cache=cache,
                        graph_name=graph_name,
                        prefix=prefix,
                    ),
                }
            )
        except Exception as exc:
            result.append(
                {
                    "label": label,
                    "status": "ERROR",
                    "type": type(exc).__name__,
                    "detail": str(exc),
                }
            )
            break
    print(name, json.dumps(result, indent=2))
    return result


print("onnxruntime", ort.__version__)
print("available_providers", ort.get_available_providers())

# Unexpected: B fails because it reuses the TensorRT engine cached for A.
run_sequence("same_graph_name_cache_enabled")

# Controls: all pass.
run_sequence("cache_disabled", cache=False)
run_sequence("different_graph_names", graph_names=("graph_A", "graph_B"))
run_sequence("different_cache_prefixes", prefixes=("A", "B"))

Expected behavior

Both models should run successfully. In particular, the 4x4 Add model should not reuse an incompatible engine built for the 3x3 model.

Actual behavior

same_graph_name_cache_enabled fails on the second model:

onnxruntime 1.17.1
available_providers ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'AzureExecutionProvider', 'CPUExecutionProvider']
same_graph_name_cache_enabled [
  {
    "label": "A_3x3",
    "status": "PASS",
    "shapes": [[3, 3]]
  },
  {
    "label": "B_4x4",
    "status": "ERROR",
    "type": "RuntimeException",
    "detail": "[ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : ... TensorRT EP failed to call nvinfer1::IExecutionContext::setInputShape() for input 'x'"
  }
]

The TensorRT log contains:

Error Code 3: API Usage Error
Parameter check failed at: runtime/api/executionContext.cpp::setInputShape::2264
condition: engineDims.d[i] == dims.d[i]
Static dimension mismatch while setting input shape.

The control cases pass:

cache_disabled: A_3x3 PASS, B_4x4 PASS
different_graph_names: A_3x3 PASS, B_4x4 PASS
different_cache_prefixes: A_3x3 PASS, B_4x4 PASS

I repeated the default-cache case three times on the same setup; it failed 3/3. The controls passed.

Environment

date: 2026-08-25T14:53:20+08:00
host: autodl-container-21104cb00b-3b1b3d4b
python: Python 3.10.20
onnxruntime/numpy/onnx:
onnxruntime 1.17.1
providers ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'AzureExecutionProvider', 'CPUExecutionProvider']
numpy 1.26.4
onnx 1.17.0
nvidia-smi:
NVIDIA GeForce RTX 3080 Ti, 580.105.08, 12288 MiB
nvcc:
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
TensorRT path: /root/TensorRT-8.6.1.6
lrwxrwxrwx 1 root root        19 Apr 27  2023 /root/TensorRT-8.6.1.6/targets/x86_64-linux-gnu/lib/libnvinfer.so -> libnvinfer.so.8.6.1
lrwxrwxrwx 1 root root        19 Apr 27  2023 /root/TensorRT-8.6.1.6/targets/x86_64-linux-gnu/lib/libnvinfer.so.8 -> libnvinfer.so.8.6.1
-rwxr-xr-x 1 root root 235369632 Apr 27  2023 /root/TensorRT-8.6.1.6/targets/x86_64-linux-gnu/lib/libnvinfer.so.8.6.1

Possible workaround

Use unique graph names or set distinct trt_engine_cache_prefix values for models with different static input shapes, or disable TensorRT engine caching.