Describe the issue
The official onnxruntime-gpu==1.29.0 CPython 3.14 CUDA 13 wheels that include TensorrtExecutionProvider still link TensorRT 10 on both supported x64 platforms:
- Linux x86_64 requests
libnvinfer.so.10 and libnvonnxparser.so.10.
- Windows x64 requests
nvinfer_10.dll and nvonnxparser_10.dll.
They therefore cannot load TensorrtExecutionProvider in a CPython 3.14 environment using the current TensorRT 11 CUDA 13 distribution, which supplies the TensorRT 11 ABI.
I propose moving the TensorRT-enabled CUDA 13 Python-wheel profile to CUDA 13.3 and TensorRT 11.2.1.2 coherently:
- Use CUDA 13.3 and cuDNN 9 for Windows x64 and Linux x86_64 wheels.
- Build the Windows x64 and Linux x86_64 TensorRT providers against TensorRT 11.2.1.2.
- Apply the version change to every Python ABI built by the CUDA 13 packaging pipeline, including CPython 3.14.
- Leave Linux aarch64, CUDA 12, and non-Python package pipelines unchanged. The current Linux aarch64 wheel does not contain a TensorRT provider, so it is not a target for this ABI fix.
NVIDIA publishes TensorRT 11.2.1.2 packages built against CUDA 13.3 for Linux x86_64 and Windows x64.
Urgency
This blocks TensorrtExecutionProvider in CPython 3.14 environments using the current TensorRT 11 CUDA 13 distribution.
Target platforms
- Linux x86_64: CPython 3.14 (
cp314-cp314 and cp314-cp314t)
- Windows x64: CPython 3.14 (
cp314-cp314)
- CUDA 13.3, cuDNN 9, and TensorRT 11.2.1.2 on both platforms
Build scripts
The official pipeline path is:
tools/ci_build/github/azure-pipelines/py-cuda13-packaging-pipeline.yml
tools/ci_build/github/azure-pipelines/stages/py-gpu-packaging-stage.yml
tools/ci_build/github/azure-pipelines/stages/py-linux-gpu-stage.yml
tools/ci_build/github/azure-pipelines/stages/py-win-gpu-stage.yml
tools/ci_build/github/azure-pipelines/templates/jobs/download_win_gpu_library.yml
tools/ci_build/github/linux/build_linux_python_package.sh
The current pipeline uses CUDA 13.0. Its x64 TensorRT inputs are TensorRT 10.x.
The CUDA 13 packaging pipeline therefore needs a CUDA 13.3 Linux x86_64 build image and matching Windows CUDA, cuDNN, and TensorRT inputs.
Error / output
Linux wheel:
onnxruntime_gpu-1.29.0-cp314-cp314-manylinux_2_28_x86_64.whl
SHA-256:
5c50d24541bf61e668dfe0a7a3a62de98def81959c832cc2d376814b661cb006
Relevant readelf -d libonnxruntime_providers_tensorrt.so output:
Shared library: [libcublas.so.13]
Shared library: [libnvinfer.so.10]
Shared library: [libnvonnxparser.so.10]
Shared library: [libcudart.so.13]
Shared library: [libcudnn.so.9]
Windows wheel:
onnxruntime_gpu-1.29.0-cp314-cp314-win_amd64.whl
SHA-256:
4247025e674a666f839c02b90c829277cb2e1be11576d3646cad5c86b7fc26c5
Relevant objdump -p onnxruntime_providers_tensorrt.dll output:
DLL Name: cublas64_13.dll
DLL Name: nvinfer_10.dll
DLL Name: nvonnxparser_10.dll
DLL Name: cudnn64_9.dll
Expected TensorRT dependencies:
Linux: libnvinfer.so.11, libnvonnxparser.so.11
Windows: nvinfer_11.dll, nvonnxparser_11.dll
The CPython 3.14 Linux aarch64 wheel was also inspected and contains CUDA and shared providers, but no TensorRT provider; it is therefore outside this issue's target platform set.
GCC / Compiler Version
The current Linux x86_64 CUDA 13 packaging image uses GCC 14.
References
Describe the issue
The official
onnxruntime-gpu==1.29.0CPython 3.14 CUDA 13 wheels that includeTensorrtExecutionProviderstill link TensorRT 10 on both supported x64 platforms:libnvinfer.so.10andlibnvonnxparser.so.10.nvinfer_10.dllandnvonnxparser_10.dll.They therefore cannot load
TensorrtExecutionProviderin a CPython 3.14 environment using the current TensorRT 11 CUDA 13 distribution, which supplies the TensorRT 11 ABI.I propose moving the TensorRT-enabled CUDA 13 Python-wheel profile to CUDA 13.3 and TensorRT 11.2.1.2 coherently:
NVIDIA publishes TensorRT 11.2.1.2 packages built against CUDA 13.3 for Linux x86_64 and Windows x64.
Urgency
This blocks
TensorrtExecutionProviderin CPython 3.14 environments using the current TensorRT 11 CUDA 13 distribution.Target platforms
cp314-cp314andcp314-cp314t)cp314-cp314)Build scripts
The official pipeline path is:
tools/ci_build/github/azure-pipelines/py-cuda13-packaging-pipeline.ymltools/ci_build/github/azure-pipelines/stages/py-gpu-packaging-stage.ymltools/ci_build/github/azure-pipelines/stages/py-linux-gpu-stage.ymltools/ci_build/github/azure-pipelines/stages/py-win-gpu-stage.ymltools/ci_build/github/azure-pipelines/templates/jobs/download_win_gpu_library.ymltools/ci_build/github/linux/build_linux_python_package.shThe current pipeline uses CUDA 13.0. Its x64 TensorRT inputs are TensorRT 10.x.
The CUDA 13 packaging pipeline therefore needs a CUDA 13.3 Linux x86_64 build image and matching Windows CUDA, cuDNN, and TensorRT inputs.
Error / output
Linux wheel:
onnxruntime_gpu-1.29.0-cp314-cp314-manylinux_2_28_x86_64.whlSHA-256:
5c50d24541bf61e668dfe0a7a3a62de98def81959c832cc2d376814b661cb006Relevant
readelf -d libonnxruntime_providers_tensorrt.sooutput:Windows wheel:
onnxruntime_gpu-1.29.0-cp314-cp314-win_amd64.whlSHA-256:
4247025e674a666f839c02b90c829277cb2e1be11576d3646cad5c86b7fc26c5Relevant
objdump -p onnxruntime_providers_tensorrt.dlloutput:Expected TensorRT dependencies:
The CPython 3.14 Linux aarch64 wheel was also inspected and contains CUDA and shared providers, but no TensorRT provider; it is therefore outside this issue's target platform set.
GCC / Compiler Version
The current Linux x86_64 CUDA 13 packaging image uses GCC 14.
References