ONNX-TensorRT is a TensorRT backend for ONNX that parses ONNX models for execution with NVIDIA's TensorRT inference engine.
The tool solves the problem of running ONNX-format neural network models on NVIDIA GPUs by translating ONNX operator definitions into TensorRT's optimized kernel implementations. It works by parsing ONNX model files and building them into TensorRT engines that can execute on NVIDIA hardware with hardware-specific optimizations applied during the build process.
Developers should choose this tool when deploying ONNX models on NVIDIA GPUs and seeking the performance benefits of TensorRT's optimizations. It suits production inference workloads where model latency and throughput matter. The project supports full-dimensions and dynamic shape execution, allowing flexible input dimensions at runtime. An operator support matrix documents which ONNX operators are currently supported, which developers should consult before adoption to ensure their model's operations are covered. The tool provides both C++ and Python interfaces for integration into applications.
The project maintains active development focused on supporting the latest TensorRT versions with full-dimensions and dynamic shape capabilities. Development branches track specific TensorRT versions, allowing users on older releases to find compatible code. The maintainers provide documentation including a changelog and FAQ addressing commonly encountered issues, indicating ongoing attention to user friction points. The project includes multiple implementation options for certain operators, such as InstanceNormalization, where users can select between native TensorRT and plugin-based implementations depending on their performance requirements.
