Installing TensorRT

This installation follows the same principle as the preceding installations: first select an appropriate compatible version.

1. Determine the TensorRT Version to Install

Open the official NVIDIA download page: TensorRT Downloads.

Select version 8 here. This choice is mainly related to the subsequent model conversion; the highest TensorRT version currently supported by SpireCV is version 8. Click to open it.

Select the stable 8.6 GA version and click it to download.

2. Install TensorRT

Copy the downloaded archive to your home directory and extract it:

tar xzvf TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-12.0.tar.gz

The result will look similar to the following:

Add the TensorRT library to the environment variables:

# Open the ~/.bashrc file in the home directory.
# Add:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/amov/TensorRT-8.6.1.6/lib

# Append it to the environment variables configured earlier.

Install the Python API

cd /home/amov/TensorRT-8.6.1.6/python

# Install the package that matches the Python version on the system.
# Check the Python version on the system:

python3 --version

# Select and install the API package matching the displayed version.

# The Python version here is 3.8.10, so select cp38.

pip install tensorrt-8.6.1-cp38-none-linux_x86_64.whl

Install the Python UFF package to support TensorFlow model conversion

cd /home/amov/TensorRT-8.6.1.6/uff

pip install uff-0.6.9-py2.py3-none-any.whl

# Open a new terminal in the same location and test whether Python UFF was installed successfully.
which convert-to-uff

# Output:
/home/amov/.local/bin/convert-to-uff

Install graphsurgeon to support custom structures

cd /home/amov/TensorRT-8.6.1.6/graphsurgeon

pip install graphsurgeon-0.4.6-py2.py3-none-any.whl

3. Verify TensorRT

Open a new terminal.

python3

import tensorrt as trt
print(trt.__version__)

If installation was successful, the output is:
8.6.1

4. About TensorRT

NVIDIA® TensorRT™ is an API ecosystem for high-performance deep-learning inference. TensorRT includes an inference runtime and model optimizations that deliver low latency and high throughput for production applications. The TensorRT ecosystem includes TensorRT, TensorRT-LLM, TensorRT Model Optimizer, and TensorRT Cloud.

TensorRT is a C++ inference framework that runs on a range of NVIDIA GPU hardware platforms. Models trained using PyTorch, TensorFlow, or other frameworks can be converted to TensorRT format and then run with the TensorRT inference engine, substantially increasing model execution speed on NVIDIA GPUs.

Its main functions include:

1. Model optimization: Optimizes trained models to reduce latency and increase throughput. 2. Precision selection: Supports FP32, FP16, and INT8 precision to balance performance and accuracy requirements. 3. Efficient inference: Accelerates deep-learning model inference on GPUs. 4. Cross-platform support: Supports multiple hardware platforms, including data centers and edge devices. 5. Framework compatibility: Integrates with deep-learning frameworks such as TensorFlow and PyTorch to simplify deployment.

TensorRT can significantly improve the real-time performance of deep-learning applications and is a key tool for inference deployment.