Installing CUDA + cuDNN + TensorRT

cd ~/SpireCV/scripts/x86-cuda

# CUDA116+CUDNN841+TensorRT8406
chmod +x ubuntu2004-cuda-cudnn-11-6.sh && ./ubuntu2004-cuda-cudnn-11-6.sh
# Power off and restart the computer after installation is complete.

---

If downloads are slow, you can download our bundled package from Baidu Netdisk:
Dependencies: https://pan.baidu.com/s/1IC1oZ-k1C_0Va28tuknT-Q?pwd=yp2u

After downloading the files, place them in the specified location. In the ubuntu2004-cuda-cudnn-11-6.sh script, add # before each of the four sections beginning with wget https to comment them out, and then run the script.

1. CUDA

CUDA (Compute Unified Device Architecture) is an integrated hardware and software technology developed by NVIDIA and is the company's official name for GPGPU computing. This technology allows users to use NVIDIA GPUs for computations beyond graphics processing and was the first development environment that enabled a GPU to be used with a C compiler. The CUDA Toolkit can compile only NVIDIA's own CUDA C code (it provides only linking capabilities for OpenCL)—that is, the portion that runs on the GPU—into PTX intermediate code or machine code for a specific NVIDIA GPU architecture (officially called "device code" by NVIDIA). The C/C++ code that runs on the CPU (officially called "host code" by NVIDIA) still relies on an external compiler, such as Microsoft Visual Studio on Microsoft Windows or primarily GCC on Linux.

The NVIDIA® CUDA® Toolkit provides a development environment for creating high-performance, GPU-accelerated applications. It enables you to develop, optimize, and deploy applications on GPU-accelerated embedded systems, desktop workstations, enterprise data centers, cloud-based platforms, and supercomputers. The toolkit includes GPU-accelerated libraries, debugging and optimization tools, a C/C++ compiler, and runtime libraries.

Its main functions include:

1. Accelerated computing: Uses the GPU's parallel-processing capabilities to accelerate scientific computing, machine learning, image processing, and other tasks. 2. Parallel processing: Supports large-scale parallel computing to improve task execution efficiency. 3. Development flexibility: Provides extensive libraries and tools that help developers implement complex algorithms on GPUs. 4. Cross-platform support: Supports multiple operating systems and programming languages, enabling broad use across different fields. 5. Deep-learning support: Provides optimized support for deep-learning frameworks such as TensorFlow and PyTorch to increase training speed.

CUDA significantly improves the performance of compute-intensive tasks, allowing complex computations to be completed in less time.

2. cuDNN

The NVIDIA CUDA® Deep Neural Network library (cuDNN) is a GPU-accelerated library of primitives for deep neural networks. cuDNN provides highly tuned implementations of standard routines such as forward and backward convolution, attention, matrix multiplication, pooling, and normalization.

Its main functions include:

1. Accelerating deep-learning computations: Provides efficient convolution, pooling, normalization, and other operations to accelerate neural-network training and inference. 2. Performance optimization: Optimizes operations for different GPU architectures to improve computational efficiency. 3. Support for multiple frameworks: Compatible with mainstream deep-learning frameworks such as TensorFlow, PyTorch, and Caffe. 4. Simplified development: Provides easy-to-use APIs, reducing the work required for developers to implement complex algorithms.

Using cuDNN significantly increases the training speed and performance of deep-learning models, making it an important component of the deep-learning workflow.

To learn about the relationship between CUDA and cuDNN, read What Exactly Is the Relationship Between CUDA and cuDNN?.

3. TensorRT

NVIDIA® TensorRT™ is an API ecosystem for high-performance deep-learning inference. TensorRT includes an inference runtime and model optimizations that deliver low latency and high throughput for production applications. The TensorRT ecosystem includes TensorRT, TensorRT-LLM, TensorRT Model Optimizer, and TensorRT Cloud.

TensorRT is a C++ inference framework that runs on a range of NVIDIA GPU hardware platforms. Models trained using PyTorch, TensorFlow, or other frameworks can be converted to TensorRT format and then run with the TensorRT inference engine, substantially increasing model execution speed on NVIDIA GPUs.

Its main functions include:

1. Model optimization: Optimizes trained models to reduce latency and increase throughput. 2. Precision selection: Supports FP32, FP16, and INT8 precision to balance performance and accuracy requirements. 3. Efficient inference: Accelerates deep-learning model inference on GPUs. 4. Cross-platform support: Supports multiple hardware platforms, including data centers and edge devices. 5. Framework compatibility: Integrates with deep-learning frameworks such as TensorFlow and PyTorch to simplify deployment.

TensorRT can significantly improve the real-time performance of deep-learning applications and is a key tool for inference deployment.