NVIDIA TensorRT Overview

TensorRT is NVIDIA's deep learning inference platform designed for high-performance deployment on GPUs. It delivers up to 40x faster inference speeds compared to CPU-only implementations while supporting INT8 and FP16 precision optimizations. TensorRT integrates with major frameworks including TensorFlow, Caffe, MXNet, and PyTorch, with MXNet and PyTorch models requiring conversion to ONNX format first.

Written in C++, TensorRT provides both C++ and Python APIs. The Python API facilitates integration with data processing libraries like NumPy and SciPy, while the C++ API offers maximum performance efficiency.

Core Optimization Techniques

Layer and Tensor Fusion

TensorRT combines layers through horizontal and vertical fusion, significantly reducing computational graph complexity. Horizontal fusion merges convolution, bias, and activation layers into single CBR (Convolution-Bias-ReLU) structures, while vertical fusion combines layers with identical structures but different weights into wider layers. Both optimizations minimize CUDA core usage.

Precision Calibration

Training typically uses FP32 precision, but inference can leverage lower precision formats (FP16/INT8) since backward propagation isn't required. Reduced precision decreases memory usage, latency, and model size.

Kernel Auto-Tuning

TensorRT automatically tunes CUDA kernels for specific algorithms, model architectures, and GPU platforms. This requires platform-specific conversion - engines built on one GPU architecture won't optimize for different architectures.

Dynamic Tensor Memory Management

The platform allocates memory for each tensor during its usage period, eliminating redundent memory allocation and improving reuse efficiency.

Installation and Dependencies

TensorRT Installation

Verify compatibility with your CUDA, cuDNN, PyTorch, and ONNX versions before installation. Available installation methods include Debian packages, RPM, Tar archives, and Windows Zip files.

System Installation Example:

sudo dpkg -i nv-tensorrt-repo-ubuntu1804-cuda10.2-trt7.1.3.4-ga-20200617_1-1_amd64.deb
sudo apt-key add /var/nv-tensorrt-repo-cuda10.2-trt7.1.3.4-ga-20200617/7fa2af80.pub
sudo apt-get update
sudo apt-get install tensorrt
# For Python 3.x:
sudo apt-get install python3-libnvinfer-dev

Tar Archive Installation:

tar -xzvf TensorRT-7.1.3.4.Ubuntu-18.04.x86_64-gnu.cuda-10.2.cudnn8.0.tar.gz
cd TensorRT-7.1.3.4/python
pip install tensorrt-7.1.3.4-cp37-none-linux_x86_64.whl
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/TensorRT-7.1.3.4/lib

PyTorch and CUDA Setup

Install PyTorch with appropriate CUDA version:

# CUDA 10.2
conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch
# Verify installation
python -c "import torch; print(torch.__version__); print(torch.version.cuda)"

cuDNN Installation

Download from cuDNN Archive and install:

tar -xzvf cudnn-11.2-linux-x64-v8.1.1.33.tgz
sudo cp cuda/include/* /usr/local/cuda/include/
sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64/
sudo chmod a+r /usr/local/cuda/include/cudnn.h
sudo chmod a+r /usr/local/cuda/lib64/libcudnn*

Model Conversion Pipeline

PyTorch to ONNX Conversion

Convert trained models to ONNX format:

import torch
import torchvision
import onnx

model = torchvision.models.resnet18(pretrained=True).cuda()
input_sample = torch.randn(1, 3, 224, 224, device='cuda')

torch.onnx.export(model, input_sample, "resnet18.onnx", 
                 input_names=["input"], output_names=["output"],
                 opset_version=10, do_constant_folding=True)

# Validate ONNX model
onnx_model = onnx.load("resnet18.onnx")
onnx.checker.check_model(onnx_model)

ONNX to TensorRT Conversion

Convert ONNX models to TensorRT engines:

import tensorrt as trt

def build_engine(onnx_path, engine_path, precision='fp32', batch_size=1):
    logger = trt.Logger(trt.Logger.WARNING)
    builder = trt.Builder(logger)
    network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))
    parser = trt.OnnxParser(network, logger)
    
    builder.max_batch_size = batch_size
    builder.max_workspace_size = 1 << 30
    
    if precision == 'fp16':
        builder.fp16_mode = True
    elif precision == 'int8':
        builder.int8_mode = True
    
    with open(onnx_path, 'rb') as model:
        if not parser.parse(model.read()):
            for error in range(parser.num_errors):
                print(parser.get_error(error))
            raise ValueError("ONNX parsing failed")
    
    engine = builder.build_cuda_engine(network)
    with open(engine_path, "wb") as f:
        f.write(engine.serialize())
    return engine

Command Line Conversion with trtexec

Use the trtexec tool for conversion:

# Static batch size
trtexec --onnx=model.onnx --saveEngine=model.engine --fp16

# Dynamic batch size
trtexec --onnx=model.onnx --minShapes=input:1x3x224x224 \
        --optShapes=input:8x3x224x224 --maxShapes=input:16x3x224x224 \
        --saveEngine=model_dynamic.engine --fp16

Inference Implementation

Python Inference Example

import tensorrt as trt
import pycuda.driver as cuda
import pycuda.autoinit
import numpy as np

def load_engine(engine_path):
    with open(engine_path, "rb") as f, trt.Runtime(trt.Logger(trt.Logger.WARNING)) as runtime:
        return runtime.deserialize_cuda_engine(f.read())

def inference(engine, input_data):
    context = engine.create_execution_context()
    
    # Allocate device memory
    d_input = cuda.mem_alloc(input_data.nbytes)
    d_output = cuda.mem_alloc(engine.get_binding_shape(1)[0] * np.dtype(np.float32).itemsize)
    bindings = [int(d_input), int(d_output)]
    
    stream = cuda.Stream()
    cuda.memcpy_htod_async(d_input, input_data, stream)
    context.execute_async_v2(bindings, stream.handle)
    
    output = np.empty(engine.get_binding_shape(1), dtype=np.float32)
    cuda.memcpy_dtoh_async(output, d_output, stream)
    stream.synchronize()
    
    return output

C++ Inference Workflow

For embedded deployment, use the weight-to-serialized engine appproach:

1. Convert PyTorch model to weights file:

# pth_to_wts.py
import torch
import struct

def save_weights(model, output_path):
    with open(output_path, 'w') as f:
        f.write(f"{len(model.state_dict())}\n")
        for name, param in model.state_dict().items():
            flattened = param.cpu().numpy().ravel()
            f.write(f"{name} {len(flattened)} ")
            for value in flattened:
                f.write(struct.pack('>f', float(value)).hex())
            f.write('\n')

2. Build engine from weights in C++ using TensorRT's layer-by-layer network construction API
3. Perform inference with optimized engine on target hardware

Performance Considerations

• Always build engines on the target deployment hardware for optimal performance
• Use appropriate precision modes (FP32, FP16, INT8) based on accuracy and speed requirements
• Leverage TensorRT's profiling tools (NVIDIA Nsight Systems, DLProf) for performance analysis
• Consider memory constraints when deploying to embedded systems

Related Tools and Integration

• NVIDIA Triton Inference Server: Orchestrates multiple models across CPU/GPU with REST/gRPC endpoints
• NVIDIA DALI: High-performance data preprocessnig with TensorRT integration
• TensorFlow-TensorRT (TF-TRT): Direct TensorRT integration within TensorFlow graphs
• PyTorch Quantization Toolkit: Tools for training reduced-precision models compatible with TensorRT
• PyTorch Automatic SParsity (ASP): Enables structured sparsity for improved performance on Ampere GPUs