When a model has to run inside a C++ product (a robot, an imaging pipeline, a trading system, a game engine or a high-throughput service), the hard problems are rarely the model itself. They are the export path from Python, the choice of runtime, memory ownership across CPU and GPU, concurrency, and keeping latency predictable under load.
This course covers the main production routes side by side. You export models with torch.export and package them with AOTInductor for C++ loading, and use LibTorch where its full API is needed (noting TorchScript's legacy status). You run ONNX models through the ONNX Runtime C++ API with execution providers and I/O binding. You build TensorRT engines from ONNX, manage precision (FP16, BF16, FP8, INT8 and FP4 where hardware supports it) through strongly typed networks and explicit quantization, handle dynamic shapes with optimization profiles, and run them with the TensorRT runtime API, including TensorRT for RTX on client GPUs. You then write CUDA code where libraries stop: custom kernels, TensorRT plugins, CUDA graphs, streams and pinned memory, with CUTLASS as a reference point. The final modules build concurrent inference services (thread pools, request batching, lifetime management with RAII, lock-free queues) and cover profiling with Nsight Systems and Nsight Compute, numerical validation against the Python reference, and build, packaging and ABI concerns. Version-specific facts are as of October 2026 (PyTorch 2.14, ONNX Runtime 1.31, TensorRT 11.4 with migration notes for teams still on 10.x, TensorRT for RTX 1.7, CUDA 13.4); TensorRT 11 made strongly typed networks mandatory, so reduced precision is expressed in the model graph rather than through builder flags.