Wirewalk Instituteof AI
dev-303 Expert Available

C++ for AI: LibTorch, ONNX Runtime, TensorRT and CUDA Integration

For senior C++ engineers who must run models in-process at production latency on CPUs and NVIDIA GPUs, and build safe, high-throughput inference services around them.

Level
Expert
Study time
24 hours
Lessons
21 in 6 modules
Exam
25 questions

About this course

When a model has to run inside a C++ product (a robot, an imaging pipeline, a trading system, a game engine or a high-throughput service), the hard problems are rarely the model itself. They are the export path from Python, the choice of runtime, memory ownership across CPU and GPU, concurrency, and keeping latency predictable under load.

This course covers the main production routes side by side. You export models with torch.export and package them with AOTInductor for C++ loading, and use LibTorch where its full API is needed (noting TorchScript's legacy status). You run ONNX models through the ONNX Runtime C++ API with execution providers and I/O binding. You build TensorRT engines from ONNX, manage precision (FP16, BF16, FP8, INT8 and FP4 where hardware supports it) through strongly typed networks and explicit quantization, handle dynamic shapes with optimization profiles, and run them with the TensorRT runtime API, including TensorRT for RTX on client GPUs. You then write CUDA code where libraries stop: custom kernels, TensorRT plugins, CUDA graphs, streams and pinned memory, with CUTLASS as a reference point. The final modules build concurrent inference services (thread pools, request batching, lifetime management with RAII, lock-free queues) and cover profiling with Nsight Systems and Nsight Compute, numerical validation against the Python reference, and build, packaging and ABI concerns. Version-specific facts are as of October 2026 (PyTorch 2.14, ONNX Runtime 1.31, TensorRT 11.4 with migration notes for teams still on 10.x, TensorRT for RTX 1.7, CUDA 13.4); TensorRT 11 made strongly typed networks mandatory, so reduced precision is expressed in the model graph rather than through builder flags.

What you will be able to do

  • Choose an export and runtime path (AOTInductor/LibTorch, ONNX Runtime, TensorRT, OpenVINO or llama.cpp) for a model, hardware target and latency budget, and justify it
  • Export a PyTorch model with torch.export, package it with AOTInductor and run it from C++, validating outputs numerically against the Python reference
  • Run models through the ONNX Runtime C++ API with execution providers, I/O binding and session configuration tuned for throughput or latency
  • Build and run a TensorRT engine with dynamic shapes, optimization profiles and reduced precision, and diagnose accuracy loss from calibration or precision choices
  • Write and integrate a CUDA kernel or TensorRT plugin, using streams, pinned memory and CUDA graphs to remove launch and copy overhead
  • Design a concurrent C++ inference service with batching, bounded queues, RAII-managed GPU resources and graceful overload behavior
  • Profile an end-to-end inference path with Nsight Systems and Nsight Compute and turn the findings into measured improvements

Syllabus

6 modules · 21 lessons · about 24 hours

Module 1: Runtimes and Export Paths

How a Python-trained model becomes something C++ can run.
3 lessons
  1. The C++ Inference Landscape: LibTorch, ONNX Runtime, TensorRT, OpenVINO and llama.cpp
  2. From PyTorch to C++: torch.export, AOTInductor Packages and the Legacy of TorchScript
  3. ONNX Export Pitfalls: Dynamic Axes, Unsupported Ops and Opset Choices

Module 2: LibTorch and ONNX Runtime in C++

Two general-purpose runtimes and how to use them well.
4 lessons
  1. LibTorch Tensors, Devices and Inference Mode in Production Code
  2. Loading AOTInductor Packages from C++ and Validating Against the Reference
  3. ONNX Runtime C++ API: Sessions, Execution Providers and I/O Binding
  4. Thread Safety and Session Sharing Under Concurrent Load

Module 3: TensorRT Engines

Building, tuning and running NVIDIA's optimizing runtime.
4 lessons
  1. Building Engines from ONNX: Builder Config, Dynamic Shapes and Optimization Profiles
  2. Precision and Quantization: FP16, BF16, FP8, INT8 and FP4 Trade-offs
  3. The Runtime API: Execution Contexts, enqueueV3 and Engine Caching
  4. TensorRT for RTX and Portable Engines on Client GPUs

Module 4: CUDA Integration and Custom Operations

Where the libraries stop and your own kernels begin.
4 lessons
  1. Streams, Events, Pinned Memory and Overlapping Copy with Compute
  2. Writing a Custom CUDA Kernel for Pre- or Post-Processing
  3. TensorRT Plugins and Custom Operators for ONNX Runtime
  4. CUDA Graphs and Launch Overhead in Small-Batch Inference

Module 5: Safe, High-Performance Inference Services

Concurrency, memory safety and overload behavior in C++.
3 lessons
  1. RAII for GPU Resources: Ownership, Lifetimes and Leak-Free Shutdown
  2. Dynamic Batching, Bounded Queues and Backpressure
  3. Embedding a Backend in Triton Inference Server vs Building Your Own Service

Module 6: Profiling, Validation and Release

Proving performance and correctness, then shipping it.
3 lessons
  1. Nsight Systems and Nsight Compute: Finding the Real Bottleneck
  2. Numerical Validation, Tolerances and Regression Tests for Model Upgrades
  3. Builds, ABI and Packaging: CUDA Versions, Driver Compatibility and Containers

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by PyTorch (LibTorch, torch.export, AOTInductor), Microsoft (ONNX Runtime C++ API), NVIDIA (TensorRT, TensorRT for RTX, CUDA, CUTLASS, Nsight Systems, Nsight Compute, Triton Inference Server), Intel (OpenVINO), or ggml-org (llama.cpp). Product and company names are trademarks of their respective owners.

More in AI in Software Development

All 19 courses in this track
dev-001 Free intro Available

Your First AI API Call: A One-Hour Hands-On Start

A free, one-hour, hands-on start built on one realistic job: sorting a property manager's 1,150 monthly maintenance requests. You get an API key the safe way, make real calls with curl and an official Python SDK, run them over forty work orders, read stop reasons, usage and errors, estimate the monthly cost, and decide whether the script is ready. Exercises run on your own provider account for well under a dollar.

1 h · 7 lessons Free intro
dev-201 Intermediate Available

Python AI Integration: SDKs, Async, Pydantic, Packaging and Testing

A code-first course on integrating LLMs into Python the way production code needs it: uv projects and secrets, the OpenAI, Anthropic and Google GenAI SDKs side by side, async concurrency and streaming, Pydantic structured outputs and tools, retries, local models, embeddings and RAG, FastAPI and pandas integration, testing and evals with pytest, observability, cost tracking and deployment.

15 h · 19 lessons See price at checkout
dev-202 Intermediate Waitlist

Perl and AI: Modernizing Scripts and Text Pipelines with LLM APIs

Call OpenAI, Anthropic, Gemini and local Ollama models from Perl with real CPAN modules, add streaming, concurrency and validated JSON output to text pipelines, and use AI coding assistants to modernize legacy Perl behind characterization tests.

12 h · 18 lessons Join the waitlist