Rust has become a serious option for the parts of an AI system where Python is too slow, too memory-hungry or too hard to deploy: tokenizers, pre- and post-processing, embedding and reranking services, edge inference and the serving layer itself. Doing this well requires more than calling a crate. You need to understand how tensors are laid out and moved, where the FFI boundary sits and what invariants it must uphold, how GPU memory and streams interact with async runtimes, and how to measure rather than guess.
The course starts with the ecosystem map and the decision between pure-Rust frameworks (Candle, Burn) and bindings to native runtimes (ONNX Runtime via ort, llama.cpp via bindgen-generated or community bindings). You load safetensors and GGUF weights, run transformer inference with Candle, build models and import ONNX graphs with Burn and its CubeCL backends, and run ONNX models with execution providers. You then go below the safe APIs: writing sound FFI wrappers with bindgen and cxx, ownership and lifetimes across the boundary, Send/Sync for native handles, and CUDA integration through cudarc. The final modules build production services with Tokio and Axum (dynamic batching, backpressure, cancellation, streaming), expose Rust to Python with PyO3 and maturin, and cover profiling, allocation control, SIMD and release engineering. Crate versions move quickly; versions and API details are stated as of October 2026, and several key crates (including ort 2.x) were still on release candidates at that date.