Wirewalk Instituteof AI
dev-301 Expert Available

Rust for AI: Candle, Burn, ONNX Runtime and High-Performance Services

For experienced Rust and systems engineers who want to run models in-process, bind to native inference libraries safely, and build memory-safe, high-throughput AI services.

Level
Expert
Study time
22 hours
Lessons
22 in 6 modules
Exam
25 questions

About this course

Rust has become a serious option for the parts of an AI system where Python is too slow, too memory-hungry or too hard to deploy: tokenizers, pre- and post-processing, embedding and reranking services, edge inference and the serving layer itself. Doing this well requires more than calling a crate. You need to understand how tensors are laid out and moved, where the FFI boundary sits and what invariants it must uphold, how GPU memory and streams interact with async runtimes, and how to measure rather than guess.

The course starts with the ecosystem map and the decision between pure-Rust frameworks (Candle, Burn) and bindings to native runtimes (ONNX Runtime via ort, llama.cpp via bindgen-generated or community bindings). You load safetensors and GGUF weights, run transformer inference with Candle, build models and import ONNX graphs with Burn and its CubeCL backends, and run ONNX models with execution providers. You then go below the safe APIs: writing sound FFI wrappers with bindgen and cxx, ownership and lifetimes across the boundary, Send/Sync for native handles, and CUDA integration through cudarc. The final modules build production services with Tokio and Axum (dynamic batching, backpressure, cancellation, streaming), expose Rust to Python with PyO3 and maturin, and cover profiling, allocation control, SIMD and release engineering. Crate versions move quickly; versions and API details are stated as of October 2026, and several key crates (including ort 2.x) were still on release candidates at that date.

What you will be able to do

  • Choose between Candle, Burn, ONNX Runtime (ort), llama.cpp bindings and mistral.rs for a given model, target hardware and deployment constraint, and defend the choice
  • Implement in-process transformer inference in Rust that loads safetensors or GGUF weights, tokenizes with the tokenizers crate and runs on CPU or CUDA
  • Write a sound safe wrapper around a C inference API, documenting the invariants behind every unsafe block and handling ownership, lifetimes, thread safety and error mapping
  • Integrate GPU execution with CUDA streams and device memory without blocking the async runtime or leaking device allocations
  • Build an Axum/Tokio inference service with dynamic batching, backpressure, cancellation, streaming responses and Prometheus-style metrics
  • Expose a Rust inference or preprocessing component to Python with PyO3 and maturin, releasing the GIL and avoiding unnecessary copies
  • Profile and optimize a Rust AI workload using flamegraphs, allocation tracking, SIMD and quantization, and verify gains with reproducible benchmarks

Syllabus

6 modules · 22 lessons · about 22 hours

Module 1: The Rust AI Ecosystem and Its Trade-offs

Pure-Rust frameworks versus bindings to native runtimes, and how to pick.
3 lessons
  1. Ecosystem Map: Candle, Burn, ort, llama.cpp Bindings and mistral.rs
  2. Tensors, Memory Layout and Weight Formats: safetensors, GGUF and ONNX
  3. Tokenizers in Rust: The tokenizers Crate and Byte-Level Pitfalls

Module 2: Inference with Candle and Burn

Running and building models with pure-Rust frameworks.
4 lessons
  1. Candle: Loading Hugging Face Models and Running Transformer Inference
  2. Quantized Inference in Candle: GGUF and K-Quants on CPU and GPU
  3. Burn: Backends, CubeCL Kernels and Importing ONNX Models
  4. Writing a Custom Operation and Testing It Against a Reference

Module 3: ONNX Runtime and Native Libraries Through FFI

Using mature C/C++ runtimes from Rust without giving up soundness.
4 lessons
  1. ONNX Runtime with ort: Sessions, Execution Providers and IoBinding
  2. bindgen and cxx: Generating and Hardening Bindings to C and C++
  3. Sound Wrappers: Ownership, Lifetimes, Send/Sync and Panics Across FFI
  4. Embedding llama.cpp from Rust: Model, Context and Sampling Lifecycles

Module 4: GPU and CUDA Integration

Device memory, streams and kernels from Rust, and how they meet async code.
3 lessons
  1. cudarc: Device Memory, Streams and Launching Kernels
  2. Keeping GPUs Busy Without Blocking Tokio: spawn_blocking, Channels and Dedicated Threads
  3. Zero-Copy Paths: Pinned Memory, Shared Buffers and Avoiding Host Round Trips

Module 5: High-Performance AI Services

Turning an inference engine into a service that holds up under load.
4 lessons
  1. Axum and Tokio Service Design: Dynamic Batching, Backpressure and Cancellation
  2. Streaming Tokens with SSE and WebSockets
  3. PyO3 and maturin: Shipping Rust Speed to Python Teams
  4. Observability: Tracing, Metrics and Per-Request Cost Accounting

Module 6: Performance Engineering and Production Hardening

Measuring, optimizing and releasing with confidence.
4 lessons
  1. Profiling Rust AI Workloads: Flamegraphs, perf, Allocation Tracking and Criterion
  2. SIMD, Threading and Allocator Choices for Pre- and Post-Processing
  3. Fuzzing, Miri and Sanitizers for Unsafe and FFI Code
  4. Release Engineering: Static Linking, Container Images and Supply-Chain Checks

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by Hugging Face (Candle, tokenizers, safetensors), Tracel AI (Burn, CubeCL), Microsoft (ONNX Runtime via the ort crate), ggml-org (llama.cpp), mistral.rs, NVIDIA (CUDA, cudarc), PyO3, or Tokio and Axum. Product and company names are trademarks of their respective owners.

More in AI in Software Development

All 19 courses in this track
dev-001 Free intro Available

Your First AI API Call: A One-Hour Hands-On Start

A free, one-hour, hands-on start built on one realistic job: sorting a property manager's 1,150 monthly maintenance requests. You get an API key the safe way, make real calls with curl and an official Python SDK, run them over forty work orders, read stop reasons, usage and errors, estimate the monthly cost, and decide whether the script is ready. Exercises run on your own provider account for well under a dollar.

1 h · 7 lessons Free intro
dev-201 Intermediate Available

Python AI Integration: SDKs, Async, Pydantic, Packaging and Testing

A code-first course on integrating LLMs into Python the way production code needs it: uv projects and secrets, the OpenAI, Anthropic and Google GenAI SDKs side by side, async concurrency and streaming, Pydantic structured outputs and tools, retries, local models, embeddings and RAG, FastAPI and pandas integration, testing and evals with pytest, observability, cost tracking and deployment.

15 h · 19 lessons See price at checkout
dev-202 Intermediate Waitlist

Perl and AI: Modernizing Scripts and Text Pipelines with LLM APIs

Call OpenAI, Anthropic, Gemini and local Ollama models from Perl with real CPAN modules, add streaming, concurrency and validated JSON output to text pipelines, and use AI coding assistants to modernize legacy Perl behind characterization tests.

12 h · 18 lessons Join the waitlist