For ML engineers who train models on one GPU or one node and want them to fit in memory, stay numerically stable and run fast.
The curriculum is being prepared and reviewed. Enrollment is not open yet.
About this course
Before a job is worth distributing, it should be efficient on one GPU. This course teaches the three levers that decide that: numeric precision, memory and per-step speed. You will learn the range and precision of each training format, why BF16 replaced FP16 loss scaling for most work, how FP8 training works with per-tensor and block scaling, and what the Blackwell-generation MXFP8 and NVFP4 formats change. You will apply them with torch.autocast, Transformer Engine and torchao and learn to recognize the loss curves that signal a precision problem.
The memory half uses PyTorch's own tooling (the caching allocator statistics and memory snapshots) to show where memory goes, then fits larger models with activation checkpointing, gradient accumulation and memory-light optimizers. The speed half covers torch.compile, scaled dot-product attention backends including FlashAttention and FlexAttention, fused kernels and losses, and the data loader. Multi-GPU sharding and parallelism are covered in trn-310; deep profiling methodology in trn-340.
What you will be able to do
Compare FP32, TF32, BF16, FP16, FP8, MXFP8 and NVFP4 by range, precision and hardware support and choose a format for a given job
Configure torch.autocast, gradient scaling and FP8 training recipes and diagnose precision-related divergence
Read CUDA caching allocator statistics and memory snapshots to explain where GPU memory is used
Fit a larger model or batch with activation checkpointing, gradient accumulation and memory-light optimizers
Speed up a training step with torch.compile, fused attention backends and fused kernels
Remove data-loading bottlenecks with workers, pinned memory and sequence packing
Run a disciplined single-GPU tuning loop that changes one variable at a time and verifies loss equivalence
Syllabus
6 modules · 17 lessons · about 14 hours
1
Module 1: Number Formats for Training
What each floating-point format can represent and what Tensor Cores do with it.3 lessons
FP32, TF32, FP16 and BF16: Range, Precision and Tensor Core Throughput
FP8, MXFP8 and NVFP4: Scaling Factors and Block-Scaled Formats
Where Precision Breaks: Underflow, Overflow, Accumulation Error and Loss Spikes
2
Module 2: Mixed Precision in Practice
Apply lower precision with the tools PyTorch and NVIDIA provide, safely.3 lessons
torch.autocast and GradScaler: What Runs in Which Precision
Master Weights, Optimizer State Precision and Stochastic Rounding
FP8 Training Recipes with Transformer Engine and torchao
3
Module 3: Where GPU Memory Goes
Measure memory instead of guessing.3 lessons
The CUDA Caching Allocator: Allocated, Reserved and Fragmentation
Memory Snapshots with torch.cuda.memory._record_memory_history
Activation Memory in Transformers: Batch, Sequence Length and Attention
4
Module 4: Fitting Bigger Models and Batches
Trade compute or convergence behavior for memory, knowing the cost of each trade.3 lessons
Activation Checkpointing: Full, Selective and torch.utils.checkpoint
Gradient Accumulation and Effective Batch Size
Memory-Light Optimizers: 8-bit Adam, Adafactor and CPU Offload
5
Module 5: Making Each Step Faster
Fewer, bigger, fused kernels and attention that does not materialize the score matrix.3 lessons
torch.compile: Modes, Graph Breaks and Recompilations
Attention Backends: SDPA, FlashAttention and FlexAttention
Fused Kernels and Losses: Liger Kernel and Chunked Cross-Entropy
6
Module 6: Input Pipeline and the Tuning Loop
Keep the GPU fed and tune without fooling yourself.2 lessons
DataLoader Workers, Pinned Memory and Sequence Packing
A Single-GPU Tuning Loop: Baseline, One Change, Re-Measure, Check the Loss
Exam and certificate
The exam unlocks once you have completed every lesson. It draws 20 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 40 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.
Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.
The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by PyTorch, NVIDIA Transformer Engine, PyTorch torchao, NVIDIA Tensor Cores, FlashAttention, bitsandbytes, or Liger Kernel. Product and company names are trademarks of their respective owners.
A hands-on introduction to CUDA C++: the execution model, the memory hierarchy, the classic parallel patterns, streams and CUDA Graphs, and profiling with Nsight. It finishes with the kernels deep learning actually needs, written in CUDA, Triton and the tile-based cuTile Python DSL and plugged into PyTorch.
A practitioner's course on how large-model training is actually distributed: data parallelism and sharding (DDP, ZeRO, FSDP2), tensor, pipeline, context and expert parallelism, how to combine them, and how to launch, checkpoint, measure and debug the result on a real cluster.
How to turn the scaling-laws literature into a defensible compute plan: counting training FLOPs, what Kaplan, Chinchilla and their replications actually found, when to deliberately train past compute-optimal, how to fit your own laws from small runs, and how to convert a model and token budget into GPUs, weeks, risk buffers and cost.