Wirewalk Instituteof AI
trn-210 Intermediate Waitlist

Mixed Precision, Memory and Training Performance

For ML engineers who train models on one GPU or one node and want them to fit in memory, stay numerically stable and run fast.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

Before a job is worth distributing, it should be efficient on one GPU. This course teaches the three levers that decide that: numeric precision, memory and per-step speed. You will learn the range and precision of each training format, why BF16 replaced FP16 loss scaling for most work, how FP8 training works with per-tensor and block scaling, and what the Blackwell-generation MXFP8 and NVFP4 formats change. You will apply them with torch.autocast, Transformer Engine and torchao and learn to recognize the loss curves that signal a precision problem.

The memory half uses PyTorch's own tooling (the caching allocator statistics and memory snapshots) to show where memory goes, then fits larger models with activation checkpointing, gradient accumulation and memory-light optimizers. The speed half covers torch.compile, scaled dot-product attention backends including FlashAttention and FlexAttention, fused kernels and losses, and the data loader. Multi-GPU sharding and parallelism are covered in trn-310; deep profiling methodology in trn-340.

What you will be able to do

  • Compare FP32, TF32, BF16, FP16, FP8, MXFP8 and NVFP4 by range, precision and hardware support and choose a format for a given job
  • Configure torch.autocast, gradient scaling and FP8 training recipes and diagnose precision-related divergence
  • Read CUDA caching allocator statistics and memory snapshots to explain where GPU memory is used
  • Fit a larger model or batch with activation checkpointing, gradient accumulation and memory-light optimizers
  • Speed up a training step with torch.compile, fused attention backends and fused kernels
  • Remove data-loading bottlenecks with workers, pinned memory and sequence packing
  • Run a disciplined single-GPU tuning loop that changes one variable at a time and verifies loss equivalence

Syllabus

6 modules · 17 lessons · about 14 hours

Module 1: Number Formats for Training

What each floating-point format can represent and what Tensor Cores do with it.
3 lessons
  1. FP32, TF32, FP16 and BF16: Range, Precision and Tensor Core Throughput
  2. FP8, MXFP8 and NVFP4: Scaling Factors and Block-Scaled Formats
  3. Where Precision Breaks: Underflow, Overflow, Accumulation Error and Loss Spikes

Module 2: Mixed Precision in Practice

Apply lower precision with the tools PyTorch and NVIDIA provide, safely.
3 lessons
  1. torch.autocast and GradScaler: What Runs in Which Precision
  2. Master Weights, Optimizer State Precision and Stochastic Rounding
  3. FP8 Training Recipes with Transformer Engine and torchao

Module 3: Where GPU Memory Goes

Measure memory instead of guessing.
3 lessons
  1. The CUDA Caching Allocator: Allocated, Reserved and Fragmentation
  2. Memory Snapshots with torch.cuda.memory._record_memory_history
  3. Activation Memory in Transformers: Batch, Sequence Length and Attention

Module 4: Fitting Bigger Models and Batches

Trade compute or convergence behavior for memory, knowing the cost of each trade.
3 lessons
  1. Activation Checkpointing: Full, Selective and torch.utils.checkpoint
  2. Gradient Accumulation and Effective Batch Size
  3. Memory-Light Optimizers: 8-bit Adam, Adafactor and CPU Offload

Module 5: Making Each Step Faster

Fewer, bigger, fused kernels and attention that does not materialize the score matrix.
3 lessons
  1. torch.compile: Modes, Graph Breaks and Recompilations
  2. Attention Backends: SDPA, FlashAttention and FlexAttention
  3. Fused Kernels and Losses: Liger Kernel and Chunked Cross-Entropy

Module 6: Input Pipeline and the Tuning Loop

Keep the GPU fed and tune without fooling yourself.
2 lessons
  1. DataLoader Workers, Pinned Memory and Sequence Packing
  2. A Single-GPU Tuning Loop: Baseline, One Change, Re-Measure, Check the Loss

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 20 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 40 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by PyTorch, NVIDIA Transformer Engine, PyTorch torchao, NVIDIA Tensor Cores, FlashAttention, bitsandbytes, or Liger Kernel. Product and company names are trademarks of their respective owners.

More in Training at Scale

All 7 courses in this track
trn-201 Intermediate Waitlist

GPU Programming Fundamentals with CUDA

A hands-on introduction to CUDA C++: the execution model, the memory hierarchy, the classic parallel patterns, streams and CUDA Graphs, and profiling with Nsight. It finishes with the kernels deep learning actually needs, written in CUDA, Triton and the tile-based cuTile Python DSL and plugged into PyTorch.

14 h · 17 lessons Join the waitlist
trn-310 Expert Available

Distributed Training at Scale: DDP, FSDP, DeepSpeed and Megatron

A practitioner's course on how large-model training is actually distributed: data parallelism and sharding (DDP, ZeRO, FSDP2), tensor, pipeline, context and expert parallelism, how to combine them, and how to launch, checkpoint, measure and debug the result on a real cluster.

20 h · 18 lessons See price at checkout
trn-320 Expert Waitlist

Scaling Laws and Compute Planning

How to turn the scaling-laws literature into a defensible compute plan: counting training FLOPs, what Kaplan, Chinchilla and their replications actually found, when to deliberately train past compute-optimal, how to fit your own laws from small runs, and how to convert a model and token budget into GPUs, weeks, risk buffers and cost.

16 h · 17 lessons Join the waitlist