Wirewalk Instituteof AI
trn-340 Expert Waitlist

Profiling and Optimizing Training Throughput

For ML and performance engineers who need to find out why a training job is slow and make it measurably faster without changing its results.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

Performance work fails in two common ways: tuning by folklore, and declaring victory on a noisy measurement. This course teaches a repeatable method instead. You will build a benchmark harness that controls warm-up, variance and loss equivalence, then learn to read every layer of evidence: PyTorch Profiler and Kineto traces in Perfetto, Nsight Systems timelines annotated with NVTX, Nsight Compute kernel analysis, Holistic Trace Analysis across many ranks, and DCGM profiling metrics from the fleet.

Each fix is tied to a diagnosis. Host-side problems (slow input pipelines, launch-bound steps, hidden synchronisations) are fixed with prefetching, CUDA Graphs and removing syncs. GPU-side problems are fixed with torch.compile, better attention and GEMM shapes, and, when justified, custom kernels in Triton, CUTLASS or the CuTe DSL. Communication problems are exposed in distributed traces and addressed with bucketing, prefetch and overlap. The course closes with a worked optimization campaign on a hypothetical model, continuous performance testing and how to report results honestly.

What you will be able to do

  • Define and measure throughput, step time, MFU and goodput with a reproducible benchmark harness
  • Classify a training step's bottleneck as compute-, memory-, launch-, input- or communication-bound from trace evidence
  • Capture and interpret traces with the PyTorch Profiler, Nsight Systems, Nsight Compute and Holistic Trace Analysis
  • Remove host-side bottlenecks including slow data loading, launch overhead and hidden device synchronisations
  • Improve GPU-side efficiency with torch.compile, attention and GEMM shape choices and justified custom kernels
  • Quantify exposed communication and increase compute-communication overlap
  • Run and report an optimization campaign that verifies numerical equivalence and guards against regressions

Syllabus

6 modules · 19 lessons · about 18 hours

Module 1: A Method for Performance Work

Metrics, bottleneck classes and a harness you can trust.
3 lessons
  1. Metrics That Matter: Tokens per Second, Step Time, MFU and Goodput
  2. Roofline Thinking for Training: Compute, Memory, Launch, Input and Communication Bounds
  3. Building a Reproducible Benchmark Harness with Warm-Up, Variance and Loss Checks

Module 2: Reading the Evidence

The profilers, what each one is for and how to read their output.
4 lessons
  1. PyTorch Profiler and Kineto Traces in Perfetto
  2. Nsight Systems with NVTX Ranges for Multi-GPU Timelines
  3. Nsight Compute for the Kernels That Dominate the Step
  4. Holistic Trace Analysis: Temporal Breakdown, Idle Time and Overlap Across Ranks

Module 3: Fixing the Host Side

When the GPU is waiting on the CPU, the data or a synchronisation.
3 lessons
  1. Diagnosing Input-Bound Steps: Data-Loader Wait in Traces, Tokenization Off the Critical Path and Prefetch Depth
  2. Launch-Bound Steps: CPU Overhead and CUDA Graphs
  3. Hidden Synchronisations: .item(), Host Copies and Allocator Stalls

Module 4: Fixing the GPU Side

Make the kernels themselves do more useful work per second.
3 lessons
  1. torch.compile and Inductor: Reading the Generated Kernels
  2. GEMM and Attention Shapes: Tensor Core Alignment, Wave Quantization and Proving Which Backend Ran
  3. When a Custom Kernel Is Worth It: Triton, CUTLASS and the CuTe DSL

Module 5: Fixing Communication

Find the communication the GPU actually waits on and hide it.
3 lessons
  1. Measuring Exposed Communication in Distributed Traces
  2. Overlap Techniques: DDP Bucketing, FSDP2 Prefetch and Async Tensor Parallelism
  3. Slow Ranks: Per-Rank Timelines and DCGM Profiling Metrics

Module 6: Running an Optimization Campaign

Chain fixes into a measurable result and keep it.
3 lessons
  1. A Worked Campaign on a Hypothetical 8B-Parameter Model, One Change at a Time
  2. Regression Guards: Continuous Performance Testing
  3. Reporting Results Honestly: Variance, Baselines and Like-for-Like Comparisons

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by PyTorch, NVIDIA Nsight Systems, NVIDIA Nsight Compute, NVIDIA NVTX, NVIDIA DCGM, NVIDIA CUTLASS, Holistic Trace Analysis, Perfetto, OpenAI Triton, or FlashAttention. Product and company names are trademarks of their respective owners.

More in Training at Scale

All 7 courses in this track
trn-201 Intermediate Waitlist

GPU Programming Fundamentals with CUDA

A hands-on introduction to CUDA C++: the execution model, the memory hierarchy, the classic parallel patterns, streams and CUDA Graphs, and profiling with Nsight. It finishes with the kernels deep learning actually needs, written in CUDA, Triton and the tile-based cuTile Python DSL and plugged into PyTorch.

14 h · 17 lessons Join the waitlist
trn-210 Intermediate Waitlist

Mixed Precision, Memory and Training Performance

The single-node foundations of fast training: what FP32, TF32, BF16, FP16, FP8, MXFP8 and NVFP4 actually trade away, how to use autocast and FP8 recipes safely, where every gigabyte of GPU memory goes, and the techniques (checkpointing, accumulation, compiled and fused kernels) that make a step fit and run faster.

14 h · 17 lessons Join the waitlist
trn-310 Expert Available

Distributed Training at Scale: DDP, FSDP, DeepSpeed and Megatron

A practitioner's course on how large-model training is actually distributed: data parallelism and sharding (DDP, ZeRO, FSDP2), tensor, pipeline, context and expert parallelism, how to combine them, and how to launch, checkpoint, measure and debug the result on a real cluster.

20 h · 18 lessons See price at checkout