For ML and performance engineers who need to find out why a training job is slow and make it measurably faster without changing its results.
The curriculum is being prepared and reviewed. Enrollment is not open yet.
About this course
Performance work fails in two common ways: tuning by folklore, and declaring victory on a noisy measurement. This course teaches a repeatable method instead. You will build a benchmark harness that controls warm-up, variance and loss equivalence, then learn to read every layer of evidence: PyTorch Profiler and Kineto traces in Perfetto, Nsight Systems timelines annotated with NVTX, Nsight Compute kernel analysis, Holistic Trace Analysis across many ranks, and DCGM profiling metrics from the fleet.
Each fix is tied to a diagnosis. Host-side problems (slow input pipelines, launch-bound steps, hidden synchronisations) are fixed with prefetching, CUDA Graphs and removing syncs. GPU-side problems are fixed with torch.compile, better attention and GEMM shapes, and, when justified, custom kernels in Triton, CUTLASS or the CuTe DSL. Communication problems are exposed in distributed traces and addressed with bucketing, prefetch and overlap. The course closes with a worked optimization campaign on a hypothetical model, continuous performance testing and how to report results honestly.
What you will be able to do
Define and measure throughput, step time, MFU and goodput with a reproducible benchmark harness
Classify a training step's bottleneck as compute-, memory-, launch-, input- or communication-bound from trace evidence
Capture and interpret traces with the PyTorch Profiler, Nsight Systems, Nsight Compute and Holistic Trace Analysis
Remove host-side bottlenecks including slow data loading, launch overhead and hidden device synchronisations
Improve GPU-side efficiency with torch.compile, attention and GEMM shape choices and justified custom kernels
Quantify exposed communication and increase compute-communication overlap
Run and report an optimization campaign that verifies numerical equivalence and guards against regressions
Syllabus
6 modules · 19 lessons · about 18 hours
1
Module 1: A Method for Performance Work
Metrics, bottleneck classes and a harness you can trust.3 lessons
Metrics That Matter: Tokens per Second, Step Time, MFU and Goodput
Roofline Thinking for Training: Compute, Memory, Launch, Input and Communication Bounds
Building a Reproducible Benchmark Harness with Warm-Up, Variance and Loss Checks
2
Module 2: Reading the Evidence
The profilers, what each one is for and how to read their output.4 lessons
PyTorch Profiler and Kineto Traces in Perfetto
Nsight Systems with NVTX Ranges for Multi-GPU Timelines
Nsight Compute for the Kernels That Dominate the Step
Holistic Trace Analysis: Temporal Breakdown, Idle Time and Overlap Across Ranks
3
Module 3: Fixing the Host Side
When the GPU is waiting on the CPU, the data or a synchronisation.3 lessons
Diagnosing Input-Bound Steps: Data-Loader Wait in Traces, Tokenization Off the Critical Path and Prefetch Depth
Launch-Bound Steps: CPU Overhead and CUDA Graphs
Hidden Synchronisations: .item(), Host Copies and Allocator Stalls
4
Module 4: Fixing the GPU Side
Make the kernels themselves do more useful work per second.3 lessons
torch.compile and Inductor: Reading the Generated Kernels
GEMM and Attention Shapes: Tensor Core Alignment, Wave Quantization and Proving Which Backend Ran
When a Custom Kernel Is Worth It: Triton, CUTLASS and the CuTe DSL
5
Module 5: Fixing Communication
Find the communication the GPU actually waits on and hide it.3 lessons
Measuring Exposed Communication in Distributed Traces
Overlap Techniques: DDP Bucketing, FSDP2 Prefetch and Async Tensor Parallelism
Slow Ranks: Per-Rank Timelines and DCGM Profiling Metrics
6
Module 6: Running an Optimization Campaign
Chain fixes into a measurable result and keep it.3 lessons
A Worked Campaign on a Hypothetical 8B-Parameter Model, One Change at a Time
Regression Guards: Continuous Performance Testing
Reporting Results Honestly: Variance, Baselines and Like-for-Like Comparisons
Exam and certificate
The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.
Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.
The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by PyTorch, NVIDIA Nsight Systems, NVIDIA Nsight Compute, NVIDIA NVTX, NVIDIA DCGM, NVIDIA CUTLASS, Holistic Trace Analysis, Perfetto, OpenAI Triton, or FlashAttention. Product and company names are trademarks of their respective owners.
A hands-on introduction to CUDA C++: the execution model, the memory hierarchy, the classic parallel patterns, streams and CUDA Graphs, and profiling with Nsight. It finishes with the kernels deep learning actually needs, written in CUDA, Triton and the tile-based cuTile Python DSL and plugged into PyTorch.
The single-node foundations of fast training: what FP32, TF32, BF16, FP16, FP8, MXFP8 and NVFP4 actually trade away, how to use autocast and FP8 recipes safely, where every gigabyte of GPU memory goes, and the techniques (checkpointing, accumulation, compiled and fused kernels) that make a step fit and run faster.
A practitioner's course on how large-model training is actually distributed: data parallelism and sharding (DDP, ZeRO, FSDP2), tensor, pipeline, context and expert parallelism, how to combine them, and how to launch, checkpoint, measure and debug the result on a real cluster.