Distributed Training at Scale: DDP, FSDP, DeepSpeed and Megatron
For ML and infrastructure engineers who must take a training job from one node to hundreds of GPUs and keep it fast, correct and running.
Level
Expert
Study time
20 hours
Lessons
18 in 6 modules
Exam
25 questions
About this course
Distributed training is a chain of trade-offs between memory, compute and communication. This course teaches you to reason about each link with numbers rather than folklore. You will account for every byte a GPU holds during training, compute what each collective costs on NVLink and InfiniBand, and see exactly what DDP buckets, ZeRO stages, FSDP2 sharding, Megatron tensor and pipeline parallelism, context parallelism and expert parallelism change in that budget.
The second half is operational. You will choose a parallel layout for a large model with a worked calculation, launch it with torchrun and Slurm, tune NCCL for the fabric, write sharded checkpoints you can actually resume from, measure model FLOPs utilization and scaling efficiency honestly, and diagnose the failures that dominate real runs: hangs, NCCL timeouts, desynchronized ranks and stragglers. Code, launch scripts and configuration are shown against current PyTorch, DeepSpeed and Megatron-Core APIs as of the review date.
What you will be able to do
Calculate per-GPU memory for weights, gradients, optimizer state and activations under DDP, ZeRO 1/2/3 and FSDP2
Estimate the communication cost of all-reduce, all-gather and reduce-scatter and relate it to NVLink and InfiniBand bandwidth
Configure DDP, DeepSpeed ZeRO and PyTorch FSDP2 correctly, including mixed precision, offload and gradient accumulation
Explain tensor, sequence, pipeline, context and expert parallelism and compute pipeline bubble overhead for a given schedule
Choose and justify a combined parallel layout for a large model on a given cluster
Launch multi-node jobs with torchrun and Slurm, tune NCCL for the fabric, and write resumable sharded checkpoints
Measure MFU and scaling efficiency and diagnose hangs, NCCL timeouts and stragglers
Syllabus
6 modules · 18 lessons · about 20 hours
1
Module 1: Why Distribute and What It Costs
The compute, memory and communication arithmetic that every later decision rests on.3 lessons
Why Distribute: Compute, Memory and Time-to-Train
Per-GPU Memory Accounting: Weights, Gradients, Optimizer State and Activations
Collectives and Their Cost: All-Reduce, All-Gather and Reduce-Scatter
2
Module 2: Data Parallelism and Sharding
Replicate the model and split the data, then shard the training state to fit bigger models.3 lessons
DDP Internals: Gradient Buckets, Overlap and no_sync
ZeRO Stages 1, 2 and 3 with DeepSpeed
FSDP2 in PyTorch: fully_shard, DeviceMesh, HSDP and Mixed Precision
3
Module 3: Model Parallelism
Split the model itself: within layers, across layers, along the sequence and across experts.3 lessons
Tensor Parallelism and Sequence Parallelism
Pipeline Parallelism: GPipe, 1F1B, Interleaved Schedules and the Bubble
Context Parallelism and Expert Parallelism for Long Sequences and MoE
4
Module 4: Composing Parallelism and Saving Memory
Combine the dimensions, pick a layout with numbers, and trade compute for memory and precision.3 lessons
Combining Dimensions in Megatron-Core: TP, PP, CP, EP and DP
Choosing a Layout: A Worked 70B Example on 256 GPUs
Precision and Recompute: bf16, FP8 and Activation Checkpointing
5
Module 5: Running on a Real Cluster
Launchers, the network under NCCL, and checkpoints that survive a restart.3 lessons
Launchers: torchrun, Slurm srun and the DeepSpeed Launcher
NCCL and the Fabric: NVLink, InfiniBand, RoCE and Topology
Sharded Checkpoints: PyTorch DCP, DeepSpeed and Megatron
6
Module 6: Performance and Failure at Scale
Measure efficiency honestly and diagnose the failures that dominate large runs.3 lessons
Measuring MFU and Scaling Efficiency
Failure Modes: Hangs, NCCL Timeouts and Desynchronized Ranks
Stragglers, Bad Nodes and a Pre-Flight Checklist
Exam and certificate
The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.
Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.
The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by PyTorch, Microsoft DeepSpeed, NVIDIA Megatron-Core, NVIDIA NCCL, NVIDIA Transformer Engine, NVIDIA InfiniBand, or Slurm. Product and company names are trademarks of their respective owners.
A hands-on introduction to CUDA C++: the execution model, the memory hierarchy, the classic parallel patterns, streams and CUDA Graphs, and profiling with Nsight. It finishes with the kernels deep learning actually needs, written in CUDA, Triton and the tile-based cuTile Python DSL and plugged into PyTorch.
The single-node foundations of fast training: what FP32, TF32, BF16, FP16, FP8, MXFP8 and NVFP4 actually trade away, how to use autocast and FP8 recipes safely, where every gigabyte of GPU memory goes, and the techniques (checkpointing, accumulation, compiled and fused kernels) that make a step fit and run faster.
How to turn the scaling-laws literature into a defensible compute plan: counting training FLOPs, what Kaplan, Chinchilla and their replications actually found, when to deliberately train past compute-optimal, how to fit your own laws from small runs, and how to convert a model and token budget into GPUs, weeks, risk buffers and cost.