Wirewalk Instituteof AI
trn-310 Expert AvailableFlagship

Distributed Training at Scale: DDP, FSDP, DeepSpeed and Megatron

For ML and infrastructure engineers who must take a training job from one node to hundreds of GPUs and keep it fast, correct and running.

Level
Expert
Study time
20 hours
Lessons
18 in 6 modules
Exam
25 questions

About this course

Distributed training is a chain of trade-offs between memory, compute and communication. This course teaches you to reason about each link with numbers rather than folklore. You will account for every byte a GPU holds during training, compute what each collective costs on NVLink and InfiniBand, and see exactly what DDP buckets, ZeRO stages, FSDP2 sharding, Megatron tensor and pipeline parallelism, context parallelism and expert parallelism change in that budget.

The second half is operational. You will choose a parallel layout for a large model with a worked calculation, launch it with torchrun and Slurm, tune NCCL for the fabric, write sharded checkpoints you can actually resume from, measure model FLOPs utilization and scaling efficiency honestly, and diagnose the failures that dominate real runs: hangs, NCCL timeouts, desynchronized ranks and stragglers. Code, launch scripts and configuration are shown against current PyTorch, DeepSpeed and Megatron-Core APIs as of the review date.

What you will be able to do

  • Calculate per-GPU memory for weights, gradients, optimizer state and activations under DDP, ZeRO 1/2/3 and FSDP2
  • Estimate the communication cost of all-reduce, all-gather and reduce-scatter and relate it to NVLink and InfiniBand bandwidth
  • Configure DDP, DeepSpeed ZeRO and PyTorch FSDP2 correctly, including mixed precision, offload and gradient accumulation
  • Explain tensor, sequence, pipeline, context and expert parallelism and compute pipeline bubble overhead for a given schedule
  • Choose and justify a combined parallel layout for a large model on a given cluster
  • Launch multi-node jobs with torchrun and Slurm, tune NCCL for the fabric, and write resumable sharded checkpoints
  • Measure MFU and scaling efficiency and diagnose hangs, NCCL timeouts and stragglers

Syllabus

6 modules · 18 lessons · about 20 hours

Module 1: Why Distribute and What It Costs

The compute, memory and communication arithmetic that every later decision rests on.
3 lessons
  1. Why Distribute: Compute, Memory and Time-to-Train
  2. Per-GPU Memory Accounting: Weights, Gradients, Optimizer State and Activations
  3. Collectives and Their Cost: All-Reduce, All-Gather and Reduce-Scatter

Module 2: Data Parallelism and Sharding

Replicate the model and split the data, then shard the training state to fit bigger models.
3 lessons
  1. DDP Internals: Gradient Buckets, Overlap and no_sync
  2. ZeRO Stages 1, 2 and 3 with DeepSpeed
  3. FSDP2 in PyTorch: fully_shard, DeviceMesh, HSDP and Mixed Precision

Module 3: Model Parallelism

Split the model itself: within layers, across layers, along the sequence and across experts.
3 lessons
  1. Tensor Parallelism and Sequence Parallelism
  2. Pipeline Parallelism: GPipe, 1F1B, Interleaved Schedules and the Bubble
  3. Context Parallelism and Expert Parallelism for Long Sequences and MoE

Module 4: Composing Parallelism and Saving Memory

Combine the dimensions, pick a layout with numbers, and trade compute for memory and precision.
3 lessons
  1. Combining Dimensions in Megatron-Core: TP, PP, CP, EP and DP
  2. Choosing a Layout: A Worked 70B Example on 256 GPUs
  3. Precision and Recompute: bf16, FP8 and Activation Checkpointing

Module 5: Running on a Real Cluster

Launchers, the network under NCCL, and checkpoints that survive a restart.
3 lessons
  1. Launchers: torchrun, Slurm srun and the DeepSpeed Launcher
  2. NCCL and the Fabric: NVLink, InfiniBand, RoCE and Topology
  3. Sharded Checkpoints: PyTorch DCP, DeepSpeed and Megatron

Module 6: Performance and Failure at Scale

Measure efficiency honestly and diagnose the failures that dominate large runs.
3 lessons
  1. Measuring MFU and Scaling Efficiency
  2. Failure Modes: Hangs, NCCL Timeouts and Desynchronized Ranks
  3. Stragglers, Bad Nodes and a Pre-Flight Checklist

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by PyTorch, Microsoft DeepSpeed, NVIDIA Megatron-Core, NVIDIA NCCL, NVIDIA Transformer Engine, NVIDIA InfiniBand, or Slurm. Product and company names are trademarks of their respective owners.

More in Training at Scale

All 7 courses in this track
trn-201 Intermediate Waitlist

GPU Programming Fundamentals with CUDA

A hands-on introduction to CUDA C++: the execution model, the memory hierarchy, the classic parallel patterns, streams and CUDA Graphs, and profiling with Nsight. It finishes with the kernels deep learning actually needs, written in CUDA, Triton and the tile-based cuTile Python DSL and plugged into PyTorch.

14 h · 17 lessons Join the waitlist
trn-210 Intermediate Waitlist

Mixed Precision, Memory and Training Performance

The single-node foundations of fast training: what FP32, TF32, BF16, FP16, FP8, MXFP8 and NVFP4 actually trade away, how to use autocast and FP8 recipes safely, where every gigabyte of GPU memory goes, and the techniques (checkpointing, accumulation, compiled and fused kernels) that make a step fit and run faster.

14 h · 17 lessons Join the waitlist
trn-320 Expert Waitlist

Scaling Laws and Compute Planning

How to turn the scaling-laws literature into a defensible compute plan: counting training FLOPs, what Kaplan, Chinchilla and their replications actually found, when to deliberately train past compute-optimal, how to fit your own laws from small runs, and how to convert a model and token budget into GPUs, weeks, risk buffers and cost.

16 h · 17 lessons Join the waitlist