Wirewalk Instituteof AI
trn-330 Expert Waitlist

Checkpointing and Fault Tolerance for Long Training Runs

For engineers who run multi-day or multi-week training jobs and need them to survive node failures, hangs and bad data without losing days of progress.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

On a large cluster, failure is a matter of when, not if. GPUs throw XID errors, links flap, nodes hang inside a collective, and a single bad batch can spike the loss. What separates a productive run from an expensive one is how much work each failure costs. This course treats that as an engineering problem with a metric, goodput, and a toolkit.

You will start with the failure taxonomy and the Young/Daly reasoning for checkpoint intervals, then define exactly what a resumable checkpoint must contain, including data-loader position and RNG state, and how to reshard it onto a different parallel layout. The fast-checkpointing module covers asynchronous saving with PyTorch Distributed Checkpoint, Megatron-Core distributed checkpointing, local and replicated in-memory checkpoints, and the storage bandwidth they need. Detection covers heartbeats, the NCCL watchdog and PyTorch's flight recorder, DCGM health gates and loss-spike monitoring. Recovery covers Slurm requeue and hot spares, the NVIDIA Resiliency Extension's fault-tolerance and in-process restart features, and replica-level recovery with torchft. Feature maturity is stated as of the review date, since several of these tools are evolving quickly.

What you will be able to do

  • Classify the failure modes of long training runs and quantify their cost as lost goodput
  • Choose a checkpoint interval from failure rate and save cost and explain the assumptions behind it
  • Specify a complete training-state checkpoint and verify that a resumed run is equivalent to an uninterrupted one
  • Implement fast checkpointing with asynchronous, distributed and in-memory techniques and size the storage bandwidth they need
  • Detect hangs, stragglers, unhealthy nodes and loss spikes before they waste hours of cluster time
  • Configure automatic recovery with scheduler requeue, hot spares, in-process restart or elastic replica recovery
  • Write a resilience runbook and post-incident review process for a multi-week run

Syllabus

6 modules · 20 lessons · about 18 hours

Module 1: Failure at Scale

What breaks in long runs, how to measure the damage and how often to checkpoint.
3 lessons
  1. Why Long Runs Fail: Hardware, Fabric, Software and Data Faults
  2. Goodput: Accounting for Time Lost to Failures, Restarts and Saves
  3. Choosing a Checkpoint Interval: Young/Daly and Its Assumptions

Module 2: What a Checkpoint Must Contain

Capture all the state needed to resume exactly, even on a different layout.
3 lessons
  1. Complete Training State: Weights, Optimizer, Scheduler, RNG and Data-Loader Position
  2. Verifying Resume: Bitwise and Statistical Equivalence Tests
  3. Resharding on Load: Changing Parallel Layout or GPU Count Between Runs

Module 3: Fast Checkpointing

Shrink the time training waits on a save.
4 lessons
  1. Asynchronous Saves with PyTorch DCP: CPU Staging, Pinned Memory and Thread- or Process-Based Writers
  2. Megatron-Core Distributed Checkpointing and Fully Parallel Saves
  3. Local and In-Memory Checkpoints with Peer Replication
  4. Storage Targets: Parallel File Systems, Object Storage and Write-Bandwidth Budgets

Module 4: Detecting Failures Early

Find the hang, the sick node or the bad batch before it costs hours.
3 lessons
  1. Hangs and Stragglers: Heartbeats, the NCCL Watchdog and the PyTorch Flight Recorder
  2. Node Health Gates: DCGM Diagnostics, XID Errors and Pre-Job Checks
  3. Silent Data Corruption and Loss-Spike Detection

Module 5: Recovering Automatically

From whole-job restarts to keeping the job alive while a replica is replaced.
4 lessons
  1. Job-Level Restart: Slurm Requeue, Hot Spares and Node Exclusion
  2. In-Job Recovery: torchrun Elastic, NVIDIA Resiliency Extension and In-Process Restart
  3. Replica-Level Fault Tolerance with torchft
  4. Rollback Policies for Loss Spikes and Corrupted Data

Module 6: Operating a Resilient Run

Process and evidence that keep the run on track week after week.
3 lessons
  1. A Resilience Runbook for a Multi-Week Run
  2. Checkpoint Retention, Validation and Off-Cluster Copies
  3. Post-Incident Reviews That Raise Goodput

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by PyTorch, NVIDIA Megatron-Core, NVIDIA Resiliency Extension, NVIDIA DCGM, Microsoft DeepSpeed, torchft, or Slurm. Product and company names are trademarks of their respective owners.

More in Training at Scale

All 7 courses in this track
trn-201 Intermediate Waitlist

GPU Programming Fundamentals with CUDA

A hands-on introduction to CUDA C++: the execution model, the memory hierarchy, the classic parallel patterns, streams and CUDA Graphs, and profiling with Nsight. It finishes with the kernels deep learning actually needs, written in CUDA, Triton and the tile-based cuTile Python DSL and plugged into PyTorch.

14 h · 17 lessons Join the waitlist
trn-210 Intermediate Waitlist

Mixed Precision, Memory and Training Performance

The single-node foundations of fast training: what FP32, TF32, BF16, FP16, FP8, MXFP8 and NVFP4 actually trade away, how to use autocast and FP8 recipes safely, where every gigabyte of GPU memory goes, and the techniques (checkpointing, accumulation, compiled and fused kernels) that make a step fit and run faster.

14 h · 17 lessons Join the waitlist
trn-310 Expert Available

Distributed Training at Scale: DDP, FSDP, DeepSpeed and Megatron

A practitioner's course on how large-model training is actually distributed: data parallelism and sharding (DDP, ZeRO, FSDP2), tensor, pipeline, context and expert parallelism, how to combine them, and how to launch, checkpoint, measure and debug the result on a real cluster.

20 h · 18 lessons See price at checkout