Checkpointing and Fault Tolerance for Long Training Runs
For engineers who run multi-day or multi-week training jobs and need them to survive node failures, hangs and bad data without losing days of progress.
The curriculum is being prepared and reviewed. Enrollment is not open yet.
About this course
On a large cluster, failure is a matter of when, not if. GPUs throw XID errors, links flap, nodes hang inside a collective, and a single bad batch can spike the loss. What separates a productive run from an expensive one is how much work each failure costs. This course treats that as an engineering problem with a metric, goodput, and a toolkit.
You will start with the failure taxonomy and the Young/Daly reasoning for checkpoint intervals, then define exactly what a resumable checkpoint must contain, including data-loader position and RNG state, and how to reshard it onto a different parallel layout. The fast-checkpointing module covers asynchronous saving with PyTorch Distributed Checkpoint, Megatron-Core distributed checkpointing, local and replicated in-memory checkpoints, and the storage bandwidth they need. Detection covers heartbeats, the NCCL watchdog and PyTorch's flight recorder, DCGM health gates and loss-spike monitoring. Recovery covers Slurm requeue and hot spares, the NVIDIA Resiliency Extension's fault-tolerance and in-process restart features, and replica-level recovery with torchft. Feature maturity is stated as of the review date, since several of these tools are evolving quickly.
What you will be able to do
Classify the failure modes of long training runs and quantify their cost as lost goodput
Choose a checkpoint interval from failure rate and save cost and explain the assumptions behind it
Specify a complete training-state checkpoint and verify that a resumed run is equivalent to an uninterrupted one
Implement fast checkpointing with asynchronous, distributed and in-memory techniques and size the storage bandwidth they need
Detect hangs, stragglers, unhealthy nodes and loss spikes before they waste hours of cluster time
Configure automatic recovery with scheduler requeue, hot spares, in-process restart or elastic replica recovery
Write a resilience runbook and post-incident review process for a multi-week run
Syllabus
6 modules · 20 lessons · about 18 hours
1
Module 1: Failure at Scale
What breaks in long runs, how to measure the damage and how often to checkpoint.3 lessons
Why Long Runs Fail: Hardware, Fabric, Software and Data Faults
Goodput: Accounting for Time Lost to Failures, Restarts and Saves
Choosing a Checkpoint Interval: Young/Daly and Its Assumptions
2
Module 2: What a Checkpoint Must Contain
Capture all the state needed to resume exactly, even on a different layout.3 lessons
Complete Training State: Weights, Optimizer, Scheduler, RNG and Data-Loader Position
Verifying Resume: Bitwise and Statistical Equivalence Tests
Resharding on Load: Changing Parallel Layout or GPU Count Between Runs
3
Module 3: Fast Checkpointing
Shrink the time training waits on a save.4 lessons
Asynchronous Saves with PyTorch DCP: CPU Staging, Pinned Memory and Thread- or Process-Based Writers
Megatron-Core Distributed Checkpointing and Fully Parallel Saves
Local and In-Memory Checkpoints with Peer Replication
Storage Targets: Parallel File Systems, Object Storage and Write-Bandwidth Budgets
4
Module 4: Detecting Failures Early
Find the hang, the sick node or the bad batch before it costs hours.3 lessons
Hangs and Stragglers: Heartbeats, the NCCL Watchdog and the PyTorch Flight Recorder
Node Health Gates: DCGM Diagnostics, XID Errors and Pre-Job Checks
Silent Data Corruption and Loss-Spike Detection
5
Module 5: Recovering Automatically
From whole-job restarts to keeping the job alive while a replica is replaced.4 lessons
Job-Level Restart: Slurm Requeue, Hot Spares and Node Exclusion
In-Job Recovery: torchrun Elastic, NVIDIA Resiliency Extension and In-Process Restart
Replica-Level Fault Tolerance with torchft
Rollback Policies for Loss Spikes and Corrupted Data
6
Module 6: Operating a Resilient Run
Process and evidence that keep the run on track week after week.3 lessons
A Resilience Runbook for a Multi-Week Run
Checkpoint Retention, Validation and Off-Cluster Copies
Post-Incident Reviews That Raise Goodput
Exam and certificate
The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.
Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.
The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by PyTorch, NVIDIA Megatron-Core, NVIDIA Resiliency Extension, NVIDIA DCGM, Microsoft DeepSpeed, torchft, or Slurm. Product and company names are trademarks of their respective owners.
A hands-on introduction to CUDA C++: the execution model, the memory hierarchy, the classic parallel patterns, streams and CUDA Graphs, and profiling with Nsight. It finishes with the kernels deep learning actually needs, written in CUDA, Triton and the tile-based cuTile Python DSL and plugged into PyTorch.
The single-node foundations of fast training: what FP32, TF32, BF16, FP16, FP8, MXFP8 and NVFP4 actually trade away, how to use autocast and FP8 recipes safely, where every gigabyte of GPU memory goes, and the techniques (checkpointing, accumulation, compiled and fused kernels) that make a step fit and run faster.
A practitioner's course on how large-model training is actually distributed: data parallelism and sharding (DDP, ZeRO, FSDP2), tensor, pipeline, context and expert parallelism, how to combine them, and how to launch, checkpoint, measure and debug the result on a real cluster.