In a large training run, a single GPU with a degrading link or a node with a slow file system mount can stall thousands of GPUs, and every hour of that stall is paid for. Operations for AI clusters is therefore less about whether servers are up and more about whether jobs make progress. This course teaches the operational system around the hardware: what to measure, how to detect and remove bad nodes before jobs hit them, how to respond when they do, and how to learn from it. Single-GPU diagnostics and error codes are covered in inf-201; here you run them across a fleet.
You begin by defining SLOs that matter for AI: GPU availability, job goodput, time to detect and time to remove a bad node. You then build a telemetry pipeline with the DCGM exporter, node exporters, fabric and storage collectors, Slurm and Kubernetes metrics, Prometheus or a compatible long-term store, Grafana and log aggregation, and design alerts that page on symptoms. The health modules cover layered checks (boot, prolog, periodic, epilog), Node Health Check, DCGM diagnostics, NCCL and storage probes, and automated drain-and-repair loops, with notes on vendor suites such as NVIDIA Mission Control and Base Command Manager. Further modules cover fleet-scale triage of XID patterns, fabric and file system degradation, checkpoint-restart and job resilience, structured incident response and blameless reviews, and node lifecycle management with spares, RMA and burn-in on return. Facts are stated as of October 2026.