Wirewalk Instituteof AI
inf-350 Expert Waitlist

AI Cluster Operations: Monitoring, Health Checks and Incidents

For SREs and cluster operators who run production GPU clusters and leave able to build the telemetry, automated health checks and incident practice that keep GPU availability high.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

In a large training run, a single GPU with a degrading link or a node with a slow file system mount can stall thousands of GPUs, and every hour of that stall is paid for. Operations for AI clusters is therefore less about whether servers are up and more about whether jobs make progress. This course teaches the operational system around the hardware: what to measure, how to detect and remove bad nodes before jobs hit them, how to respond when they do, and how to learn from it. Single-GPU diagnostics and error codes are covered in inf-201; here you run them across a fleet.

You begin by defining SLOs that matter for AI: GPU availability, job goodput, time to detect and time to remove a bad node. You then build a telemetry pipeline with the DCGM exporter, node exporters, fabric and storage collectors, Slurm and Kubernetes metrics, Prometheus or a compatible long-term store, Grafana and log aggregation, and design alerts that page on symptoms. The health modules cover layered checks (boot, prolog, periodic, epilog), Node Health Check, DCGM diagnostics, NCCL and storage probes, and automated drain-and-repair loops, with notes on vendor suites such as NVIDIA Mission Control and Base Command Manager. Further modules cover fleet-scale triage of XID patterns, fabric and file system degradation, checkpoint-restart and job resilience, structured incident response and blameless reviews, and node lifecycle management with spares, RMA and burn-in on return. Facts are stated as of October 2026.

What you will be able to do

  • Define SLOs and SLIs for an AI cluster, including GPU availability, job goodput and bad-node detection time
  • Build a telemetry pipeline that correlates GPU, fabric, storage, scheduler and facility signals for the same job and node
  • Design alerts that page on user-visible symptoms and route cause-level signals to tickets and dashboards
  • Implement layered automated health checks that drain unhealthy nodes before jobs are scheduled on them
  • Triage GPU, fabric and storage degradation across a fleet and decide between reset, reimage, repair and RMA
  • Run a structured incident from detection to post-incident review with clear roles and communication
  • Manage node lifecycle with spares pools, RMA tracking, return-to-service burn-in and failure trend analysis

Syllabus

6 modules · 20 lessons · about 20 hours

Module 1: What Good Looks Like

Reliability targets that reflect whether AI work is actually making progress.
3 lessons
  1. Why AI Cluster Operations Differs: Synchronous Jobs and the Cost of One Bad Node
  2. SLOs and SLIs for GPU Clusters: Availability, Goodput and Time to Remove
  3. Failure Rates at Scale: Reading Published Large-Cluster Reliability Reports

Module 2: Building the Telemetry Pipeline

Collecting and correlating signals from every layer of the cluster.
4 lessons
  1. GPU Metrics with the DCGM Exporter: Which Fields to Collect and Why
  2. Fabric, Storage and Facility Signals: UFM, Switch Telemetry, File System and BMS Metrics
  3. Scheduler-Aware Metrics: Joining Slurm and Kubernetes Job Data to Node Telemetry
  4. Storage and Retention: Prometheus, Long-Term Stores, Logs and OpenTelemetry

Module 3: Alerting and Dashboards

Paging on what matters and giving responders the right view quickly.
3 lessons
  1. Symptom-Based Alerts Versus Cause-Based Signals
  2. Dashboards for Operators, Users and Management
  3. Reducing Alert Noise: Thresholds, Hysteresis and Ownership

Module 4: Automated Health Checks

Detecting bad nodes before jobs do and removing them automatically.
4 lessons
  1. Layered Checks: Boot, Prolog, Periodic and Epilog
  2. Node Health Check, DCGM Diagnostics, NCCL Probes and Storage Probes
  3. Automated Drain, Repair and Return-to-Service Loops
  4. Vendor Operations Suites: Mission Control and Base Command Manager in Practice

Module 5: Fleet-Scale Triage and Job Resilience

Finding patterns across many failures and keeping long jobs alive through them.
3 lessons
  1. Triage at Fleet Scale: XID Patterns, Link Degradation and Slow Nodes
  2. File System and Fabric Degradation: Recognizing Shared-Infrastructure Incidents
  3. Checkpoint-Restart, Hot Spares and Elastic Training

Module 6: Incidents and Lifecycle

Responding well, learning systematically and managing hardware through its life.
3 lessons
  1. Incident Response for Clusters: Roles, Severity, Communication and Change Freezes
  2. Blameless Post-Incident Reviews and Action Tracking
  3. Spares, RMA Workflow, Burn-In on Return and Failure Trend Analysis

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by NVIDIA, Prometheus, Grafana, OpenTelemetry, SchedMD, Kubernetes, PagerDuty, IBM, or Lustre. Product and company names are trademarks of their respective owners.

More in AI Infrastructure and HPC

All 14 courses in this track
inf-101 Basic Waitlist

AI Infrastructure Explained

A plain-language tour of the stack that runs AI: what a GPU server is, why clusters need special networks and storage, how jobs are scheduled, what an AI data center needs in power and cooling, and how cloud and on-premises options differ. No prior infrastructure background required.

6 h · 15 lessons Join the waitlist
inf-201 Intermediate Available

GPU Infrastructure for AI

The operator's view of data-center GPUs: what the current NVIDIA and AMD systems are, how they are packaged and connected, how the driver, CUDA/ROCm and container stack fit together, how to monitor and triage failing GPUs, how to share and size GPU memory, and what power, cooling and sourcing decisions follow.

12 h · 16 lessons See price at checkout
inf-210 Intermediate Waitlist

Linux Administration for AI Clusters

The Linux skills that differ on an AI cluster: choosing and imaging the OS, packaging GPU and RDMA drivers that survive kernel updates, NUMA and PCIe topology tuning, cgroup v2 isolation, shared identity and file systems, scientific software environments with Lmod, Spack and containers, and patching a fleet without breaking jobs.

14 h · 19 lessons Join the waitlist