Wirewalk Instituteof AI
inf-230 Intermediate Waitlist

Slurm for AI and HPC

For cluster administrators and power users who leave able to configure, tune and troubleshoot Slurm for GPU-heavy AI and HPC workloads.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

Slurm schedules a majority of the world's largest AI and HPC systems, and since NVIDIA acquired SchedMD in December 2025 it has remained open source on a six-month major release cadence (25.11 and 26.05 are the current lines as of October 2026). Yet many clusters still run defaults that waste GPUs: jobs that can see GPUs they were not allocated, eight-GPU jobs spread across switches, priority rules nobody can explain, and node health problems discovered by users rather than by the scheduler.

This course works through a Slurm deployment as an administrator would. You configure slurmctld, slurmd, slurmdbd and slurmrestd; describe GPUs with gres.conf and NVML autodetection; confine jobs with cgroup v2; and describe the network with the tree and block topology plugins so jobs land inside a switch or an NVLink domain. You then design fair-share, QOS and preemption policies, run containers with Pyxis and Enroot or Apptainer, launch multi-node PyTorch with srun and PMIx, and use prolog, epilog and HealthCheckProgram hooks. The course closes with accounting and reporting with sacct and sreport, troubleshooting stuck jobs and drained nodes, rolling upgrades, and where Slurm and Kubernetes meet, including Slinky. Facts are stated as of October 2026.

What you will be able to do

  • Explain the roles of slurmctld, slurmd, slurmdbd and slurmrestd and design a highly available control plane
  • Configure GPU GRES, cgroup v2 confinement and select/cons_tres so jobs receive exactly the GPUs, CPUs and memory they request
  • Describe network and NVLink topology with the tree and block plugins and verify placement of multi-node jobs
  • Design partitions, QOS, fair-share and preemption policies for a stated mix of research and production users
  • Run containerized multi-node training with Pyxis, Enroot and PMIx and diagnose launch failures
  • Use prolog, epilog and HealthCheckProgram to keep unhealthy GPU nodes out of service automatically
  • Report utilization with sacct and sreport and plan a rolling Slurm upgrade

Syllabus

6 modules · 19 lessons · about 14 hours

Module 1: Slurm Architecture and First Configuration

The daemons, the configuration files and a control plane that survives failure.
3 lessons
  1. Daemons and Data Flow: slurmctld, slurmd, slurmdbd and slurmrestd
  2. Writing slurm.conf: Nodes, Partitions and select/cons_tres
  3. Control Plane High Availability, Munge and the Accounting Database

Module 2: GPUs, Confinement and Topology

Making Slurm allocate GPUs precisely and place jobs where the network is fastest.
3 lessons
  1. gres.conf, NVML Autodetection and GPU Binding Options
  2. cgroup v2 Confinement: ConstrainDevices, Cores and Memory
  3. Topology-Aware Scheduling: topology/tree and topology/block for NVLink Racks

Module 3: Policy: Who Runs Next

Priority, fairness and preemption policies that users can understand and administrators can defend.
3 lessons
  1. The Multifactor Priority Plugin and Fair-Share Trees
  2. QOS, Limits and TRES Accounting
  3. Backfill and Preemption: Keeping GPUs Busy Without Starving Big Jobs

Module 4: Running AI Jobs Well

Job scripts, containers and multi-node launch for distributed training.
3 lessons
  1. sbatch, srun and salloc for GPU Jobs: Arrays, Dependencies and Heterogeneous Jobs
  2. Containers with Pyxis and Enroot, and with Apptainer
  3. Multi-Node PyTorch: srun, PMIx, torchrun and NCCL Environment

Module 5: Automation and Health

Hooks and checks that keep bad nodes away from jobs, and APIs that let other systems drive Slurm.
3 lessons
  1. Prolog, Epilog and HealthCheckProgram with Node Health Check
  2. Job Submit Plugins and Lua Policy Enforcement
  3. Automating with slurmrestd and the REST API

Module 6: Day-2 Administration

Reporting, troubleshooting, upgrades and living alongside Kubernetes.
4 lessons
  1. Accounting and Reporting with sacct, sreport and sshare
  2. Troubleshooting Pending Jobs, Drained Nodes and Launch Failures
  3. Rolling Upgrades and Version Compatibility Rules
  4. Slurm and Kubernetes Together: Slinky and Hybrid Patterns

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 20 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 40 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by SchedMD, NVIDIA, Slurm, PMIx, Open MPI, Enroot, Pyxis, Apptainer, or Kubernetes. Product and company names are trademarks of their respective owners.

More in AI Infrastructure and HPC

All 14 courses in this track
inf-101 Basic Waitlist

AI Infrastructure Explained

A plain-language tour of the stack that runs AI: what a GPU server is, why clusters need special networks and storage, how jobs are scheduled, what an AI data center needs in power and cooling, and how cloud and on-premises options differ. No prior infrastructure background required.

6 h · 15 lessons Join the waitlist
inf-201 Intermediate Available

GPU Infrastructure for AI

The operator's view of data-center GPUs: what the current NVIDIA and AMD systems are, how they are packaged and connected, how the driver, CUDA/ROCm and container stack fit together, how to monitor and triage failing GPUs, how to share and size GPU memory, and what power, cooling and sourcing decisions follow.

12 h · 16 lessons See price at checkout
inf-210 Intermediate Waitlist

Linux Administration for AI Clusters

The Linux skills that differ on an AI cluster: choosing and imaging the OS, packaging GPU and RDMA drivers that survive kernel updates, NUMA and PCIe topology tuning, cgroup v2 isolation, shared identity and file systems, scientific software environments with Lmod, Spack and containers, and patching a fleet without breaking jobs.

14 h · 19 lessons Join the waitlist