Wirewalk Instituteof AI
inf-210 Intermediate Waitlist

Linux Administration for AI Clusters

For Linux administrators moving onto GPU clusters who leave able to build, tune and maintain compute nodes that run AI jobs reliably and reproducibly.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

A GPU compute node is a Linux server, but the habits that work for web servers break on it. A routine kernel update can strand the NVIDIA driver; a wrong IOMMU or PCIe ACS setting can quietly halve GPU-to-NIC bandwidth; a user's job can land its memory on the far NUMA node; and a hundred researchers need a hundred conflicting Python and CUDA versions on the same machines. This course teaches the node-level administration that keeps an AI cluster fast and boring.

You start with distribution choice and lifecycle (RHEL 10 and its rebuilds Rocky Linux and AlmaLinux, Ubuntu 26.04 LTS), then stateless and stateful provisioning with Warewulf, NVIDIA Base Command Manager and Ansible. The core modules cover the NVIDIA driver in its packaging variants (open kernel modules, DKMS and precompiled streams), DOCA-OFED for InfiniBand and RoCE, Fabric Manager and persistence; then hardware-aware tuning with hwloc, numactl, tuned profiles, hugepages, IOMMU and ACS for GPUDirect peer-to-peer and RDMA. The later modules cover cgroup v2 resource isolation, SSSD/LDAP identity and Munge, NFS and parallel file system clients, user software environments with Lmod, Spack, EasyBuild, Apptainer and Enroot, and the patching, logging and kdump practices that let you change a fleet safely. Facts are stated as of October 2026.

What you will be able to do

  • Select a Linux distribution and lifecycle for a GPU cluster and justify it against driver, vendor-support and kernel-update constraints
  • Build a reproducible compute node image and provision it with Warewulf, Base Command Manager or Ansible
  • Install and pin the NVIDIA driver, Fabric Manager and DOCA-OFED so that kernel updates do not break GPUs or RDMA
  • Inspect NUMA, PCIe and GPU-NIC topology with hwloc and nvidia-smi topo, and tune BIOS, IOMMU, ACS, hugepages and tuned profiles accordingly
  • Configure cgroup v2, identity (SSSD/LDAP, Munge) and shared file system clients for a multi-user cluster
  • Provide user software environments with Lmod, Spack or EasyBuild and with Apptainer or Enroot containers
  • Plan and execute a rolling patch of a GPU fleet with canaries, health checks and a rollback path

Syllabus

6 modules · 19 lessons · about 14 hours

Module 1: Choosing and Imaging the Node OS

Distribution choice for GPU nodes and the provisioning patterns that make a hundred nodes identical.
3 lessons
  1. RHEL 10, Rocky Linux, AlmaLinux and Ubuntu 26.04 LTS for GPU Nodes
  2. Stateless Versus Stateful Nodes: Warewulf, Base Command Manager and PXE
  3. Golden Images and Configuration Management with Ansible

Module 2: Drivers That Survive Updates

Packaging the GPU and RDMA stacks so kernel and driver changes are deliberate, not accidental.
4 lessons
  1. NVIDIA Driver Packaging: Open Kernel Modules, DKMS and Precompiled Streams
  2. Fabric Manager, Persistence Mode and the Services a GPU Node Needs
  3. DOCA-OFED and the Move Away from MLNX_OFED
  4. Version-Locking Kernels and Drivers Across a Fleet

Module 3: Hardware-Aware Tuning

Making the operating system respect the NUMA, PCIe and GPU topology underneath it.
3 lessons
  1. Reading Topology: hwloc, lstopo, numactl and nvidia-smi topo
  2. BIOS, IOMMU and PCIe ACS Settings for GPUDirect Peer-to-Peer and RDMA
  3. CPU Governors, tuned Profiles, Hugepages and IRQ Affinity

Module 4: A Multi-User Node

Isolation, identity and shared storage so many users can share nodes safely.
3 lessons
  1. cgroup v2 for CPU, Memory and Device Isolation
  2. Identity on Clusters: SSSD, LDAP, Munge and SSH Access Control
  3. Shared File System Clients: NFS, Storage Scale and Lustre Mounts

Module 5: User Software Environments

Delivering conflicting CUDA, MPI and Python stacks to users without breaking the base OS.
3 lessons
  1. Environment Modules with Lmod and Hierarchical Module Trees
  2. Building Scientific Software with Spack and EasyBuild
  3. Containers Without Root: Apptainer, Enroot and Podman

Module 6: Keeping the Fleet Healthy

Logging, crash capture and safe patching for a cluster that cannot all go down at once.
3 lessons
  1. Logs and Crash Capture: journald, rsyslog, kdump and the Kernel Ring Buffer
  2. Rolling Patches with Canaries, Drains and Rollback
  3. Security Baselines for Compute Nodes: SELinux, Firewalls and CIS Benchmarks

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 20 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 40 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by Red Hat, Rocky Linux, AlmaLinux, Canonical, NVIDIA, Ansible, Warewulf, Spack, EasyBuild, Lmod, Apptainer, Enroot, or Podman. Product and company names are trademarks of their respective owners.

More in AI Infrastructure and HPC

All 14 courses in this track
inf-101 Basic Waitlist

AI Infrastructure Explained

A plain-language tour of the stack that runs AI: what a GPU server is, why clusters need special networks and storage, how jobs are scheduled, what an AI data center needs in power and cooling, and how cloud and on-premises options differ. No prior infrastructure background required.

6 h · 15 lessons Join the waitlist
inf-201 Intermediate Available

GPU Infrastructure for AI

The operator's view of data-center GPUs: what the current NVIDIA and AMD systems are, how they are packaged and connected, how the driver, CUDA/ROCm and container stack fit together, how to monitor and triage failing GPUs, how to share and size GPU memory, and what power, cooling and sourcing decisions follow.

12 h · 16 lessons See price at checkout
inf-220 Intermediate Waitlist

Kubernetes for AI Workloads

How to make Kubernetes a good home for GPU work: the GPU and Network Operators, device plugins versus Dynamic Resource Allocation, GPU sharing, RDMA with Multus, batch queueing and gang scheduling with Kueue, JobSet and KAI Scheduler, distributed training and multi-node inference patterns, and the observability and cost controls that keep a shared GPU platform fair.

14 h · 19 lessons Join the waitlist