Wirewalk Instituteof AI
inf-330 Expert Waitlist

Parallel Storage for AI: Storage Scale, Lustre, WEKA and VAST

For storage and HPC engineers who run AI data platforms and leave able to configure, tune, benchmark and troubleshoot the four leading parallel storage systems for training and inference.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

Storage decides whether expensive GPUs compute or wait. AI workloads mix millions of small random reads during data loading, bursts of very large sequential writes during checkpoints, metadata storms from tokenized shards, and increasingly the offload of inference KV cache to shared storage. Each of the leading parallel systems handles that mix differently, and each has tunables that matter and many that do not. Cluster-level storage tiering and sizing are introduced in inf-310; this course goes inside the file systems themselves.

You start with the I/O physics of AI workloads, then work through each platform as an administrator: IBM Storage Scale (NSDs, Erasure Code Edition and Storage Scale System, filesets, the policy engine, AFM and the data acceleration tier introduced in Storage Scale 6.0); Lustre (MDTs and OSTs, LNet, DNE, Progressive File Layouts, Data-on-MDT and the features of the 2.16 and 2.17 releases, including DDN EXAScaler packaging); WEKA NeuralMesh; and VAST Data's DASE architecture and its AI-oriented data services. Cross-platform modules cover GPUDirect Storage and RDMA clients, client-side tuning, benchmarking with fio, IOR, mdtest and the MLPerf Storage benchmark while defeating cache effects, and operations: tiering to object, quotas, snapshots, replication, upgrades and failure handling. Facts are stated as of October 2026.

What you will be able to do

  • Characterize the I/O patterns of data loading, checkpointing and KV-cache offload and translate them into storage requirements
  • Administer and tune IBM Storage Scale, including filesets, pools, the policy engine, AFM and client parameters that matter
  • Administer and tune Lustre, including LNet, DNE, Progressive File Layouts and Data-on-MDT
  • Explain the architectures of WEKA NeuralMesh and VAST Data and the operational consequences of each
  • Enable and verify GPUDirect Storage and RDMA clients and measure their benefit
  • Run storage benchmarks that defeat cache and concurrency traps and produce numbers that predict job behavior
  • Operate a parallel file system day to day: tiering, quotas, snapshots, replication, upgrades and failure recovery
  • Compare the four platforms for a stated AI workload and defend the choice

Syllabus

6 modules · 20 lessons · about 22 hours

Module 1: AI I/O Patterns

What training and inference do to storage, measured from the client side.
3 lessons
  1. Data Loading: Small Random Reads, Shards and Metadata Load
  2. Checkpointing: Synchronous, Asynchronous and Sharded Writes
  3. Inference Storage: Model Loading, KV-Cache Offload and RAG Corpora

Module 2: IBM Storage Scale in Depth

Running Storage Scale (GPFS) for AI, from NSDs to the policy engine.
4 lessons
  1. Architecture: NSD Servers, Erasure Code Edition, Storage Scale System and Remote Clusters
  2. Filesets, Storage Pools and the Policy Engine with mmapplypolicy
  3. AFM, the Data Acceleration Tier and Caching Patterns for AI
  4. Tuning That Matters: pagepool, Prefetch, Block Size and Client Settings

Module 3: Lustre in Depth

Running Lustre and its commercial distributions for AI clusters.
4 lessons
  1. Architecture: MGS, MDTs, OSTs and LNet Networking
  2. Layouts: Striping, Progressive File Layouts and Data-on-MDT
  3. Scaling Metadata with DNE, and What Changed in Lustre 2.16 and 2.17
  4. EXAScaler and Managed Lustre Services: Operational Differences

Module 4: WEKA and VAST

Two flash-native designs with very different architectures and operating models.
3 lessons
  1. WEKA NeuralMesh: Distributed Metadata, Client Modes and Converged Deployments
  2. VAST Data: Disaggregated Shared-Everything, Similarity Reduction and Multiprotocol Access
  3. Choosing Among the Four: A Structured Comparison for AI Workloads

Module 5: Getting Data to GPUs Fast

RDMA clients, GPUDirect Storage and benchmarks that predict real job performance.
3 lessons
  1. GPUDirect Storage and RDMA Clients: Setup, Verification and Limits
  2. Benchmarking Without Fooling Yourself: fio, IOR, mdtest and MLPerf Storage
  3. Reading Results: Cache Effects, Under-Driven Concurrency and Tail Latency

Module 6: Operating Storage for AI

The day-2 practices that keep data safe and performance predictable.
3 lessons
  1. Tiering to Object Storage and Data Lifecycle Policies
  2. Quotas, Snapshots and Replication Between Sites
  3. Upgrades, Rebuilds and Failure Handling Without Losing the File System

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by IBM, DDN, WEKA, VAST Data, NVIDIA, or Lustre. Product and company names are trademarks of their respective owners.

More in AI Infrastructure and HPC

All 14 courses in this track
inf-101 Basic Waitlist

AI Infrastructure Explained

A plain-language tour of the stack that runs AI: what a GPU server is, why clusters need special networks and storage, how jobs are scheduled, what an AI data center needs in power and cooling, and how cloud and on-premises options differ. No prior infrastructure background required.

6 h · 15 lessons Join the waitlist
inf-201 Intermediate Available

GPU Infrastructure for AI

The operator's view of data-center GPUs: what the current NVIDIA and AMD systems are, how they are packaged and connected, how the driver, CUDA/ROCm and container stack fit together, how to monitor and triage failing GPUs, how to share and size GPU memory, and what power, cooling and sourcing decisions follow.

12 h · 16 lessons See price at checkout
inf-210 Intermediate Waitlist

Linux Administration for AI Clusters

The Linux skills that differ on an AI cluster: choosing and imaging the OS, packaging GPU and RDMA drivers that survive kernel updates, NUMA and PCIe topology tuning, cgroup v2 isolation, shared identity and file systems, scientific software environments with Lmod, Spack and containers, and patching a fleet without breaking jobs.

14 h · 19 lessons Join the waitlist