Wirewalk Instituteof AI
hw-205 Intermediate Waitlist

Memory for AI: HBM, DDR5, CXL and Capacity Planning

For hardware and infrastructure engineers who need to understand every memory tier in an AI system, from HBM stacks to DDR5, LPDDR5X and CXL, and plan capacity and bandwidth for real models.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

Most AI workloads are bounded by memory capacity or memory bandwidth long before they run out of arithmetic. This course explains the memory technologies in AI systems from the DRAM cell up, so engineers can predict what fits, how fast it moves and what it costs. It starts with DRAM organization (banks, ranks, channels, refresh and timing) and why bandwidth, latency and capacity trade against one another.

It then covers each tier in turn. HBM: stacks, TSVs, base dies, 2.5D packaging, the HBM3E and HBM4 generations (JEDEC JESD270-4) and what HBM4E and custom base dies change. Host memory: DDR5 RDIMM speeds and channel counts, MRDIMM, and the bandwidth per core of current Xeon and EPYC platforms. LPDDR5X and SOCAMM modules in Grace and Vera systems, and the coherent CPU-GPU memory they enable. CXL: Type 3 memory expansion, CXL 2.0, 3.x and the CXL 4.0 specification released in November 2025, pooling, switching and the latency cost. Reliability: ECC, on-die ECC, chipkill-class protection, row remapping and error telemetry. The course closes with capacity planning arithmetic for training and inference, including KV cache growth, offload tiers and a worked memory plan. The course ends with an automated exam and a Wirewalk Institute certificate of completion and assessment; it is not a vendor certification.

What you will be able to do

  • Explain how DRAM organization, timing and refresh determine bandwidth, latency and capacity
  • Describe HBM stack architecture and packaging and compare HBM3E and HBM4 on bandwidth, capacity and interface width
  • Calculate host memory bandwidth for a DDR5 RDIMM or MRDIMM configuration and judge whether it is balanced for the core count
  • Explain the role of LPDDR5X and SOCAMM modules in coherent CPU-GPU systems such as Grace and Vera
  • Evaluate CXL memory expansion and pooling for a workload, including latency, bandwidth and software support
  • Interpret memory error telemetry and choose reliability settings appropriate to a large fleet
  • Size memory for weights, KV cache, activations and optimizer state, and design an offload plan across HBM, host DRAM, CXL and NVMe

Syllabus

6 modules · 19 lessons · about 12 hours

Module 1: DRAM Fundamentals

How DRAM is organized and why bandwidth, latency and capacity trade off.
3 lessons
  1. Cells, Banks, Ranks and Channels
  2. Timing, Refresh and the Bandwidth-Latency-Capacity Triangle
  3. The Memory Wall in AI: Arithmetic Intensity and Bytes per FLOP

Module 2: High-Bandwidth Memory

HBM stacks, packaging and generations, and the supply realities that shape accelerator roadmaps.
3 lessons
  1. HBM Anatomy: TSVs, Base Dies and 2.5D Packaging
  2. HBM3E and HBM4: Interface Width, Stack Height and Bandwidth per Stack
  3. HBM4E, Custom Base Dies and HBM Supply Constraints

Module 3: Host and Coherent Memory

DDR5, MRDIMM, LPDDR5X and SOCAMM, and how CPUs and GPUs share memory.
4 lessons
  1. DDR5 RDIMM: Speeds, Channels and Population Rules
  2. MRDIMM: How Multiplexed Ranks Raise Bandwidth
  3. LPDDR5X and SOCAMM in Grace and Vera Systems
  4. Coherent CPU-GPU Memory: NVLink-C2C and Unified Addressing

Module 4: CXL Memory

Memory expansion, pooling and sharing over CXL, and where it pays off.
3 lessons
  1. CXL Protocols and Device Types: From 2.0 to 4.0
  2. Type 3 Expansion and Pooling: Latency, Bandwidth and Tiering
  3. CXL for AI: KV Cache, Embeddings and Realistic Use Cases

Module 5: Memory Reliability

ECC schemes, error behavior at scale, and the telemetry that predicts failures.
3 lessons
  1. ECC, On-Die ECC and Chipkill-Class Protection
  2. HBM Errors, Row Remapping and Page Retirement
  3. Reading Memory Error Telemetry Across a Fleet

Module 6: Capacity Planning

Worked arithmetic for training and inference memory, and multi-tier offload design.
3 lessons
  1. Training Memory: Weights, Gradients, Optimizer State and Activations
  2. Inference Memory: KV Cache Growth, Batching and Quantization
  3. Worked Plan: Tiering Across HBM, Host DRAM, CXL and NVMe

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 20 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 40 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by SK hynix, Samsung, Micron, JEDEC, CXL Consortium, NVIDIA, AMD, Intel, or Astera Labs. Product and company names are trademarks of their respective owners.

hw-001 Free intro Available

Inside an AI Data Center: GPUs, Networks and Storage in One Hour

A plain-English, one-hour walk through an AI data center: what a model asks of hardware, what is inside a GPU server, how GPUs talk over NVLink, InfiniBand and Ethernet, where the data lives, and why AI racks need so much power and cooling.

1 h · 6 lessons Free intro
hw-101 Basic Available

AI Hardware Fundamentals: GPUs, CPUs, Memory, Networking and Storage

A plain-language tour of the hardware that runs AI: why AI is limited by compute, memory bandwidth and data movement; what GPUs, HBM and precision formats are; how servers, networks, storage, power and cooling fit together; and how to size a simple inference or training system and weigh cloud against buying.

8 h · 20 lessons See price at checkout
hw-201 Intermediate Available

GPU Hardware Deep Dive: Architecture, Memory and Interconnects

An engineer's tour of the modern data center GPU: how SMs and CUs, warps and wavefronts, tensor and matrix cores and the on-chip memory hierarchy work; what FP64 through FP4 and sparsity really buy you; how HBM, PCIe, NVLink, Infinity Fabric, UALink and CXL move data; and how power, cooling and reliability features shape real deployments. Reviewed against vendor sources as of October 2026.

14 h · 21 lessons See price at checkout