Most AI workloads are bounded by memory capacity or memory bandwidth long before they run out of arithmetic. This course explains the memory technologies in AI systems from the DRAM cell up, so engineers can predict what fits, how fast it moves and what it costs. It starts with DRAM organization (banks, ranks, channels, refresh and timing) and why bandwidth, latency and capacity trade against one another.
It then covers each tier in turn. HBM: stacks, TSVs, base dies, 2.5D packaging, the HBM3E and HBM4 generations (JEDEC JESD270-4) and what HBM4E and custom base dies change. Host memory: DDR5 RDIMM speeds and channel counts, MRDIMM, and the bandwidth per core of current Xeon and EPYC platforms. LPDDR5X and SOCAMM modules in Grace and Vera systems, and the coherent CPU-GPU memory they enable. CXL: Type 3 memory expansion, CXL 2.0, 3.x and the CXL 4.0 specification released in November 2025, pooling, switching and the latency cost. Reliability: ECC, on-die ECC, chipkill-class protection, row remapping and error telemetry. The course closes with capacity planning arithmetic for training and inference, including KV cache growth, offload tiers and a worked memory plan. The course ends with an automated exam and a Wirewalk Institute certificate of completion and assessment; it is not a vendor certification.