Wirewalk Instituteof AI

trn  Track

Courses for GPU programming and distributed AI training

GPU programming, distributed training, communication and throughput at cluster scale.

7 courses1 available now2 Intermediate5 Expert Filter in the catalog

Choose an AI learning path for your work

Focus on the bottleneck you need to understand before selecting an advanced training topic. The catalog covers GPU programming, precision and memory, distributed training, collective communication and profiling. Compute planning and checkpointing provide related paths for longer runs. Compare prerequisites with your programming and systems background, and read the syllabus to identify the methods addressed. Check each course listing for availability and study requirements before planning a technical training sequence.

Who these courses are for

  • Distributed training engineers
  • GPU performance engineers
  • Research computing specialists

Learning questions to explore

  • Compare memory and precision tradeoffs in training
  • Explore profiling methods for throughput bottlenecks
  • Plan study of checkpointing and collective communication

Find your starting point

Build your grounding with AI foundations, then check prerequisites for this track’s available courses.

Available courses are open for enrollment. Waitlisted courses are still being prepared or reviewed.

Compare this track in the course catalog

Related learning: Model Development · AI Infrastructure and HPC · AI Hardware

Intermediate

For practitioners who use or build with AI. · 2 courses
trn-201 Intermediate Waitlist

GPU Programming Fundamentals with CUDA

A hands-on introduction to CUDA C++: the execution model, the memory hierarchy, the classic parallel patterns, streams and CUDA Graphs, and profiling with Nsight. It finishes with the kernels deep learning actually needs, written in CUDA, Triton and the tile-based cuTile Python DSL and plugged into PyTorch.

14 h · 17 lessons Join the waitlist
trn-210 Intermediate Waitlist

Mixed Precision, Memory and Training Performance

The single-node foundations of fast training: what FP32, TF32, BF16, FP16, FP8, MXFP8 and NVFP4 actually trade away, how to use autocast and FP8 recipes safely, where every gigabyte of GPU memory goes, and the techniques (checkpointing, accumulation, compiled and fused kernels) that make a step fit and run faster.

14 h · 17 lessons Join the waitlist

Expert

Deep technical work for specialists. · 5 courses
trn-310 Expert Available

Distributed Training at Scale: DDP, FSDP, DeepSpeed and Megatron

A practitioner's course on how large-model training is actually distributed: data parallelism and sharding (DDP, ZeRO, FSDP2), tensor, pipeline, context and expert parallelism, how to combine them, and how to launch, checkpoint, measure and debug the result on a real cluster.

20 h · 18 lessons See price at checkout
trn-320 Expert Waitlist

Scaling Laws and Compute Planning

How to turn the scaling-laws literature into a defensible compute plan: counting training FLOPs, what Kaplan, Chinchilla and their replications actually found, when to deliberately train past compute-optimal, how to fit your own laws from small runs, and how to convert a model and token budget into GPUs, weeks, risk buffers and cost.

16 h · 17 lessons Join the waitlist
trn-330 Expert Waitlist

Checkpointing and Fault Tolerance for Long Training Runs

An engineering course on keeping long training runs productive: measuring goodput, choosing checkpoint intervals, capturing complete and reshardable training state, saving fast with asynchronous and in-memory checkpoints, detecting hangs, bad nodes and silent corruption, and recovering automatically through job restarts, in-process restarts and elastic training.

18 h · 20 lessons Join the waitlist
trn-340 Expert Waitlist

Profiling and Optimizing Training Throughput

A methodical approach to training performance: define the right metrics, classify a step as compute-, memory-, launch-, input- or communication-bound, capture and read traces with the PyTorch Profiler, Nsight Systems, Nsight Compute and Holistic Trace Analysis, then fix the bottleneck on the host, the GPU or the network and prove the gain is real.

18 h · 19 lessons Join the waitlist
trn-350 Expert Waitlist

Collective Communication: NCCL and the Network

A deep course on how collectives move data across GPUs: the algorithms and cost models, NCCL's internals (channels, algorithms, protocols and topology search), NVLink and rack-scale NVLink domains, InfiniBand and RoCE scale-out networks, rigorous measurement with nccl-tests and NCCL observability, and newer device-initiated and all-to-all patterns used by mixture-of-experts training.

18 h · 18 lessons Join the waitlist

Certification programs using this track

Certified AI Infrastructure Engineer

Designs and operates GPU clusters for training and inference.

  1. GPU Infrastructure for AI
  2. HPC Cluster Design for AI: Compute, Fabric and Storage
  3. Distributed Training at Scale: DDP, FSDP, DeepSpeed and Megatron
  4. LLM Inference and Serving: vLLM, SGLang, TensorRT-LLM and Triton
4 courses · 74 h See price at checkout

Certified Model Development Engineer

Trains, fine-tunes and evaluates models.

  1. Transformers from the Ground Up
  2. Fine-Tuning LLMs: Full, LoRA and QLoRA
  3. Model Evaluation and Benchmarking
  4. Distributed Training at Scale: DDP, FSDP, DeepSpeed and Megatron
4 courses · 74 h See price at checkout