Wirewalk Instituteof AI
trn-350 Expert Waitlist

Collective Communication: NCCL and the Network

For ML infrastructure and network engineers who need to understand, measure and tune the collectives that distributed training depends on.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

Distributed training is only as fast as its slowest collective. This course explains what actually happens when a framework calls all-reduce, all-gather, reduce-scatter or all-to-all. You will derive the cost of ring and tree algorithms with an alpha-beta model, learn why nccl-tests reports bus bandwidth rather than algorithm bandwidth, and map each kind of parallelism to the collectives it generates.

You will then open up NCCL: communicators, channels and proxy threads; the Ring, Tree, CollNet, NVLS and PAT algorithms and the Simple, LL and LL128 protocols; and how topology detection builds its graph from PCIe, NVLink and NIC placement. The network modules cover NVLink and NVLink SHARP inside a node or rack-scale domain, GPUDirect RDMA and NIC affinity, then the scale-out fabric from NCCL's side: its IB verbs transport and network plugins, PXN, how adaptive routing, congestion and SHARP in-network reduction show up in collective performance, the RoCE settings NCCL depends on, Spectrum-X, and where the Ultra Ethernet Consortium specification fits. Operating the fabric itself (subnet managers, PFC and DCQCN tuning, switch configuration) is covered in inf-320. The course closes with measurement and tuning, NCCL logging and profiler plugins, device-initiated communication with the NCCL Device API (experimental as of NCCL 2.28) and NVSHMEM, all-to-all kernels for expert parallelism, and a structured approach to debugging collective timeouts. AMD's RCCL is noted where it differs. Version-specific NCCL features are stated as of the review date.

What you will be able to do

  • Derive the latency and bandwidth cost of ring and tree collectives and relate algorithm bandwidth to bus bandwidth
  • Explain how NCCL selects algorithms, protocols and channels from the detected topology
  • Describe how NVLink, NVSwitch, PCIe, GPUDirect RDMA and NIC placement shape intra-node and rack-scale communication
  • Configure NCCL's InfiniBand and RoCE transport settings and attribute collective slowdowns to fabric effects such as congestion, adaptive routing or missing in-network reduction
  • Benchmark collectives with nccl-tests and judge results against what the hardware should deliver
  • Tune and observe NCCL with environment variables, logging and profiler plugins without masking underlying faults
  • Explain device-initiated communication and all-to-all patterns for mixture-of-experts and debug collective timeouts systematically

Syllabus

6 modules · 18 lessons · about 18 hours

Module 1: Collectives from First Principles

What each collective does, what it costs and which training patterns produce it.
3 lessons
  1. The Collective Operations: All-Reduce, All-Gather, Reduce-Scatter, All-to-All and Broadcast
  2. Cost Models: Alpha-Beta Analysis, Algorithm Bandwidth and Bus Bandwidth
  3. From Parallelism to Traffic: Which Training Layouts Generate Which Collectives

Module 2: Inside NCCL

How the library turns one call into coordinated transfers across GPUs.
3 lessons
  1. Communicators, Channels and Proxy Threads: How NCCL Executes a Collective
  2. Algorithms and Protocols: Ring, Tree, CollNet, NVLS and PAT; Simple, LL and LL128
  3. Topology Detection, Graph Search and Custom Topology Files

Module 3: Inside the Node and the Rack

The scale-up fabric and the path from GPU memory to the NIC.
3 lessons
  1. NVLink, NVSwitch and NVLink SHARP Multicast
  2. PCIe Topology, GPUDirect RDMA and GPU-NIC Affinity
  3. Rack-Scale NVLink Domains: Multi-Node NVLink and the IMEX Service

Module 4: The Scale-Out Network as NCCL Sees It

How NCCL drives the inter-node fabric and how fabric behavior surfaces in collective performance.
3 lessons
  1. NCCL's Network Transport: IB Verbs, Net Plugins, Queue Pairs per Connection and PXN
  2. Fabric Effects on Collectives: Adaptive Routing, Congestion and In-Network Reduction Through CollNet
  3. Ethernet Fabrics from NCCL's Side: RoCE Settings, Spectrum-X and Where Ultra Ethernet Fits

Module 5: Measuring, Tuning and Observing

Benchmark what the fabric delivers and see what NCCL is doing in production.
3 lessons
  1. nccl-tests Done Right: Message Sizes, Bus Bandwidth and Expected Results
  2. Tuning Variables and When Not to Touch Them: NCCL_IB_HCA, NCCL_SOCKET_IFNAME, NCCL_ALGO and Others
  3. Observability: NCCL_DEBUG, Profiler Plugins and NCCL Inspector

Module 6: Beyond Host-Launched Collectives

Kernel-initiated communication, expert-parallel all-to-all and debugging failures.
3 lessons
  1. Device-Initiated Communication: The NCCL Device API, NVSHMEM and Symmetric Memory
  2. All-to-All for Mixture-of-Experts: Dispatch and Combine Kernels and Overlap
  3. Debugging Collective Failures: Timeouts, Mismatched Calls and Fabric Errors

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by NVIDIA NCCL, NVIDIA NVLink and NVSwitch, NVIDIA InfiniBand (Quantum, SHARP), NVIDIA Spectrum-X, NVIDIA GPUDirect RDMA, NVIDIA NVSHMEM, AMD RCCL, PyTorch, Ultra Ethernet Consortium, or DeepSeek DeepEP. Product and company names are trademarks of their respective owners.

More in Training at Scale

All 7 courses in this track
trn-201 Intermediate Waitlist

GPU Programming Fundamentals with CUDA

A hands-on introduction to CUDA C++: the execution model, the memory hierarchy, the classic parallel patterns, streams and CUDA Graphs, and profiling with Nsight. It finishes with the kernels deep learning actually needs, written in CUDA, Triton and the tile-based cuTile Python DSL and plugged into PyTorch.

14 h · 17 lessons Join the waitlist
trn-210 Intermediate Waitlist

Mixed Precision, Memory and Training Performance

The single-node foundations of fast training: what FP32, TF32, BF16, FP16, FP8, MXFP8 and NVFP4 actually trade away, how to use autocast and FP8 recipes safely, where every gigabyte of GPU memory goes, and the techniques (checkpointing, accumulation, compiled and fused kernels) that make a step fit and run faster.

14 h · 17 lessons Join the waitlist
trn-310 Expert Available

Distributed Training at Scale: DDP, FSDP, DeepSpeed and Megatron

A practitioner's course on how large-model training is actually distributed: data parallelism and sharding (DDP, ZeRO, FSDP2), tensor, pipeline, context and expert parallelism, how to combine them, and how to launch, checkpoint, measure and debug the result on a real cluster.

20 h · 18 lessons See price at checkout