Distributed training is only as fast as its slowest collective. This course explains what actually happens when a framework calls all-reduce, all-gather, reduce-scatter or all-to-all. You will derive the cost of ring and tree algorithms with an alpha-beta model, learn why nccl-tests reports bus bandwidth rather than algorithm bandwidth, and map each kind of parallelism to the collectives it generates.
You will then open up NCCL: communicators, channels and proxy threads; the Ring, Tree, CollNet, NVLS and PAT algorithms and the Simple, LL and LL128 protocols; and how topology detection builds its graph from PCIe, NVLink and NIC placement. The network modules cover NVLink and NVLink SHARP inside a node or rack-scale domain, GPUDirect RDMA and NIC affinity, then the scale-out fabric from NCCL's side: its IB verbs transport and network plugins, PXN, how adaptive routing, congestion and SHARP in-network reduction show up in collective performance, the RoCE settings NCCL depends on, Spectrum-X, and where the Ultra Ethernet Consortium specification fits. Operating the fabric itself (subnet managers, PFC and DCQCN tuning, switch configuration) is covered in inf-320. The course closes with measurement and tuning, NCCL logging and profiler plugins, device-initiated communication with the NCCL Device API (experimental as of NCCL 2.28) and NVSHMEM, all-to-all kernels for expert parallelism, and a structured approach to debugging collective timeouts. AMD's RCCL is noted where it differs. Version-specific NCCL features are stated as of the review date.