A fabric that passes cable checks can still deliver half its bandwidth to a training job. The causes live above the physical layer: a routing engine that concentrates flows, adaptive routing switched off, PFC storms, ECN thresholds copied from a storage network, a NIC on the wrong PCIe root, or NCCL choosing a ring when a tree or NVLink SHARP would win. This course is about those layers. Physical design, optics and switch counts are covered in hw-203 and inf-310; here you operate and tune the fabric.
You start from the traffic itself: RDMA verbs, GPUDirect RDMA and the all-reduce, all-gather and all-to-all patterns that training and mixture-of-experts inference generate. The InfiniBand modules cover the subnet manager and UFM, routing engines (fat-tree, up/down and adaptive routing), partitions and P_Keys, SHARP in-network reduction and Quantum-X800 era features. The Ethernet modules cover RoCEv2 lossless configuration with PFC, ECN and DCQCN, buffer and QoS design, Spectrum-X adaptive routing and congestion control, SONiC-based and merchant-silicon fabrics, and the Ultra Ethernet 1.0 transport. You then tune NCCL and RCCL, isolate tenants, and work through a troubleshooting method using perftest, NCCL tests, ibdiagnet and switch telemetry. Facts are stated as of October 2026.