Wirewalk Instituteof AI
inf-320 Expert Waitlist

High-Performance Networking: InfiniBand, RoCE and Ethernet for AI

For network and HPC engineers who run AI fabrics and leave able to configure, tune and troubleshoot InfiniBand and lossless Ethernet so collective communication runs at line rate.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

A fabric that passes cable checks can still deliver half its bandwidth to a training job. The causes live above the physical layer: a routing engine that concentrates flows, adaptive routing switched off, PFC storms, ECN thresholds copied from a storage network, a NIC on the wrong PCIe root, or NCCL choosing a ring when a tree or NVLink SHARP would win. This course is about those layers. Physical design, optics and switch counts are covered in hw-203 and inf-310; here you operate and tune the fabric.

You start from the traffic itself: RDMA verbs, GPUDirect RDMA and the all-reduce, all-gather and all-to-all patterns that training and mixture-of-experts inference generate. The InfiniBand modules cover the subnet manager and UFM, routing engines (fat-tree, up/down and adaptive routing), partitions and P_Keys, SHARP in-network reduction and Quantum-X800 era features. The Ethernet modules cover RoCEv2 lossless configuration with PFC, ECN and DCQCN, buffer and QoS design, Spectrum-X adaptive routing and congestion control, SONiC-based and merchant-silicon fabrics, and the Ultra Ethernet 1.0 transport. You then tune NCCL and RCCL, isolate tenants, and work through a troubleshooting method using perftest, NCCL tests, ibdiagnet and switch telemetry. Facts are stated as of October 2026.

What you will be able to do

  • Relate collective operations in training and inference to the traffic patterns and bottlenecks they create on a fabric
  • Configure and operate an InfiniBand subnet, including subnet manager placement, routing engine choice, partitions and SHARP
  • Configure a lossless RoCEv2 fabric with PFC, ECN and DCQCN and justify buffer and threshold settings
  • Compare Spectrum-X, merchant-silicon Ethernet with SONiC and Ultra Ethernet transport for a stated AI workload
  • Tune NCCL or RCCL algorithms, protocols and environment variables using measured results rather than folklore
  • Design tenant isolation on InfiniBand and Ethernet fabrics without breaking RDMA performance
  • Diagnose a slow or unstable fabric with perftest, NCCL tests, ibdiagnet, counters and telemetry, and locate the faulty layer
  • Write an acceptance and regression test plan for fabric changes

Syllabus

6 modules · 20 lessons · about 22 hours

Module 1: Traffic First: RDMA and Collectives

What AI jobs actually send over the network and why it stresses fabrics differently from ordinary traffic.
3 lessons
  1. RDMA Verbs, Queue Pairs and GPUDirect RDMA
  2. All-Reduce, All-Gather and All-to-All: Collective Patterns in Training and MoE Inference
  3. Scale-Up NVLink Domains Versus Scale-Out Fabrics: Where Each Collective Runs

Module 2: Operating InfiniBand

The control plane and features that make or break an InfiniBand fabric.
4 lessons
  1. Subnet Managers, OpenSM and UFM: Placement, Failover and Sweeps
  2. Routing Engines: Fat-Tree, Up/Down and Adaptive Routing
  3. Partitions, P_Keys and Quality of Service
  4. SHARP In-Network Reduction: When It Helps and How to Enable It

Module 3: Lossless Ethernet for AI

Making RDMA over Converged Ethernet behave under load.
3 lessons
  1. RoCEv2 Fundamentals and Why Loss Hurts RDMA
  2. PFC, ECN and DCQCN: Configuration and Threshold Tuning
  3. Buffers, QoS Classes and Avoiding PFC Storms and Deadlocks

Module 4: AI Ethernet Platforms and Standards

Vendor AI Ethernet designs and the open standards changing the transport layer.
3 lessons
  1. Spectrum-X: Adaptive Routing, Congestion Control and SuperNIC Integration
  2. Merchant-Silicon Fabrics with SONiC: Broadcom, Arista, Cisco and Juniper Approaches
  3. Ultra Ethernet 1.0: Packet Spraying, New Congestion Control and What Changes for Operators

Module 5: Tuning Collective Communication

Getting the communication library to use the fabric well, and keeping tenants apart.
3 lessons
  1. NCCL Internals for Operators: Rings, Trees, NVLS and Protocols
  2. NCCL and RCCL Environment Tuning with Measured Before-and-After Results
  3. Multi-Tenant Isolation: Partitions, VRFs, VXLAN and RDMA

Module 6: Troubleshooting and Acceptance

A layered method for proving where a fabric problem lives, and tests that stop regressions.
4 lessons
  1. Measuring Correctly: perftest, NCCL Tests and Expected Bus Bandwidth
  2. Reading the Evidence: ibdiagnet, Port Counters, Switch Telemetry and Congestion Signals
  3. Case Walkthroughs: Hypothetical Slow All-Reduce, Flapping Link and Hot Spine
  4. Fabric Change Control and Regression Testing

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by NVIDIA, Broadcom, AMD, Arista, Cisco, Juniper, Ultra Ethernet Consortium, NCCL, RCCL, or SONiC. Product and company names are trademarks of their respective owners.

More in AI Infrastructure and HPC

All 14 courses in this track
inf-101 Basic Waitlist

AI Infrastructure Explained

A plain-language tour of the stack that runs AI: what a GPU server is, why clusters need special networks and storage, how jobs are scheduled, what an AI data center needs in power and cooling, and how cloud and on-premises options differ. No prior infrastructure background required.

6 h · 15 lessons Join the waitlist
inf-201 Intermediate Available

GPU Infrastructure for AI

The operator's view of data-center GPUs: what the current NVIDIA and AMD systems are, how they are packaged and connected, how the driver, CUDA/ROCm and container stack fit together, how to monitor and triage failing GPUs, how to share and size GPU memory, and what power, cooling and sourcing decisions follow.

12 h · 16 lessons See price at checkout
inf-210 Intermediate Waitlist

Linux Administration for AI Clusters

The Linux skills that differ on an AI cluster: choosing and imaging the OS, packaging GPU and RDMA drivers that survive kernel updates, NUMA and PCIe topology tuning, cgroup v2 isolation, shared identity and file systems, scientific software environments with Lmod, Spack and containers, and patching a fleet without breaking jobs.

14 h · 19 lessons Join the waitlist