Wirewalk Instituteof AI
trn-201 Intermediate Waitlist

GPU Programming Fundamentals with CUDA

For Python and C++ developers who want to understand how GPU kernels really run and leave able to write, measure and fix their own.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

Most AI engineers use GPUs through frameworks that hide the hardware. That works until a model is slow, runs out of memory or needs an operation the framework does not fuse. This course gives you the mental model and the hands-on skills underneath: how a kernel launch becomes a grid of blocks scheduled onto streaming multiprocessors, why memory access patterns usually matter more than arithmetic, and how to tell from a profiler which of the two is limiting you.

You will write and progressively optimize real kernels (vector operations, a transpose, a reduction, a scan and a tiled matrix multiply), overlap transfers with compute using streams, cut launch overhead with CUDA Graphs, and find race conditions with Compute Sanitizer. The final module moves to deep-learning kernels: a fused operation in Triton, the same idea in cuTile Python, the Python DSL for NVIDIA's CUDA Tile programming model introduced in CUDA 13.1, and registering a custom operator so PyTorch and torch.compile can call it. Toolkit specifics are stated against CUDA 13.x as of the review date.

What you will be able to do

  • Write, compile and launch CUDA C++ kernels with a launch configuration chosen for the problem size and the GPU
  • Explain how threads, warps, blocks and streaming multiprocessors map onto the hardware and how this affects occupancy
  • Restructure memory accesses for coalescing and use shared memory tiling without bank conflicts
  • Implement reductions, scans and tiled matrix multiplication and explain why vendor libraries such as cuBLAS still outperform them
  • Overlap data transfer and compute with streams and reduce launch overhead with CUDA Graphs
  • Profile kernels with Nsight Systems and Nsight Compute and identify whether a kernel is memory-, compute- or latency-bound
  • Write a fused kernel in Triton or cuTile and expose a custom operator to PyTorch

Syllabus

6 modules · 17 lessons · about 14 hours

Module 1: The CUDA Execution Model

From a host program to a grid of threads running on streaming multiprocessors.
3 lessons
  1. Your First Kernel: nvcc, __global__ and the Launch Configuration
  2. Threads, Warps, Blocks and Streaming Multiprocessors
  3. Explicit cudaMalloc/cudaMemcpy versus Unified Memory

Module 2: The Memory Hierarchy and Access Patterns

Why where data lives and how threads touch it usually decides kernel speed.
3 lessons
  1. Global Memory Coalescing and the Cost of a Strided Load
  2. Shared Memory Tiling and Bank Conflicts: Optimizing a Matrix Transpose
  3. Registers, Occupancy and the Roofline Model

Module 3: Classic Parallel Patterns

The building blocks behind most GPU algorithms, written and optimized step by step.
3 lessons
  1. Parallel Reduction with Warp Shuffles and Cooperative Groups
  2. Prefix Scan and Histograms with Atomics
  3. Tiled Matrix Multiply and Why cuBLAS and Tensor Cores Still Win

Module 4: Concurrency and Correctness

Keep the GPU busy and prove your kernels are correct.
3 lessons
  1. Streams, Events and Overlapping Copies with Compute
  2. CUDA Graphs for Launch-Bound Workloads
  3. Finding Races and Memory Errors with Compute Sanitizer and cuda-gdb

Module 5: Measuring Kernels

Time kernels correctly and read what the profilers are telling you.
2 lessons
  1. Timing Kernels Correctly and Reading an Nsight Systems Timeline
  2. Nsight Compute: Speed of Light, Memory Workload and Warp Stall Reasons

Module 6: Kernels for Deep Learning

Write the fused kernels models need and call them from PyTorch.
3 lessons
  1. Writing a Fused Softmax Kernel in Triton
  2. Tile-Based Kernels with cuTile Python: Tiles, Arrays and Compiler-Managed Tensor Cores
  3. Custom Operators in PyTorch with torch.library and torch.compile

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 20 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 40 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by NVIDIA CUDA, NVIDIA Nsight Systems, NVIDIA Nsight Compute, NVIDIA cuBLAS, NVIDIA CUDA Tile (cuTile Python), OpenAI Triton, or PyTorch. Product and company names are trademarks of their respective owners.

More in Training at Scale

All 7 courses in this track
trn-210 Intermediate Waitlist

Mixed Precision, Memory and Training Performance

The single-node foundations of fast training: what FP32, TF32, BF16, FP16, FP8, MXFP8 and NVFP4 actually trade away, how to use autocast and FP8 recipes safely, where every gigabyte of GPU memory goes, and the techniques (checkpointing, accumulation, compiled and fused kernels) that make a step fit and run faster.

14 h · 17 lessons Join the waitlist
trn-310 Expert Available

Distributed Training at Scale: DDP, FSDP, DeepSpeed and Megatron

A practitioner's course on how large-model training is actually distributed: data parallelism and sharding (DDP, ZeRO, FSDP2), tensor, pipeline, context and expert parallelism, how to combine them, and how to launch, checkpoint, measure and debug the result on a real cluster.

20 h · 18 lessons See price at checkout
trn-320 Expert Waitlist

Scaling Laws and Compute Planning

How to turn the scaling-laws literature into a defensible compute plan: counting training FLOPs, what Kaplan, Chinchilla and their replications actually found, when to deliberately train past compute-optimal, how to fit your own laws from small runs, and how to convert a model and token budget into GPUs, weeks, risk buffers and cost.

16 h · 17 lessons Join the waitlist