For Python and C++ developers who want to understand how GPU kernels really run and leave able to write, measure and fix their own.
The curriculum is being prepared and reviewed. Enrollment is not open yet.
About this course
Most AI engineers use GPUs through frameworks that hide the hardware. That works until a model is slow, runs out of memory or needs an operation the framework does not fuse. This course gives you the mental model and the hands-on skills underneath: how a kernel launch becomes a grid of blocks scheduled onto streaming multiprocessors, why memory access patterns usually matter more than arithmetic, and how to tell from a profiler which of the two is limiting you.
You will write and progressively optimize real kernels (vector operations, a transpose, a reduction, a scan and a tiled matrix multiply), overlap transfers with compute using streams, cut launch overhead with CUDA Graphs, and find race conditions with Compute Sanitizer. The final module moves to deep-learning kernels: a fused operation in Triton, the same idea in cuTile Python, the Python DSL for NVIDIA's CUDA Tile programming model introduced in CUDA 13.1, and registering a custom operator so PyTorch and torch.compile can call it. Toolkit specifics are stated against CUDA 13.x as of the review date.
What you will be able to do
Write, compile and launch CUDA C++ kernels with a launch configuration chosen for the problem size and the GPU
Explain how threads, warps, blocks and streaming multiprocessors map onto the hardware and how this affects occupancy
Restructure memory accesses for coalescing and use shared memory tiling without bank conflicts
Implement reductions, scans and tiled matrix multiplication and explain why vendor libraries such as cuBLAS still outperform them
Overlap data transfer and compute with streams and reduce launch overhead with CUDA Graphs
Profile kernels with Nsight Systems and Nsight Compute and identify whether a kernel is memory-, compute- or latency-bound
Write a fused kernel in Triton or cuTile and expose a custom operator to PyTorch
Syllabus
6 modules · 17 lessons · about 14 hours
1
Module 1: The CUDA Execution Model
From a host program to a grid of threads running on streaming multiprocessors.3 lessons
Your First Kernel: nvcc, __global__ and the Launch Configuration
Threads, Warps, Blocks and Streaming Multiprocessors
Explicit cudaMalloc/cudaMemcpy versus Unified Memory
2
Module 2: The Memory Hierarchy and Access Patterns
Why where data lives and how threads touch it usually decides kernel speed.3 lessons
Global Memory Coalescing and the Cost of a Strided Load
Shared Memory Tiling and Bank Conflicts: Optimizing a Matrix Transpose
Registers, Occupancy and the Roofline Model
3
Module 3: Classic Parallel Patterns
The building blocks behind most GPU algorithms, written and optimized step by step.3 lessons
Parallel Reduction with Warp Shuffles and Cooperative Groups
Prefix Scan and Histograms with Atomics
Tiled Matrix Multiply and Why cuBLAS and Tensor Cores Still Win
4
Module 4: Concurrency and Correctness
Keep the GPU busy and prove your kernels are correct.3 lessons
Streams, Events and Overlapping Copies with Compute
CUDA Graphs for Launch-Bound Workloads
Finding Races and Memory Errors with Compute Sanitizer and cuda-gdb
5
Module 5: Measuring Kernels
Time kernels correctly and read what the profilers are telling you.2 lessons
Timing Kernels Correctly and Reading an Nsight Systems Timeline
Nsight Compute: Speed of Light, Memory Workload and Warp Stall Reasons
6
Module 6: Kernels for Deep Learning
Write the fused kernels models need and call them from PyTorch.3 lessons
Writing a Fused Softmax Kernel in Triton
Tile-Based Kernels with cuTile Python: Tiles, Arrays and Compiler-Managed Tensor Cores
Custom Operators in PyTorch with torch.library and torch.compile
Exam and certificate
The exam unlocks once you have completed every lesson. It draws 20 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 40 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.
Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.
The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by NVIDIA CUDA, NVIDIA Nsight Systems, NVIDIA Nsight Compute, NVIDIA cuBLAS, NVIDIA CUDA Tile (cuTile Python), OpenAI Triton, or PyTorch. Product and company names are trademarks of their respective owners.
The single-node foundations of fast training: what FP32, TF32, BF16, FP16, FP8, MXFP8 and NVFP4 actually trade away, how to use autocast and FP8 recipes safely, where every gigabyte of GPU memory goes, and the techniques (checkpointing, accumulation, compiled and fused kernels) that make a step fit and run faster.
A practitioner's course on how large-model training is actually distributed: data parallelism and sharding (DDP, ZeRO, FSDP2), tensor, pipeline, context and expert parallelism, how to combine them, and how to launch, checkpoint, measure and debug the result on a real cluster.
How to turn the scaling-laws literature into a defensible compute plan: counting training FLOPs, what Kaplan, Chinchilla and their replications actually found, when to deliberately train past compute-optimal, how to fit your own laws from small runs, and how to convert a model and token budget into GPUs, weeks, risk buffers and cost.