For research leads and ML infrastructure engineers who must decide how big a model to train, on how many tokens, on what cluster and for how long.
The curriculum is being prepared and reviewed. Enrollment is not open yet.
About this course
Every large training run starts with a bet: a parameter count, a token count and a compute budget. Scaling laws are the best tool for making that bet, but they are often quoted as slogans rather than used as methods. This course goes to the source papers and their assumptions. You will count training compute with the 6ND approximation and its corrections, read the Kaplan et al. (2020) and Hoffmann et al. (2022, Chinchilla) results closely, and see why later replications and analyses revised the fitted coefficients.
You will then go beyond compute-optimal: training smaller models on far more tokens to cut inference cost, scaling when unique data runs out, and how mixture-of-experts models change the accounting. The practical core is fitting your own scaling law from a ladder of small runs, with honest uncertainty, and transferring hyperparameters with muP. The course ends by building a complete compute plan, from loss target to cluster size, wall-clock time, failure and ablation overhead, and a go/no-go milestone schedule. Published results are presented as their authors reported them; no figures are invented.
What you will be able to do
Estimate training compute for dense and mixture-of-experts transformers with 6ND and its corrections
Explain the Kaplan and Chinchilla scaling laws, how their methods differed and why later work revised their coefficients
Decide when to train beyond compute-optimal token counts given expected inference demand and data limits
Design an IsoFLOP or model-ladder sweep and fit a scaling law with uncertainty estimates
Apply hyperparameter transfer (muP and batch-size and learning-rate scaling rules) from small to large runs
Convert a model and token budget into GPU count, wall-clock time and cost using realistic utilization
Build a compute plan with risk buffers and staged go/no-go milestones that a budget owner can approve
Syllabus
6 modules · 17 lessons · about 16 hours
1
Module 1: Counting Compute
The arithmetic that links parameters, tokens, FLOPs and GPU-hours.3 lessons
The 6ND Approximation and Where It Breaks: Attention, Embeddings and MoE
From FLOPs to GPU-Hours: Peak Throughput, Utilization and Wall-Clock Time
Tokens, Parameters and Datasets: Measuring What You Actually Train On
2
Module 2: Reading the Scaling-Laws Literature
What the foundational papers measured, how, and what later work corrected.3 lessons
Power Laws in Loss: Kaplan et al. (2020)
Compute-Optimal Training: Chinchilla's Three Estimation Approaches
Replications and Discrepancies: Re-Fitting Chinchilla and Reconciling It with Kaplan
3
Module 3: Beyond Compute-Optimal
Real constraints that move the optimum: inference cost, finite data and sparse models.3 lessons
Inference-Aware Scaling: Overtraining Smaller Models on Purpose
Data-Constrained Scaling: Repeated Epochs and Data Mixtures
Mixture-of-Experts Scaling: Active Parameters, Total Parameters and Granularity
4
Module 4: Fitting Your Own Scaling Laws
Run small experiments that predict large ones, and know how far to trust them.3 lessons
Designing a Sweep: IsoFLOP Profiles and Model Ladders
Fitting and Extrapolating: Loss Functions, Bootstrap Intervals and Failure Modes
Hyperparameter Transfer: muP, Learning-Rate Schedules and Critical Batch Size
5
Module 5: From Loss to Capabilities
Why lower loss does not map neatly onto benchmark scores, and how test-time compute changes the plan.2 lessons
Downstream Metrics, Emergence Claims and the Role of Metric Choice
Test-Time Compute Scaling and Its Effect on Training Budgets
6
Module 6: Building the Compute Plan
A complete, defensible plan from loss target to approved budget.3 lessons
A Worked Plan: Model Size, Tokens, Cluster Size and Calendar Time
Risk Buffers: Failures, Restarts, Ablations and Evaluation Overhead
Presenting the Plan: Cost, Energy and Go/No-Go Milestones
Exam and certificate
The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.
Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.
The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by NVIDIA. Product and company names are trademarks of their respective owners.
A hands-on introduction to CUDA C++: the execution model, the memory hierarchy, the classic parallel patterns, streams and CUDA Graphs, and profiling with Nsight. It finishes with the kernels deep learning actually needs, written in CUDA, Triton and the tile-based cuTile Python DSL and plugged into PyTorch.
The single-node foundations of fast training: what FP32, TF32, BF16, FP16, FP8, MXFP8 and NVFP4 actually trade away, how to use autocast and FP8 recipes safely, where every gigabyte of GPU memory goes, and the techniques (checkpointing, accumulation, compiled and fused kernels) that make a step fit and run faster.
A practitioner's course on how large-model training is actually distributed: data parallelism and sharding (DDP, ZeRO, FSDP2), tensor, pipeline, context and expert parallelism, how to combine them, and how to launch, checkpoint, measure and debug the result on a real cluster.