Wirewalk Instituteof AI
trn-320 Expert Waitlist

Scaling Laws and Compute Planning

For research leads and ML infrastructure engineers who must decide how big a model to train, on how many tokens, on what cluster and for how long.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

Every large training run starts with a bet: a parameter count, a token count and a compute budget. Scaling laws are the best tool for making that bet, but they are often quoted as slogans rather than used as methods. This course goes to the source papers and their assumptions. You will count training compute with the 6ND approximation and its corrections, read the Kaplan et al. (2020) and Hoffmann et al. (2022, Chinchilla) results closely, and see why later replications and analyses revised the fitted coefficients.

You will then go beyond compute-optimal: training smaller models on far more tokens to cut inference cost, scaling when unique data runs out, and how mixture-of-experts models change the accounting. The practical core is fitting your own scaling law from a ladder of small runs, with honest uncertainty, and transferring hyperparameters with muP. The course ends by building a complete compute plan, from loss target to cluster size, wall-clock time, failure and ablation overhead, and a go/no-go milestone schedule. Published results are presented as their authors reported them; no figures are invented.

What you will be able to do

  • Estimate training compute for dense and mixture-of-experts transformers with 6ND and its corrections
  • Explain the Kaplan and Chinchilla scaling laws, how their methods differed and why later work revised their coefficients
  • Decide when to train beyond compute-optimal token counts given expected inference demand and data limits
  • Design an IsoFLOP or model-ladder sweep and fit a scaling law with uncertainty estimates
  • Apply hyperparameter transfer (muP and batch-size and learning-rate scaling rules) from small to large runs
  • Convert a model and token budget into GPU count, wall-clock time and cost using realistic utilization
  • Build a compute plan with risk buffers and staged go/no-go milestones that a budget owner can approve

Syllabus

6 modules · 17 lessons · about 16 hours

Module 1: Counting Compute

The arithmetic that links parameters, tokens, FLOPs and GPU-hours.
3 lessons
  1. The 6ND Approximation and Where It Breaks: Attention, Embeddings and MoE
  2. From FLOPs to GPU-Hours: Peak Throughput, Utilization and Wall-Clock Time
  3. Tokens, Parameters and Datasets: Measuring What You Actually Train On

Module 2: Reading the Scaling-Laws Literature

What the foundational papers measured, how, and what later work corrected.
3 lessons
  1. Power Laws in Loss: Kaplan et al. (2020)
  2. Compute-Optimal Training: Chinchilla's Three Estimation Approaches
  3. Replications and Discrepancies: Re-Fitting Chinchilla and Reconciling It with Kaplan

Module 3: Beyond Compute-Optimal

Real constraints that move the optimum: inference cost, finite data and sparse models.
3 lessons
  1. Inference-Aware Scaling: Overtraining Smaller Models on Purpose
  2. Data-Constrained Scaling: Repeated Epochs and Data Mixtures
  3. Mixture-of-Experts Scaling: Active Parameters, Total Parameters and Granularity

Module 4: Fitting Your Own Scaling Laws

Run small experiments that predict large ones, and know how far to trust them.
3 lessons
  1. Designing a Sweep: IsoFLOP Profiles and Model Ladders
  2. Fitting and Extrapolating: Loss Functions, Bootstrap Intervals and Failure Modes
  3. Hyperparameter Transfer: muP, Learning-Rate Schedules and Critical Batch Size

Module 5: From Loss to Capabilities

Why lower loss does not map neatly onto benchmark scores, and how test-time compute changes the plan.
2 lessons
  1. Downstream Metrics, Emergence Claims and the Role of Metric Choice
  2. Test-Time Compute Scaling and Its Effect on Training Budgets

Module 6: Building the Compute Plan

A complete, defensible plan from loss target to approved budget.
3 lessons
  1. A Worked Plan: Model Size, Tokens, Cluster Size and Calendar Time
  2. Risk Buffers: Failures, Restarts, Ablations and Evaluation Overhead
  3. Presenting the Plan: Cost, Energy and Go/No-Go Milestones

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by NVIDIA. Product and company names are trademarks of their respective owners.

More in Training at Scale

All 7 courses in this track
trn-201 Intermediate Waitlist

GPU Programming Fundamentals with CUDA

A hands-on introduction to CUDA C++: the execution model, the memory hierarchy, the classic parallel patterns, streams and CUDA Graphs, and profiling with Nsight. It finishes with the kernels deep learning actually needs, written in CUDA, Triton and the tile-based cuTile Python DSL and plugged into PyTorch.

14 h · 17 lessons Join the waitlist
trn-210 Intermediate Waitlist

Mixed Precision, Memory and Training Performance

The single-node foundations of fast training: what FP32, TF32, BF16, FP16, FP8, MXFP8 and NVFP4 actually trade away, how to use autocast and FP8 recipes safely, where every gigabyte of GPU memory goes, and the techniques (checkpointing, accumulation, compiled and fused kernels) that make a step fit and run faster.

14 h · 17 lessons Join the waitlist
trn-310 Expert Available

Distributed Training at Scale: DDP, FSDP, DeepSpeed and Megatron

A practitioner's course on how large-model training is actually distributed: data parallelism and sharding (DDP, ZeRO, FSDP2), tensor, pipeline, context and expert parallelism, how to combine them, and how to launch, checkpoint, measure and debug the result on a real cluster.

20 h · 18 lessons See price at checkout