Wirewalk Instituteof AI
mod-340 Expert Waitlist

Distillation, Pruning and Small Language Models

For ML engineers who need a smaller, cheaper model that keeps the capability that matters, by distilling from a teacher, pruning structure away, or picking the right small model to start from.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

Many production workloads do not need the largest model available; they need a model that is good at one job, fast and cheap. This course covers the techniques that produce such models. You start with knowledge distillation in its classic form, matching a teacher's softened output distribution, then move to the variants that matter for language models: sequence-level distillation, on-policy and generalized KD that train on the student's own samples, and reasoning-trace distillation from a large teacher.

The pruning half explains unstructured, semi-structured 2:4 and structured pruning, how importance is estimated from weights, activations or gradients, and the prune-then-distill recipe used by NVIDIA's Minitron work and implemented in Model Optimizer, where depth and width are cut and the model is retrained against its parent. You learn how to evaluate a compressed model against the teacher on your own task, how teacher-output licensing terms constrain what you may train, and how to choose among small open-weight families. Quantization is touched on only as a complementary step; it is taught in depth in srv-320. Model and library facts are current as of October 2026.

What you will be able to do

  • Explain logit, feature and sequence-level knowledge distillation and choose a loss and temperature for a task
  • Run on-policy and generalized distillation with TRL and explain why it reduces exposure bias
  • Generate and filter synthetic training data from a teacher model, respecting the teacher's terms of use
  • Apply unstructured, 2:4 semi-structured and structured pruning and estimate the real speedup on target hardware
  • Execute a prune-then-distill pipeline that removes layers or width and recovers quality
  • Evaluate a compressed model against its teacher on task metrics, latency and cost per request
  • Select a small open-weight model family and size for a deployment constraint and justify the choice

Syllabus

6 modules · 20 lessons · about 18 hours

Module 1: Why small models win

The economics and the capability trade-offs of going small.
3 lessons
  1. Cost, latency and privacy drivers for small models
  2. What small models lose first: knowledge, reasoning or instruction following
  3. Compression options compared: distill, prune, quantize or fine-tune

Module 2: Knowledge distillation foundations

Transferring a teacher's behavior into a student network.
3 lessons
  1. Soft targets, temperature and the KL distillation loss
  2. Feature and hidden-state distillation
  3. Tokenizer mismatch and cross-family distillation: ULD-style and cross-tokenizer logit alignment

Module 3: Distilling language models

LLM-specific distillation recipes and their failure modes.
4 lessons
  1. Sequence-level distillation and synthetic data from a teacher
  2. On-policy and generalized KD with TRL's GKD and GOLD trainers
  3. Distilling reasoning traces into small models
  4. Teacher terms of use and data provenance

Module 4: Pruning

Removing parameters and structure without breaking the model.
3 lessons
  1. Unstructured and 2:4 semi-structured sparsity: SparseGPT and Wanda
  2. Structured pruning: heads, channels, FFN width and layers
  3. Importance scoring from activations, gradients and layer similarity

Module 5: Prune, then distill

Combining pruning and distillation to produce a new model family member.
3 lessons
  1. The Minitron recipe: depth and width pruning with distillation recovery
  2. Running pruning and distillation with NVIDIA Model Optimizer and NeMo
  3. Recovery budgets: how many tokens retraining really needs

Module 6: Evaluating and shipping small models

Proving the smaller model is good enough and choosing what to deploy.
4 lessons
  1. Teacher-student evaluation on your task, not just leaderboards
  2. Small open-weight families compared: Phi, Gemma, Qwen and Llama
  3. Measuring real latency and cost per request after compression
  4. Stacking compression: distillation plus quantization for edge targets

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by Hugging Face (TRL, Transformers), NVIDIA (Model Optimizer, NeMo, Minitron), PyTorch, Microsoft (Phi), Google (Gemma), Qwen, Meta Llama, or vLLM. Product and company names are trademarks of their respective owners.

More in Model Development

All 16 courses in this track
mod-201 Intermediate Available

Deep Learning with PyTorch

A hands-on course in modern PyTorch: tensors and autograd, nn.Module design, data loading, a training loop you understand line by line, GPU and mixed-precision training, torch.compile, and exporting a model for deployment.

14 h · 19 lessons See price at checkout
mod-202 Intermediate Available

Transformers from the Ground Up

Build a decoder-only transformer piece by piece: tokenization, embeddings, scaled dot-product attention, multi-head and grouped-query attention, positional schemes such as RoPE, and the training and sampling loop, then map it onto real open-weight model architectures.

16 h · 19 lessons See price at checkout
mod-203 Intermediate Waitlist

Classical Machine Learning in Production

Gradient-boosted trees and linear models still win most tabular problems. This course covers leakage-proof pipelines, honest validation, tuning, calibration and explainability, then packaging, serving and monitoring for drift.

14 h · 19 lessons Join the waitlist