Wirewalk Instituteof AI
mod-330 Expert Available

Model Evaluation and Benchmarking

For ML engineers and evaluation leads who must measure what a model can actually do, compare models fairly, and defend the numbers to skeptics.

Level
Expert
Study time
18 hours
Lessons
20 in 6 modules
Exam
25 questions

About this course

Benchmark numbers drive model selection, funding decisions and regulatory filings, yet small changes in prompt format, few-shot examples, answer extraction or sampling settings can move a score by several points. This course teaches evaluation as measurement science. You learn how lm-evaluation-harness, lighteval and Inspect differ, trace why the same model gets different results in each (with an optional hands-on exercise that runs one small model through two harnesses on a GPU), and learn to report scores with confidence intervals and paired significance tests rather than single numbers.

The course covers the major benchmark families (knowledge, reasoning, math, code, long context, multilingual and agentic tasks), how they saturate and how contamination is detected, and why once-standard benchmarks get retired, as with Hugging Face's Open LLM Leaderboard in 2025 and OpenAI's decision in February 2026 to stop reporting SWE-bench Verified. You then build a private, versioned evaluation suite for your own domain, including LLM-as-judge with calibration against human labels, and pairwise human preference methods such as Elo and Bradley-Terry arenas. The final module covers dangerous-capability and safety evaluations and how evaluation evidence feeds governance, including the EU AI Act obligations for general-purpose models with systemic risk. This course evaluates models; evaluating whole LLM applications is covered in bld-230. Facts are current as of October 2026.

What you will be able to do

  • Explain how log-likelihood, exact-match and generative scoring work and why harness choices change results
  • Reproduce a published benchmark score and account for any gap from prompt, extraction or decoding differences
  • Quantify evaluation uncertainty with bootstrap confidence intervals and paired tests across models
  • Detect and mitigate benchmark contamination and recognize saturated benchmarks
  • Design a private, versioned evaluation suite with held-out data for a specific domain
  • Calibrate LLM-as-judge scoring against human labels and control for position and length bias
  • Plan capability, agentic and safety evaluations and report results in a form governance teams can use

Syllabus

6 modules · 20 lessons · about 18 hours

Module 1: What a benchmark score really measures

The mechanics of scoring and the hidden choices behind every number.
3 lessons
  1. Log-likelihood, exact-match and generative scoring
  2. Prompt format, few-shot selection and answer extraction
  3. Decoding settings, reasoning budgets and why reruns differ

Module 2: Evaluation harnesses in practice

Running standardized evaluations reproducibly.
4 lessons
  1. lm-evaluation-harness: tasks, configs and serving with vLLM
  2. Inspect: datasets, solvers, scorers and sandboxed agent tasks
  3. HELM and lighteval: holistic and multi-metric reporting
  4. Reproducing a model card's numbers

Module 3: The benchmark landscape

Which benchmarks measure what, and how they stop being useful.
4 lessons
  1. Knowledge and reasoning: MMLU-Pro, GPQA and Humanity's Last Exam
  2. Math and code: competition math, LiveCodeBench and SWE-bench variants
  3. Long-context, multilingual and multimodal benchmarks
  4. Saturation and retirement: the Open LLM Leaderboard and SWE-bench Verified cases

Module 4: Statistics for model comparison

Turning scores into claims you can defend.
3 lessons
  1. Sample size, standard error and bootstrap confidence intervals
  2. Paired comparisons, multiple testing and effect sizes
  3. Contamination detection: n-gram overlap, perturbation and canaries

Module 5: Custom and human-centered evaluation

Measuring what public benchmarks miss.
3 lessons
  1. Building a private domain eval set: sampling, labeling and versioning
  2. LLM-as-judge: rubrics, calibration and bias controls
  3. Human preference arenas: Elo, Bradley-Terry and the LMArena method

Module 6: Frontier, agentic and safety evaluation

Evaluating capabilities that carry risk and reporting them responsibly.
3 lessons
  1. Agentic evaluations: tool use, long-horizon tasks and environment design
  2. Dangerous-capability and red-team evaluations
  3. Evaluation reports, model cards and EU AI Act GPAI obligations

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by EleutherAI (lm-evaluation-harness), Inspect (open-source evaluation framework), Stanford CRFM (HELM), Hugging Face (lighteval), LMArena, SWE-bench, or vLLM. Product and company names are trademarks of their respective owners.

More in Model Development

All 16 courses in this track
mod-201 Intermediate Available

Deep Learning with PyTorch

A hands-on course in modern PyTorch: tensors and autograd, nn.Module design, data loading, a training loop you understand line by line, GPU and mixed-precision training, torch.compile, and exporting a model for deployment.

14 h · 19 lessons See price at checkout
mod-202 Intermediate Available

Transformers from the Ground Up

Build a decoder-only transformer piece by piece: tokenization, embeddings, scaled dot-product attention, multi-head and grouped-query attention, positional schemes such as RoPE, and the training and sampling loop, then map it onto real open-weight model architectures.

16 h · 19 lessons See price at checkout
mod-203 Intermediate Waitlist

Classical Machine Learning in Production

Gradient-boosted trees and linear models still win most tabular problems. This course covers leakage-proof pipelines, honest validation, tuning, calibration and explainability, then packaging, serving and monitoring for drift.

14 h · 19 lessons Join the waitlist