For engineers and product owners who need to know whether an LLM feature is good enough to ship and whether a change made it better: build eval sets, graders and an eval pipeline that gates releases.
Level
Intermediate
Study time
14 hours
Lessons
19 in 6 modules
Exam
20 questions
About this course
An LLM feature that passes a demo can still fail a meaningful share of real requests, and every prompt or model change can silently move that number. This course teaches evaluation as an engineering discipline. You start with error analysis on real outputs, derive failure categories, and build a dataset that represents both typical traffic and the edge cases that matter. You then build graders: deterministic assertions, reference-based metrics, rubric-based LLM judges, and pairwise comparisons, and you measure each judge's agreement with human labels before trusting it.
The second half puts evals to work. You will wire an eval suite into CI with Promptfoo or Inspect AI, and compare open-source frameworks (DeepEval, Ragas) with hosted platforms (LangSmith, Braintrust, Arize Phoenix, Langfuse). You will handle non-determinism with repeated trials and confidence intervals, and connect offline results to online signals such as user feedback and A/B tests. Retrieval-specific metrics are covered in bld-210 and agent trajectory evals in bld-320; this course is the general foundation both build on. The course ends in an automated exam; passing earns a Wirewalk Institute of AI certificate of completion and assessment, which records what the exam measured rather than demonstrated workplace competence. Product facts were checked in October 2026; the course is re-reviewed quarterly and errors can be reported through the support contact on institute.wirewalk.com.
What you will be able to do
Run an error analysis on real model outputs and turn the observed failures into named, testable categories
Build an evaluation dataset that covers typical traffic, known failures and adversarial cases, with a documented sampling rationale
Choose between code assertions, reference metrics, human review and LLM-as-judge for each quality criterion
Write a rubric-based LLM judge and measure its agreement with human labels using precision, recall and Cohen's kappa
Quantify run-to-run variance and decide whether a score change is real using repeated trials and confidence intervals
Integrate an eval suite into CI so prompt, model and retrieval changes are gated on regression thresholds
Connect offline eval results to production traces, user feedback and online experiments
Syllabus
6 modules · 19 lessons · about 14 hours
1
Module 1: Why LLM Evaluation Is Different
Open-ended outputs, non-determinism and why accuracy on a demo tells you little.3 lessons
From Vibe Checks to Evidence: What an Eval Must Answer
Error Analysis: Reading Outputs and Building a Failure Taxonomy
Offline, Online and Human Evaluation: Where Each Fits
2
Module 2: Building Evaluation Datasets
Test cases that represent reality and keep representing it.3 lessons
Sampling Production Traffic and Writing Edge Cases
Synthetic Test Generation and Its Blind Spots
Versioning Datasets and Preventing Contamination
3
Module 3: Graders: Code, References and People
The cheapest reliable check for every criterion.3 lessons
Deterministic Assertions: Schemas, Regex, Execution and Tool-Call Checks
Reference-Based Metrics and Their Limits
Human Review Protocols: Rubrics, Double-Labeling and Adjudication
4
Module 4: LLM-as-Judge Done Properly
Writing judges, measuring their bias and trusting them only as far as the data allows.3 lessons
Writing Rubric and Pairwise Judge Prompts
Judge Biases: Position, Verbosity, Self-Preference and Leniency
Calibrating a Judge Against Human Labels
5
Module 5: Eval Tooling and CI
Running evals automatically with open-source and hosted tools.4 lessons
Promptfoo and Inspect AI: Config-Driven Eval Suites
DeepEval, Ragas, LangSmith, Braintrust, Phoenix and Langfuse Compared
Statistics for Evals: Repeated Trials, Confidence Intervals and Significance
Release Gates: Regression Thresholds in CI
6
Module 6: Evaluating in Production
Closing the loop between live traffic and the offline suite.3 lessons
Online Evals on Sampled Traces and Feedback Signals
A/B Tests and Shadow Deployments for Prompt and Model Changes
Turning Incidents into Regression Tests
Exam and certificate
The exam unlocks once you have completed every lesson. It draws 20 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 40 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.
Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.
The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by OpenAI, Anthropic, Promptfoo, Inspect AI, DeepEval, Ragas, LangSmith, Braintrust, Arize Phoenix, or Langfuse. Product and company names are trademarks of their respective owners.
A hands-on course on building no-code assistants with the builders already in your workplace tools: ChatGPT projects, custom GPTs and their successor plugins, Claude Projects and Skills, Gemini Gems, and Microsoft 365 Copilot Agent Builder and Copilot Studio. You leave with a tested, documented assistant and a plan to share it safely.
A hands-on, code-first course on the three major LLM APIs side by side: messages, streaming, structured outputs, tool calling, vision, cost, caching, batch, retries, abstraction layers, cloud routes and testing. You finish by building and hardening a small multi-vendor service.
A practitioner's course on retrieval-augmented generation: when to use it, how to ingest, chunk, embed and index documents, how to combine BM25 and vector search with reranking, and how to measure, debug and operate the result. Every module includes working Python sketches.