Wirewalk Instituteof AI
bld-230 Intermediate Available

Evaluating LLM Applications

For engineers and product owners who need to know whether an LLM feature is good enough to ship and whether a change made it better: build eval sets, graders and an eval pipeline that gates releases.

Level
Intermediate
Study time
14 hours
Lessons
19 in 6 modules
Exam
20 questions

About this course

An LLM feature that passes a demo can still fail a meaningful share of real requests, and every prompt or model change can silently move that number. This course teaches evaluation as an engineering discipline. You start with error analysis on real outputs, derive failure categories, and build a dataset that represents both typical traffic and the edge cases that matter. You then build graders: deterministic assertions, reference-based metrics, rubric-based LLM judges, and pairwise comparisons, and you measure each judge's agreement with human labels before trusting it.

The second half puts evals to work. You will wire an eval suite into CI with Promptfoo or Inspect AI, and compare open-source frameworks (DeepEval, Ragas) with hosted platforms (LangSmith, Braintrust, Arize Phoenix, Langfuse). You will handle non-determinism with repeated trials and confidence intervals, and connect offline results to online signals such as user feedback and A/B tests. Retrieval-specific metrics are covered in bld-210 and agent trajectory evals in bld-320; this course is the general foundation both build on. The course ends in an automated exam; passing earns a Wirewalk Institute of AI certificate of completion and assessment, which records what the exam measured rather than demonstrated workplace competence. Product facts were checked in October 2026; the course is re-reviewed quarterly and errors can be reported through the support contact on institute.wirewalk.com.

What you will be able to do

  • Run an error analysis on real model outputs and turn the observed failures into named, testable categories
  • Build an evaluation dataset that covers typical traffic, known failures and adversarial cases, with a documented sampling rationale
  • Choose between code assertions, reference metrics, human review and LLM-as-judge for each quality criterion
  • Write a rubric-based LLM judge and measure its agreement with human labels using precision, recall and Cohen's kappa
  • Quantify run-to-run variance and decide whether a score change is real using repeated trials and confidence intervals
  • Integrate an eval suite into CI so prompt, model and retrieval changes are gated on regression thresholds
  • Connect offline eval results to production traces, user feedback and online experiments

Syllabus

6 modules · 19 lessons · about 14 hours

Module 1: Why LLM Evaluation Is Different

Open-ended outputs, non-determinism and why accuracy on a demo tells you little.
3 lessons
  1. From Vibe Checks to Evidence: What an Eval Must Answer
  2. Error Analysis: Reading Outputs and Building a Failure Taxonomy
  3. Offline, Online and Human Evaluation: Where Each Fits

Module 2: Building Evaluation Datasets

Test cases that represent reality and keep representing it.
3 lessons
  1. Sampling Production Traffic and Writing Edge Cases
  2. Synthetic Test Generation and Its Blind Spots
  3. Versioning Datasets and Preventing Contamination

Module 3: Graders: Code, References and People

The cheapest reliable check for every criterion.
3 lessons
  1. Deterministic Assertions: Schemas, Regex, Execution and Tool-Call Checks
  2. Reference-Based Metrics and Their Limits
  3. Human Review Protocols: Rubrics, Double-Labeling and Adjudication

Module 4: LLM-as-Judge Done Properly

Writing judges, measuring their bias and trusting them only as far as the data allows.
3 lessons
  1. Writing Rubric and Pairwise Judge Prompts
  2. Judge Biases: Position, Verbosity, Self-Preference and Leniency
  3. Calibrating a Judge Against Human Labels

Module 5: Eval Tooling and CI

Running evals automatically with open-source and hosted tools.
4 lessons
  1. Promptfoo and Inspect AI: Config-Driven Eval Suites
  2. DeepEval, Ragas, LangSmith, Braintrust, Phoenix and Langfuse Compared
  3. Statistics for Evals: Repeated Trials, Confidence Intervals and Significance
  4. Release Gates: Regression Thresholds in CI

Module 6: Evaluating in Production

Closing the loop between live traffic and the offline suite.
3 lessons
  1. Online Evals on Sampled Traces and Feedback Signals
  2. A/B Tests and Shadow Deployments for Prompt and Model Changes
  3. Turning Incidents into Regression Tests

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 20 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 40 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by OpenAI, Anthropic, Promptfoo, Inspect AI, DeepEval, Ragas, LangSmith, Braintrust, Arize Phoenix, or Langfuse. Product and company names are trademarks of their respective owners.

More in Building AI Applications

All 14 courses in this track
bld-101 Basic Waitlist

No-Code AI Assistants: Custom GPTs, Projects, Gems and Copilot Studio

A hands-on course on building no-code assistants with the builders already in your workplace tools: ChatGPT projects, custom GPTs and their successor plugins, Claude Projects and Skills, Gemini Gems, and Microsoft 365 Copilot Agent Builder and Copilot Studio. You leave with a tested, documented assistant and a plan to share it safely.

7 h · 17 lessons Join the waitlist
bld-201 Intermediate Available

Building with LLM APIs Across Vendors

A hands-on, code-first course on the three major LLM APIs side by side: messages, streaming, structured outputs, tool calling, vision, cost, caching, batch, retries, abstraction layers, cloud routes and testing. You finish by building and hardening a small multi-vendor service.

14 h · 18 lessons See price at checkout
bld-210 Intermediate Available

Retrieval-Augmented Generation Systems

A practitioner's course on retrieval-augmented generation: when to use it, how to ingest, chunk, embed and index documents, how to combine BM25 and vector search with reranking, and how to measure, debug and operate the result. Every module includes working Python sketches.

14 h · 16 lessons See price at checkout