Wirewalk Instituteof AI
prd-203 Intermediate Available

Measuring AI Features: Metrics, Evals and Experiments

For product managers, analysts and data scientists: define what success means for an AI feature, connect offline evals to product metrics, and run online experiments that hold up to scrutiny.

Level
Intermediate
Study time
12 hours
Lessons
20 in 6 modules
Exam
20 questions

About this course

AI features are easy to demo and hard to measure. Usage can rise while value falls, thumbs-up rates mislead, and offline eval scores do not always predict what users do. This course gives product teams a measurement system. You start with a metric tree linking a business outcome to task success, quality and cost, adapt frameworks such as HEART and north-star metrics to AI, and decide what to log, including acceptance, edit distance, retries, escalations and abandonment.

You then connect evaluation to product decisions: how offline eval sets (covered technically in bld-230) become ship criteria, and how to check that eval gains move product metrics. The experimentation modules cover A/B test design for features whose output varies, randomization units, novelty effects, guardrail and cost metrics, CUPED variance reduction and sequential testing, using platforms such as Statsig, Optimizely, GrowthBook, LaunchDarkly and Datadog Experiments, which Datadog built on its 2025 acquisition of Eppo. The course closes with dashboards, readouts and decisions when results conflict. Facts are stated as of October 2026. It ends in an automated exam; passing earns a Wirewalk Institute of AI certificate of completion and assessment.

What you will be able to do

  • Build a metric tree that links an AI feature's business outcome to task success, quality and cost metrics
  • Choose and instrument implicit and explicit feedback signals and explain the bias in each
  • Translate offline eval results into ship criteria and test whether they predict online outcomes
  • Design an A/B test for a non-deterministic AI feature with an appropriate unit, duration and guardrails
  • Apply power analysis, CUPED variance reduction and sequential testing correctly
  • Diagnose misleading results caused by novelty effects, interference, sample ratio mismatch and metric gaming
  • Present an experiment readout that leads to a clear ship, iterate or stop decision

Syllabus

6 modules · 20 lessons · about 12 hours

Module 1: What Success Means for an AI Feature

From vanity usage to outcomes users and the business actually value.
3 lessons
  1. Why Usage and Thumbs-Up Rates Mislead for AI Features
  2. Metric Trees: Business Outcome, Task Success, Quality and Cost
  3. Adapting HEART and North-Star Metrics to AI Products

Module 2: Instrumenting Behavior and Feedback

Logging the signals that reveal whether AI output was useful.
3 lessons
  1. Implicit Signals: Acceptance, Edits, Retries, Copies and Abandonment
  2. Explicit Feedback: Ratings, Reasons and Their Selection Bias
  3. Event Design and Trace Linking in Amplitude, Mixpanel and Your Warehouse

Module 3: From Offline Evals to Product Decisions

Making eval scores useful to product teams, and checking that they predict reality.
3 lessons
  1. Eval Sets as Product Requirements: Coverage, Weighting and Thresholds
  2. Validating Evals Against Online Behavior
  3. Human Review Programs and Sampling Production for Quality Audits

Module 4: Designing Experiments for AI Features

A/B tests when the treatment itself is non-deterministic.
4 lessons
  1. Choosing the Randomization Unit and Avoiding Interference
  2. Primary, Secondary and Guardrail Metrics Including Latency and Cost
  3. Power Analysis and Test Duration for AI Changes
  4. Novelty, Learning Effects and Long-Term Holdouts

Module 5: Running and Analyzing Experiments

Platforms, statistics and the traps that invalidate results.
4 lessons
  1. Experimentation Platforms Compared: Statsig, Optimizely, GrowthBook, LaunchDarkly and Datadog Experiments
  2. CUPED Variance Reduction and Sequential Testing
  3. Sample Ratio Mismatch, Peeking and Other Result Killers
  4. Interleaving and Bandits for Ranking and Model Selection

Module 6: Reporting and Deciding

Turning measurement into decisions people trust.
3 lessons
  1. AI Feature Dashboards: Quality, Adoption, Cost and Risk on One Page
  2. Writing the Experiment Readout and Making the Ship Call
  3. When Metrics Conflict: Quality Up, Cost Up, Engagement Down

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 20 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 40 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by Statsig, Optimizely, GrowthBook, LaunchDarkly, Datadog, Amplitude, or Mixpanel. Product and company names are trademarks of their respective owners.

More in AI Product and Design

All 4 courses in this track
prd-201 Intermediate Available

AI Product Management: From Use Case to Launch

The product manager's playbook for AI features. You learn to screen use cases for AI fit, run discovery with real model output, write a spec that defines quality with an eval set, choose between APIs, open-weight models and fine-tuning, and plan a staged launch with risk, compliance and support ready on day one.

12 h · 21 lessons See price at checkout
prd-202 Intermediate Available

Designing AI User Experiences

A practical course on AI interaction design. You apply Microsoft's Guidelines for Human-AI Interaction and Google's People + AI Guidebook to real patterns: onboarding and expectation setting, chat versus embedded AI, streaming and latency, citations and uncertainty, error recovery, agent approvals, transparency labels and accessibility.

12 h · 19 lessons See price at checkout
prd-301 Expert Available

Pricing and Unit Economics of AI Products

An expert course on the economics of AI products. You build a cost-to-serve model from tokens, caching, retrieval, tools and infrastructure, compare seat, usage, credit, hybrid and outcome-based pricing using public examples, design packaging and limits that protect margin, and implement metering and billing with tools such as Stripe Billing, Metronome and Orb.

18 h · 22 lessons See price at checkout