AI features are easy to demo and hard to measure. Usage can rise while value falls, thumbs-up rates mislead, and offline eval scores do not always predict what users do. This course gives product teams a measurement system. You start with a metric tree linking a business outcome to task success, quality and cost, adapt frameworks such as HEART and north-star metrics to AI, and decide what to log, including acceptance, edit distance, retries, escalations and abandonment.
You then connect evaluation to product decisions: how offline eval sets (covered technically in bld-230) become ship criteria, and how to check that eval gains move product metrics. The experimentation modules cover A/B test design for features whose output varies, randomization units, novelty effects, guardrail and cost metrics, CUPED variance reduction and sequential testing, using platforms such as Statsig, Optimizely, GrowthBook, LaunchDarkly and Datadog Experiments, which Datadog built on its 2025 acquisition of Eppo. The course closes with dashboards, readouts and decisions when results conflict. Facts are stated as of October 2026. It ends in an automated exam; passing earns a Wirewalk Institute of AI certificate of completion and assessment.