For engineers and SREs running LLM applications and agents in production who need to see what the system did, why it failed, what it cost, and whether quality is drifting.
The curriculum is being prepared and reviewed. Enrollment is not open yet.
About this course
Traditional monitoring tells you a request succeeded in 2 seconds; it cannot tell you that the answer was wrong, that retrieval returned the wrong document, or that an agent looped through eleven tool calls and spent your daily budget. This course teaches observability for systems whose failures are semantic as well as operational. You instrument applications with OpenTelemetry and the GenAI semantic conventions (still marked Development as of October 2026, so you will learn how to cope with attribute churn), add auto-instrumentation from OpenLLMetry or OpenInference, and send traces to an LLM-focused backend such as Langfuse, Arize Phoenix, LangSmith or MLflow Tracing alongside Prometheus, Grafana or Datadog.
From there you build the metrics that matter — TTFT, inter-token latency, tokens, cost per request and per tenant, cache hit rate, error and refusal rates — and the serving-layer and GPU signals beneath them. You add online evaluation with sampled LLM-as-judge and user feedback, detect drift in inputs and quality, design SLOs and alerts for non-deterministic systems, and handle prompts and outputs that contain personal data. The course closes with debugging real failure patterns in RAG pipelines and agents from their traces. Product facts are stated as of October 2026.
What you will be able to do
Design a telemetry model for an LLM application that links each user request to its prompt, retrieval, model calls, tool calls and cost
Instrument an application with OpenTelemetry and GenAI semantic conventions and export traces to an LLM observability backend
Build dashboards for latency (TTFT, ITL, end-to-end), token usage, cost per tenant and serving-engine health
Run online evaluation on sampled production traffic and connect quality scores to traces
Detect input drift, quality regression and cost anomalies and define SLOs and alerts that avoid alert fatigue
Apply redaction, sampling and retention policies to prompts and completions that may contain personal or confidential data
Diagnose RAG and agent failures such as bad retrieval, tool-call loops and context overflow from trace evidence
Syllabus
6 modules · 19 lessons · about 18 hours
1
Module 1: What Makes LLM Systems Hard to Observe
Semantic failures, non-determinism and multi-step execution, and the telemetry model that captures them.3 lessons
Operational versus Semantic Failures in LLM Systems
Traces, Spans, Sessions and Users: A Telemetry Model for AI Applications
Build, Buy or Self-Host: Choosing an Observability Stack
2
Module 2: Instrumentation with OpenTelemetry
Capturing model calls, retrieval, tools and agent steps in a vendor-neutral way.4 lessons
OpenTelemetry GenAI Semantic Conventions and Living with Development Status
Auto-Instrumentation with OpenLLMetry and OpenInference
Manual Spans for Retrieval, Tool Calls, MCP and Agent Loops
Propagating Context Through Gateways, Queues and Async Workers
3
Module 3: Metrics: Latency, Tokens, Cost and the Serving Layer
The quantitative signals from the application down to the GPU.3 lessons
Latency That Users Feel: TTFT, ITL, End-to-End and Streaming
Token Accounting, Cost Attribution and Per-Tenant Budgets
Serving-Engine and GPU Signals: vLLM Metrics, KV Cache Usage and DCGM
4
Module 4: Monitoring Quality in Production
Scoring live traffic, capturing feedback and detecting drift in what users ask and what the system returns.3 lessons
Online Evaluation: Sampling, LLM-as-Judge and Heuristic Checks
User Feedback, Annotation Queues and Closing the Loop to Offline Evals
Detecting Input Drift, Topic Shift and Quality Regression
5
Module 5: Alerting, SLOs and Data Governance
Turning signals into actionable alerts, and handling sensitive prompt data responsibly.3 lessons
SLOs and Error Budgets for Non-Deterministic Systems
Alert Design: Cost Spikes, Refusal Rates, Provider Outages and Loops
Redaction, Sampling and Retention for Prompts and Completions
6
Module 6: Debugging from Traces
Worked investigations of common production failures in RAG systems and agents.3 lessons
Debugging RAG: Empty Retrieval, Wrong Chunks and Context Overflow
Debugging Agents: Tool-Call Loops, Bad Arguments and Runaway Cost
Running an LLM Incident Review with Trace Evidence
Exam and certificate
The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.
Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.
The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by OpenTelemetry, OpenLLMetry, OpenInference, Langfuse, Arize Phoenix, LangSmith, MLflow, Weights & Biases Weave, Prometheus, Grafana, Datadog, vLLM, or NVIDIA DCGM. Product and company names are trademarks of their respective owners.
The working practices that make machine learning reproducible and operable: versioning data, code and models together, tracking experiments, building pipelines, gating releases with CI/CD, and watching models for drift after they ship. Covers both classical ML and the newer LLM-application lifecycle.
A practical path from a model file to a production service: designing the inference API, packaging it in a lean GPU-ready container, running it on Kubernetes with KServe, autoscaling it on the right signal, and rolling out new versions safely. Focused on general ML models; LLM-specific engines are covered in srv-310.
The engineering of LLM inference from first principles to production: why decode is memory-bound, how to size the KV cache, how continuous batching, PagedAttention, prefix caching, quantization and speculative decoding change the economics, and how to run vLLM, SGLang, TensorRT-LLM and Dynamo at scale. Ends with load testing, Kubernetes autoscaling and a defensible cost per million tokens.