Wirewalk Instituteof AI
srv-330 Expert Waitlist

LLM Observability and Monitoring

For engineers and SREs running LLM applications and agents in production who need to see what the system did, why it failed, what it cost, and whether quality is drifting.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

Traditional monitoring tells you a request succeeded in 2 seconds; it cannot tell you that the answer was wrong, that retrieval returned the wrong document, or that an agent looped through eleven tool calls and spent your daily budget. This course teaches observability for systems whose failures are semantic as well as operational. You instrument applications with OpenTelemetry and the GenAI semantic conventions (still marked Development as of October 2026, so you will learn how to cope with attribute churn), add auto-instrumentation from OpenLLMetry or OpenInference, and send traces to an LLM-focused backend such as Langfuse, Arize Phoenix, LangSmith or MLflow Tracing alongside Prometheus, Grafana or Datadog.

From there you build the metrics that matter — TTFT, inter-token latency, tokens, cost per request and per tenant, cache hit rate, error and refusal rates — and the serving-layer and GPU signals beneath them. You add online evaluation with sampled LLM-as-judge and user feedback, detect drift in inputs and quality, design SLOs and alerts for non-deterministic systems, and handle prompts and outputs that contain personal data. The course closes with debugging real failure patterns in RAG pipelines and agents from their traces. Product facts are stated as of October 2026.

What you will be able to do

  • Design a telemetry model for an LLM application that links each user request to its prompt, retrieval, model calls, tool calls and cost
  • Instrument an application with OpenTelemetry and GenAI semantic conventions and export traces to an LLM observability backend
  • Build dashboards for latency (TTFT, ITL, end-to-end), token usage, cost per tenant and serving-engine health
  • Run online evaluation on sampled production traffic and connect quality scores to traces
  • Detect input drift, quality regression and cost anomalies and define SLOs and alerts that avoid alert fatigue
  • Apply redaction, sampling and retention policies to prompts and completions that may contain personal or confidential data
  • Diagnose RAG and agent failures such as bad retrieval, tool-call loops and context overflow from trace evidence

Syllabus

6 modules · 19 lessons · about 18 hours

Module 1: What Makes LLM Systems Hard to Observe

Semantic failures, non-determinism and multi-step execution, and the telemetry model that captures them.
3 lessons
  1. Operational versus Semantic Failures in LLM Systems
  2. Traces, Spans, Sessions and Users: A Telemetry Model for AI Applications
  3. Build, Buy or Self-Host: Choosing an Observability Stack

Module 2: Instrumentation with OpenTelemetry

Capturing model calls, retrieval, tools and agent steps in a vendor-neutral way.
4 lessons
  1. OpenTelemetry GenAI Semantic Conventions and Living with Development Status
  2. Auto-Instrumentation with OpenLLMetry and OpenInference
  3. Manual Spans for Retrieval, Tool Calls, MCP and Agent Loops
  4. Propagating Context Through Gateways, Queues and Async Workers

Module 3: Metrics: Latency, Tokens, Cost and the Serving Layer

The quantitative signals from the application down to the GPU.
3 lessons
  1. Latency That Users Feel: TTFT, ITL, End-to-End and Streaming
  2. Token Accounting, Cost Attribution and Per-Tenant Budgets
  3. Serving-Engine and GPU Signals: vLLM Metrics, KV Cache Usage and DCGM

Module 4: Monitoring Quality in Production

Scoring live traffic, capturing feedback and detecting drift in what users ask and what the system returns.
3 lessons
  1. Online Evaluation: Sampling, LLM-as-Judge and Heuristic Checks
  2. User Feedback, Annotation Queues and Closing the Loop to Offline Evals
  3. Detecting Input Drift, Topic Shift and Quality Regression

Module 5: Alerting, SLOs and Data Governance

Turning signals into actionable alerts, and handling sensitive prompt data responsibly.
3 lessons
  1. SLOs and Error Budgets for Non-Deterministic Systems
  2. Alert Design: Cost Spikes, Refusal Rates, Provider Outages and Loops
  3. Redaction, Sampling and Retention for Prompts and Completions

Module 6: Debugging from Traces

Worked investigations of common production failures in RAG systems and agents.
3 lessons
  1. Debugging RAG: Empty Retrieval, Wrong Chunks and Context Overflow
  2. Debugging Agents: Tool-Call Loops, Bad Arguments and Runaway Cost
  3. Running an LLM Incident Review with Trace Evidence

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by OpenTelemetry, OpenLLMetry, OpenInference, Langfuse, Arize Phoenix, LangSmith, MLflow, Weights & Biases Weave, Prometheus, Grafana, Datadog, vLLM, or NVIDIA DCGM. Product and company names are trademarks of their respective owners.

More in Inference, Serving and MLOps

All 6 courses in this track
srv-201 Intermediate Waitlist

MLOps Fundamentals

The working practices that make machine learning reproducible and operable: versioning data, code and models together, tracking experiments, building pipelines, gating releases with CI/CD, and watching models for drift after they ship. Covers both classical ML and the newer LLM-application lifecycle.

14 h · 18 lessons Join the waitlist
srv-210 Intermediate Waitlist

Deploying Models: APIs, Containers and Autoscaling

A practical path from a model file to a production service: designing the inference API, packaging it in a lean GPU-ready container, running it on Kubernetes with KServe, autoscaling it on the right signal, and rolling out new versions safely. Focused on general ML models; LLM-specific engines are covered in srv-310.

14 h · 18 lessons Join the waitlist
srv-310 Expert Available

LLM Inference and Serving: vLLM, SGLang, TensorRT-LLM and Triton

The engineering of LLM inference from first principles to production: why decode is memory-bound, how to size the KV cache, how continuous batching, PagedAttention, prefix caching, quantization and speculative decoding change the economics, and how to run vLLM, SGLang, TensorRT-LLM and Dynamo at scale. Ends with load testing, Kubernetes autoscaling and a defensible cost per million tokens.

22 h · 19 lessons See price at checkout