Wirewalk Instituteof AI
bld-330 Expert Waitlist

Production LLM Engineering: Reliability, Latency and Cost

For senior engineers running LLM features at real traffic: set SLOs, engineer for provider outages and rate limits, cut latency and cost per request, and operate the system with tracing, budgets and safe rollouts.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

Calling an LLM API is easy; keeping a feature fast, available and inside budget when traffic grows, a provider degrades or a model is deprecated is not. This expert course treats the LLM call as a dependency with unusual properties: variable latency measured in seconds, token-based cost, rate limits in two dimensions, silent quality regressions and scheduled model retirements. You will define SLOs for availability, latency and quality, design failover across providers and cloud routes (including Amazon Bedrock, Google Vertex AI and Microsoft Foundry) with circuit breakers and hedged requests, and use priority tiers, provisioned throughput and batch APIs where they fit.

Latency and cost are treated quantitatively: decomposing TTFT and generation time, streaming and speculative UI patterns, prompt-prefix design for cache hits, semantic caching and its correctness risks, model cascades and routing, and per-feature unit economics. Operations modules cover OpenTelemetry GenAI semantic conventions (still at Development stability, so pinned by version and expected to change), token and cost attribution, load testing against rate limits, and rolling out prompt and model changes with canaries and automatic rollback. Basic caching, retries and batch usage are taught in bld-201, gateway deployment, keys and quotas in bld-270, and self-hosted inference engines in srv-310; this course is about the production system around the model call. The course ends in an automated exam; passing earns a Wirewalk Institute of AI certificate of completion and assessment.

What you will be able to do

  • Define availability, latency and quality SLOs for an LLM feature and manage them with error budgets
  • Design multi-provider and multi-region failover with circuit breakers, hedging and graceful degradation, and test it with fault injection
  • Decompose end-to-end latency into queueing, TTFT and generation time and apply the right fix to each
  • Design prompts and request flows that maximize prompt-cache hit rates and quantify the saving
  • Build a routing or cascade policy that sends requests to the cheapest model that meets the quality bar, validated by evals
  • Instrument an LLM service with OpenTelemetry GenAI conventions and attribute token cost to features, tenants and users
  • Roll out prompt and model changes with canaries, shadow traffic and automatic rollback, and plan for model deprecations

Syllabus

6 modules · 20 lessons · about 20 hours

Module 1: Reliability Targets for Probabilistic Systems

What 'up' means when outputs vary and dependencies are external.
3 lessons
  1. SLIs and SLOs for LLM Features: Availability, Latency and Quality
  2. Failure Modes of LLM Dependencies: Outages, Brownouts, Rate Limits and Silent Regressions
  3. Model Lifecycle Risk: Deprecations, Snapshot Pinning and Aliases

Module 2: Resilience Engineering

Keeping the feature working when a provider does not.
4 lessons
  1. Timeouts, Circuit Breakers and Hedged Requests for Long-Running Calls
  2. Multi-Provider and Multi-Region Failover Across Direct APIs, Bedrock, Vertex AI and Foundry
  3. Capacity Options: Priority Tiers, Provisioned Throughput and Batch Queues
  4. Graceful Degradation and Chaos Testing an LLM Path

Module 3: Latency Engineering

Making responses faster, and making them feel faster.
3 lessons
  1. Decomposing Latency: Network, Queue, TTFT and Output Tokens
  2. Output Length Control, Parallel Calls and Speculative Execution
  3. Streaming UX: Progressive Rendering, Partial JSON and Perceived Speed

Module 4: Cost Engineering

Reducing cost per successful task without losing quality.
4 lessons
  1. Unit Economics: Cost per Task, per User and per Feature
  2. Prompt-Prefix Architecture for Cache Hits at Scale
  3. Semantic Caching: Hit Rates, Staleness and Correctness Risks
  4. Model Cascades and Routers Validated by Evals

Module 5: Observability and Capacity

Seeing every call, its cost and its contribution to the user's wait.
3 lessons
  1. OpenTelemetry GenAI Semantic Conventions in Practice
  2. Token Accounting, Budgets and Spend Alerts per Tenant
  3. Load Testing Against Token and Request Rate Limits

Module 6: Shipping Changes Safely

Treating prompts, models and configuration as deployable artifacts.
3 lessons
  1. Prompts and Model Config as Versioned Artifacts with Feature Flags
  2. Canaries, Shadow Traffic and Automatic Rollback on Quality Signals
  3. Incident Response for LLM Features: Runbooks and Postmortems

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by OpenAI, Anthropic, Google, Amazon Web Services, Microsoft, or OpenTelemetry. Product and company names are trademarks of their respective owners.

More in Building AI Applications

All 14 courses in this track
bld-101 Basic Waitlist

No-Code AI Assistants: Custom GPTs, Projects, Gems and Copilot Studio

A hands-on course on building no-code assistants with the builders already in your workplace tools: ChatGPT projects, custom GPTs and their successor plugins, Claude Projects and Skills, Gemini Gems, and Microsoft 365 Copilot Agent Builder and Copilot Studio. You leave with a tested, documented assistant and a plan to share it safely.

7 h · 17 lessons Join the waitlist
bld-201 Intermediate Available

Building with LLM APIs Across Vendors

A hands-on, code-first course on the three major LLM APIs side by side: messages, streaming, structured outputs, tool calling, vision, cost, caching, batch, retries, abstraction layers, cloud routes and testing. You finish by building and hardening a small multi-vendor service.

14 h · 18 lessons See price at checkout
bld-210 Intermediate Available

Retrieval-Augmented Generation Systems

A practitioner's course on retrieval-augmented generation: when to use it, how to ingest, chunk, embed and index documents, how to combine BM25 and vector search with reranking, and how to measure, debug and operate the result. Every module includes working Python sketches.

14 h · 16 lessons See price at checkout