Wirewalk Instituteof AI
srv-310 Expert AvailableFlagship

LLM Inference and Serving: vLLM, SGLang, TensorRT-LLM and Triton

For ML platform and infrastructure engineers who must run open-weight LLMs in production and leave able to size, tune, benchmark, scale and cost an inference service.

Level
Expert
Study time
22 hours
Lessons
19 in 6 modules
Exam
25 questions

About this course

Serving a large language model is a memory-bandwidth and scheduling problem far more than a compute problem, and most expensive mistakes in production come from not knowing that. This course builds the mental model first — prefill versus decode, the KV cache and the arithmetic that decides how many users a GPU can hold — and then walks through every major technique the engines use to push that boundary: continuous batching, PagedAttention, prefix caching, chunked prefill, FP8 and INT4 quantization, speculative decoding, multi-LoRA, tensor/pipeline/expert parallelism and disaggregated prefill/decode.

The second half is operational. You configure vLLM, SGLang, TensorRT-LLM (trtllm-serve), Triton Inference Server, NVIDIA Dynamo and llama.cpp with real commands; expose OpenAI-compatible endpoints; measure TTFT, inter-token latency, throughput and goodput with proper load tests; plan capacity against an SLO; autoscale on Kubernetes using queue-depth signals rather than GPU utilization; and compute the cost per million tokens of a self-hosted deployment so it can be compared honestly with API pricing. Engine facts are stated as of October 2026.

What you will be able to do

  • Calculate KV cache memory per token and per request for a given model and use it to predict the maximum concurrent batch a GPU configuration can hold
  • Explain how continuous batching, PagedAttention, prefix caching and chunked prefill change latency and throughput, and choose scheduler settings for a stated workload
  • Select a quantization scheme (FP8, INT8, INT4 AWQ/GPTQ, FP4) and a speculative decoding method for a serving goal, and verify the quality and speed impact
  • Choose between tensor, pipeline, expert and data parallelism and decide when disaggregated prefill/decode is worth its complexity
  • Deploy and tune an OpenAI-compatible endpoint with vLLM, SGLang, TensorRT-LLM or llama.cpp, and place Triton and Dynamo correctly in the architecture
  • Design and run a load test that reports TTFT, ITL, throughput and goodput against an explicit SLO
  • Produce a capacity plan and a Kubernetes autoscaling policy driven by queue depth and KV cache pressure
  • Compute cost per million tokens for a self-hosted deployment and compare it fairly with API pricing

Syllabus

6 modules · 19 lessons · about 22 hours

Module 1: The Physics of LLM Inference

Why prefill is compute-bound and decode is memory-bound, how big the KV cache really is, and how serving performance is measured.
3 lessons
  1. Prefill, Decode and Why Serving Is Memory-Bound
  2. KV Cache Sizing: The Math That Decides Your Batch
  3. Serving Metrics: TTFT, ITL, Throughput and Goodput

Module 2: Batching, Memory Management and Scheduling

How modern engines keep GPUs busy: continuous batching, paged KV memory, prefix reuse and the scheduler that decides who runs next.
3 lessons
  1. From Static to Dynamic to Continuous Batching
  2. PagedAttention, Prefix Caching and KV Reuse
  3. Chunked Prefill, Preemption and Scheduler Tuning

Module 3: Making Every Token Cheaper

Quantization, speculative decoding and multi-LoRA: the three levers that change cost per token on the same hardware.
3 lessons
  1. Quantization for Serving: FP8, INT8, INT4, AWQ and GPTQ
  2. Speculative Decoding: Draft Models, EAGLE, MTP and N-gram
  3. Multi-LoRA Serving: Many Adapters, One Base Model

Module 4: Scaling Out: Parallelism and Disaggregation

Splitting models and phases across GPUs and nodes, and routing traffic across replicas without wasting the cache.
3 lessons
  1. Tensor, Pipeline, Expert and Data Parallel Serving
  2. Disaggregated Prefill and Decode
  3. KV-Aware Routing and Multi-Replica Load Balancing

Module 5: The Engines in Practice

Hands-on configuration of the major engines, their OpenAI-compatible APIs, and how to choose between them.
4 lessons
  1. vLLM in Production: Configuration and the OpenAI-Compatible Server
  2. SGLang: RadixAttention, Structured Output and the Router
  3. TensorRT-LLM, Triton Inference Server and NVIDIA Dynamo
  4. llama.cpp, Ollama and Choosing an Engine

Module 6: Operating Inference at Production Scale

Load testing, capacity planning, Kubernetes autoscaling and the cost model that tells you whether to self-host.
3 lessons
  1. Load Testing and Benchmarking Methodology
  2. Capacity Planning and Kubernetes Autoscaling
  3. Cost per Million Tokens: Self-Hosted versus API

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by vLLM, SGLang, NVIDIA, TensorRT-LLM, Triton Inference Server, NVIDIA Dynamo, llama.cpp, Ollama, Kubernetes, KEDA, llm-d, or OpenAI. Product and company names are trademarks of their respective owners.

More in Inference, Serving and MLOps

All 6 courses in this track
srv-201 Intermediate Waitlist

MLOps Fundamentals

The working practices that make machine learning reproducible and operable: versioning data, code and models together, tracking experiments, building pipelines, gating releases with CI/CD, and watching models for drift after they ship. Covers both classical ML and the newer LLM-application lifecycle.

14 h · 18 lessons Join the waitlist
srv-210 Intermediate Waitlist

Deploying Models: APIs, Containers and Autoscaling

A practical path from a model file to a production service: designing the inference API, packaging it in a lean GPU-ready container, running it on Kubernetes with KServe, autoscaling it on the right signal, and rolling out new versions safely. Focused on general ML models; LLM-specific engines are covered in srv-310.

14 h · 18 lessons Join the waitlist
srv-320 Expert Waitlist

Quantization and Inference Optimization

The numerics and engineering of making models cheaper to run: number formats from FP16 to FP4, post-training quantization algorithms such as GPTQ, AWQ and SmoothQuant, quantization-aware training, KV cache quantization, and the compiler and kernel work (torch.compile, TensorRT, fused attention, CUDA graphs) that turns smaller numbers into real speed. Every technique is paired with a way to measure what it costs in quality.

20 h · 20 lessons Join the waitlist