LLM Inference and Serving: vLLM, SGLang, TensorRT-LLM and Triton
For ML platform and infrastructure engineers who must run open-weight LLMs in production and leave able to size, tune, benchmark, scale and cost an inference service.
Level
Expert
Study time
22 hours
Lessons
19 in 6 modules
Exam
25 questions
About this course
Serving a large language model is a memory-bandwidth and scheduling problem far more than a compute problem, and most expensive mistakes in production come from not knowing that. This course builds the mental model first — prefill versus decode, the KV cache and the arithmetic that decides how many users a GPU can hold — and then walks through every major technique the engines use to push that boundary: continuous batching, PagedAttention, prefix caching, chunked prefill, FP8 and INT4 quantization, speculative decoding, multi-LoRA, tensor/pipeline/expert parallelism and disaggregated prefill/decode.
The second half is operational. You configure vLLM, SGLang, TensorRT-LLM (trtllm-serve), Triton Inference Server, NVIDIA Dynamo and llama.cpp with real commands; expose OpenAI-compatible endpoints; measure TTFT, inter-token latency, throughput and goodput with proper load tests; plan capacity against an SLO; autoscale on Kubernetes using queue-depth signals rather than GPU utilization; and compute the cost per million tokens of a self-hosted deployment so it can be compared honestly with API pricing. Engine facts are stated as of October 2026.
What you will be able to do
Calculate KV cache memory per token and per request for a given model and use it to predict the maximum concurrent batch a GPU configuration can hold
Explain how continuous batching, PagedAttention, prefix caching and chunked prefill change latency and throughput, and choose scheduler settings for a stated workload
Select a quantization scheme (FP8, INT8, INT4 AWQ/GPTQ, FP4) and a speculative decoding method for a serving goal, and verify the quality and speed impact
Choose between tensor, pipeline, expert and data parallelism and decide when disaggregated prefill/decode is worth its complexity
Deploy and tune an OpenAI-compatible endpoint with vLLM, SGLang, TensorRT-LLM or llama.cpp, and place Triton and Dynamo correctly in the architecture
Design and run a load test that reports TTFT, ITL, throughput and goodput against an explicit SLO
Produce a capacity plan and a Kubernetes autoscaling policy driven by queue depth and KV cache pressure
Compute cost per million tokens for a self-hosted deployment and compare it fairly with API pricing
Syllabus
6 modules · 19 lessons · about 22 hours
1
Module 1: The Physics of LLM Inference
Why prefill is compute-bound and decode is memory-bound, how big the KV cache really is, and how serving performance is measured.3 lessons
Prefill, Decode and Why Serving Is Memory-Bound
KV Cache Sizing: The Math That Decides Your Batch
Serving Metrics: TTFT, ITL, Throughput and Goodput
2
Module 2: Batching, Memory Management and Scheduling
How modern engines keep GPUs busy: continuous batching, paged KV memory, prefix reuse and the scheduler that decides who runs next.3 lessons
From Static to Dynamic to Continuous Batching
PagedAttention, Prefix Caching and KV Reuse
Chunked Prefill, Preemption and Scheduler Tuning
3
Module 3: Making Every Token Cheaper
Quantization, speculative decoding and multi-LoRA: the three levers that change cost per token on the same hardware.3 lessons
Quantization for Serving: FP8, INT8, INT4, AWQ and GPTQ
Speculative Decoding: Draft Models, EAGLE, MTP and N-gram
Multi-LoRA Serving: Many Adapters, One Base Model
4
Module 4: Scaling Out: Parallelism and Disaggregation
Splitting models and phases across GPUs and nodes, and routing traffic across replicas without wasting the cache.3 lessons
Tensor, Pipeline, Expert and Data Parallel Serving
Disaggregated Prefill and Decode
KV-Aware Routing and Multi-Replica Load Balancing
5
Module 5: The Engines in Practice
Hands-on configuration of the major engines, their OpenAI-compatible APIs, and how to choose between them.4 lessons
vLLM in Production: Configuration and the OpenAI-Compatible Server
SGLang: RadixAttention, Structured Output and the Router
TensorRT-LLM, Triton Inference Server and NVIDIA Dynamo
llama.cpp, Ollama and Choosing an Engine
6
Module 6: Operating Inference at Production Scale
Load testing, capacity planning, Kubernetes autoscaling and the cost model that tells you whether to self-host.3 lessons
Load Testing and Benchmarking Methodology
Capacity Planning and Kubernetes Autoscaling
Cost per Million Tokens: Self-Hosted versus API
Exam and certificate
The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.
Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.
The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by vLLM, SGLang, NVIDIA, TensorRT-LLM, Triton Inference Server, NVIDIA Dynamo, llama.cpp, Ollama, Kubernetes, KEDA, llm-d, or OpenAI. Product and company names are trademarks of their respective owners.
The working practices that make machine learning reproducible and operable: versioning data, code and models together, tracking experiments, building pipelines, gating releases with CI/CD, and watching models for drift after they ship. Covers both classical ML and the newer LLM-application lifecycle.
A practical path from a model file to a production service: designing the inference API, packaging it in a lean GPU-ready container, running it on Kubernetes with KServe, autoscaling it on the right signal, and rolling out new versions safely. Focused on general ML models; LLM-specific engines are covered in srv-310.
The numerics and engineering of making models cheaper to run: number formats from FP16 to FP4, post-training quantization algorithms such as GPTQ, AWQ and SmoothQuant, quantization-aware training, KV cache quantization, and the compiler and kernel work (torch.compile, TensorRT, fused attention, CUDA graphs) that turns smaller numbers into real speed. Every technique is paired with a way to measure what it costs in quality.