Wirewalk Instituteof AI
srv-320 Expert Waitlist

Quantization and Inference Optimization

For ML and performance engineers who must make a model smaller and faster without losing the quality that matters, and prove it with measurements.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

Quantization is the single biggest lever on inference cost, and also the easiest way to ship a model that is quietly worse. This course starts with the numerics — integer and floating-point formats, scales and zero points, per-tensor, per-channel and group-wise granularity, and the microscaling block formats (MXFP4, MXFP8, NVFP4) supported by recent NVIDIA and AMD accelerators — and explains why outlier activations make LLMs hard to quantize.

You then apply the main post-training methods (round-to-nearest, GPTQ, AWQ, SmoothQuant, rotation-based methods) with LLM Compressor, NVIDIA Model Optimizer (formerly TensorRT Model Optimizer) and torchao, compare them with quantization-aware training, and produce checkpoints in the formats engines load: compressed-tensors, Hugging Face, GGUF and ONNX. The second half covers optimization beyond quantization — graph compilation, kernel fusion, FlashAttention-style kernels, CUDA graphs and custom kernels in the Triton language — and closes with an evaluation protocol that catches regressions perplexity misses. Engine-level serving (batching, scheduling, speculative decoding) is covered in srv-310, and pruning and distillation as model-design techniques in mod-340. Format and tool support is stated as of October 2026.

What you will be able to do

  • Explain how scale, zero point, granularity and format (INT8, INT4, FP8 E4M3/E5M2, MXFP4, NVFP4) determine quantization error and hardware support
  • Choose between weight-only, weight-and-activation and KV cache quantization for a memory-bound or compute-bound workload
  • Apply GPTQ, AWQ and SmoothQuant with a calibration set and explain why each method handles outliers differently
  • Decide when quantization-aware training is worth its cost over post-training quantization
  • Produce quantized checkpoints in the format a target engine loads and verify that it loads the intended kernels
  • Use graph compilation, kernel fusion and CUDA graphs to remove overhead, and identify the bottleneck with a profiler before optimizing
  • Design an evaluation protocol that detects task-level quality loss from quantization, not only perplexity change

Syllabus

6 modules · 20 lessons · about 20 hours

Module 1: Numerics of Quantization

Number formats, mapping functions and granularity, and why hardware support decides which formats are worth using.
3 lessons
  1. Integer and Floating-Point Formats: INT8, INT4, FP8 E4M3/E5M2 and BF16
  2. Scales, Zero Points and Granularity: Per-Tensor, Per-Channel and Group-Wise
  3. Microscaling Block Formats: MXFP8, MXFP4 and NVFP4 on Current Accelerators

Module 2: What to Quantize and Why It Is Hard

Deciding between weights, activations and KV cache, and understanding the outliers that break naive methods.
3 lessons
  1. Memory-Bound versus Compute-Bound: Choosing Weight-Only or W8A8
  2. Activation Outliers and Sensitive Layers in Transformers
  3. KV Cache Quantization and Its Effect on Long-Context Quality

Module 3: Post-Training Quantization Algorithms

The main PTQ methods, how they use calibration data, and how to run them with current tooling.
4 lessons
  1. Round-to-Nearest, Calibration Sets and Error Measurement
  2. GPTQ and AWQ: Second-Order and Activation-Aware Weight Quantization
  3. SmoothQuant and Rotation-Based Methods for W8A8 and W4A4
  4. Running PTQ with LLM Compressor, NVIDIA Model Optimizer and torchao

Module 4: Training-Aware Compression and Model Formats

When to train for quantization, how sparsity fits in, and the checkpoint formats each runtime expects.
3 lessons
  1. Quantization-Aware Training and Fake Quantization
  2. 2:4 Structured Sparsity: Hardware Support, Tooling Gaps and When to Skip It
  3. Checkpoint Formats: compressed-tensors, GGUF K-Quants, ONNX and TensorRT Engines

Module 5: Compilers, Kernels and Runtime Overheads

Turning fewer bits into real speed: profiling, compilation, fusion and custom kernels.
4 lessons
  1. Profiling Inference with PyTorch Profiler and Nsight Systems
  2. torch.compile, TensorRT and ONNX Runtime Graph Optimizations
  3. Fused Attention Kernels, CUDA Graphs and Launch Overhead
  4. Writing a Fused Dequantize-and-Matmul Kernel in Triton

Module 6: Proving the Optimization Was Worth It

Measuring quality loss and speed gain honestly, and deciding what to ship.
3 lessons
  1. Quality Evaluation Beyond Perplexity: Task Evals, Long Context and Tool Calling
  2. Microbenchmarks versus End-to-End Benchmarks
  3. A Decision Framework for Shipping a Quantized Model

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by NVIDIA, TensorRT, NVIDIA Model Optimizer, vLLM, LLM Compressor, PyTorch, torchao, Hugging Face, bitsandbytes, llama.cpp, ONNX Runtime, Triton (triton-lang), or AMD. Product and company names are trademarks of their respective owners.

More in Inference, Serving and MLOps

All 6 courses in this track
srv-201 Intermediate Waitlist

MLOps Fundamentals

The working practices that make machine learning reproducible and operable: versioning data, code and models together, tracking experiments, building pipelines, gating releases with CI/CD, and watching models for drift after they ship. Covers both classical ML and the newer LLM-application lifecycle.

14 h · 18 lessons Join the waitlist
srv-210 Intermediate Waitlist

Deploying Models: APIs, Containers and Autoscaling

A practical path from a model file to a production service: designing the inference API, packaging it in a lean GPU-ready container, running it on Kubernetes with KServe, autoscaling it on the right signal, and rolling out new versions safely. Focused on general ML models; LLM-specific engines are covered in srv-310.

14 h · 18 lessons Join the waitlist
srv-310 Expert Available

LLM Inference and Serving: vLLM, SGLang, TensorRT-LLM and Triton

The engineering of LLM inference from first principles to production: why decode is memory-bound, how to size the KV cache, how continuous batching, PagedAttention, prefix caching, quantization and speculative decoding change the economics, and how to run vLLM, SGLang, TensorRT-LLM and Dynamo at scale. Ends with load testing, Kubernetes autoscaling and a defensible cost per million tokens.

22 h · 19 lessons See price at checkout