Deploying Models: APIs, Containers and Autoscaling
For ML and backend engineers who need to turn a trained model into a reliable, versioned, autoscaled HTTP or gRPC service on containers and Kubernetes.
The curriculum is being prepared and reviewed. Enrollment is not open yet.
About this course
A model in a notebook is not a product. This course covers the engineering in between: choosing between real-time, batch and streaming inference; designing a request schema with FastAPI and Pydantic; deciding when a serving framework such as BentoML, NVIDIA Dynamo-Triton (formerly Triton Inference Server) or KServe earns its place; exporting to ONNX for portable, faster CPU and GPU inference; and building containers that start quickly and expose GPUs correctly through the NVIDIA Container Toolkit.
The second half is about running the service. You deploy on Kubernetes with resource requests, probes and GPU scheduling, autoscale with the Horizontal Pod Autoscaler, KEDA or Knative on signals that actually track load, handle cold starts and scale-to-zero, and release new model versions with canary and shadow traffic and a fast rollback. Managed options (Amazon SageMaker AI endpoints, Cloud Run with GPUs, Azure Machine Learning online endpoints) are compared on the same criteria. Product facts are stated as of October 2026.
What you will be able to do
Choose real-time, asynchronous, batch or streaming inference for a stated use case and justify the latency and cost trade-off
Design a versioned inference API with validated request and response schemas, health endpoints and clear error semantics
Export a model to ONNX and compare its latency and accuracy against the native framework
Build a minimal, reproducible GPU container image and run it with correct device and driver access
Deploy a model on Kubernetes with KServe, including probes, resource requests and GPU scheduling
Configure autoscaling on request concurrency or queue depth and mitigate cold starts
Roll out a new model version with canary or shadow traffic and a tested rollback path
Syllabus
6 modules · 18 lessons · about 14 hours
1
Module 1: Inference Patterns and API Design
Matching the serving pattern to the use case and designing an API clients can depend on.3 lessons
Real-Time, Asynchronous, Batch and Streaming Inference
Designing Inference APIs with FastAPI and Pydantic
REST versus gRPC, Payload Formats and the Open Inference Protocol
2
Module 2: Model Formats and Serving Frameworks
Portable model formats and the frameworks that add batching, multi-model hosting and metrics.3 lessons
Serialization and Portability: Pickle Risks, safetensors and ONNX
Faster Inference with ONNX Runtime Execution Providers
BentoML, Dynamo-Triton and KServe: When a Framework Earns Its Place
3
Module 3: Containerizing Models
Building images that are small, reproducible, secure and able to see the GPU.3 lessons
Multi-Stage Dockerfiles, Slim Base Images and Model Weight Handling
GPU Containers: CUDA Base Images, Driver Compatibility and the NVIDIA Container Toolkit
Image Scanning, Signing and Startup Time
4
Module 4: Running Model Services on Kubernetes
Deployments, probes, resources and GPU scheduling for inference workloads.3 lessons
Deployments, Services, Readiness Probes and Graceful Shutdown for Model Servers
Scheduling GPUs: Device Plugins, Time-Slicing and MIG
Serving with KServe InferenceServices
5
Module 5: Autoscaling and Cold Starts
Scaling on signals that track demand, and keeping latency acceptable while capacity arrives.3 lessons
Choosing a Scaling Signal: CPU, Concurrency, Queue Depth and Latency
HPA, KEDA and Knative Scale-to-Zero in Practice
Cold Start Budgets: Image Pulls, Weight Loading and Warm Pools
6
Module 6: Releasing and Operating Model Services
Safe rollouts, rollback, observability and the managed alternatives.3 lessons
Canary, Blue-Green and Shadow Deployments for Models
Service Metrics, Prometheus and Latency SLOs
Managed Endpoints Compared: SageMaker AI, Cloud Run and Azure ML Online Endpoints
Exam and certificate
The exam unlocks once you have completed every lesson. It draws 20 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 45 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.
Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.
The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by FastAPI, Pydantic, BentoML, ONNX Runtime, NVIDIA Dynamo-Triton, Docker, NVIDIA Container Toolkit, Kubernetes, KServe, KEDA, Knative, Prometheus, Amazon SageMaker AI, Google Cloud Run, or Azure Machine Learning. Product and company names are trademarks of their respective owners.
The working practices that make machine learning reproducible and operable: versioning data, code and models together, tracking experiments, building pipelines, gating releases with CI/CD, and watching models for drift after they ship. Covers both classical ML and the newer LLM-application lifecycle.
The engineering of LLM inference from first principles to production: why decode is memory-bound, how to size the KV cache, how continuous batching, PagedAttention, prefix caching, quantization and speculative decoding change the economics, and how to run vLLM, SGLang, TensorRT-LLM and Dynamo at scale. Ends with load testing, Kubernetes autoscaling and a defensible cost per million tokens.
The numerics and engineering of making models cheaper to run: number formats from FP16 to FP4, post-training quantization algorithms such as GPTQ, AWQ and SmoothQuant, quantization-aware training, KV cache quantization, and the compiler and kernel work (torch.compile, TensorRT, fused attention, CUDA graphs) that turns smaller numbers into real speed. Every technique is paired with a way to measure what it costs in quality.