Wirewalk Instituteof AI
srv-210 Intermediate Waitlist

Deploying Models: APIs, Containers and Autoscaling

For ML and backend engineers who need to turn a trained model into a reliable, versioned, autoscaled HTTP or gRPC service on containers and Kubernetes.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

A model in a notebook is not a product. This course covers the engineering in between: choosing between real-time, batch and streaming inference; designing a request schema with FastAPI and Pydantic; deciding when a serving framework such as BentoML, NVIDIA Dynamo-Triton (formerly Triton Inference Server) or KServe earns its place; exporting to ONNX for portable, faster CPU and GPU inference; and building containers that start quickly and expose GPUs correctly through the NVIDIA Container Toolkit.

The second half is about running the service. You deploy on Kubernetes with resource requests, probes and GPU scheduling, autoscale with the Horizontal Pod Autoscaler, KEDA or Knative on signals that actually track load, handle cold starts and scale-to-zero, and release new model versions with canary and shadow traffic and a fast rollback. Managed options (Amazon SageMaker AI endpoints, Cloud Run with GPUs, Azure Machine Learning online endpoints) are compared on the same criteria. Product facts are stated as of October 2026.

What you will be able to do

  • Choose real-time, asynchronous, batch or streaming inference for a stated use case and justify the latency and cost trade-off
  • Design a versioned inference API with validated request and response schemas, health endpoints and clear error semantics
  • Export a model to ONNX and compare its latency and accuracy against the native framework
  • Build a minimal, reproducible GPU container image and run it with correct device and driver access
  • Deploy a model on Kubernetes with KServe, including probes, resource requests and GPU scheduling
  • Configure autoscaling on request concurrency or queue depth and mitigate cold starts
  • Roll out a new model version with canary or shadow traffic and a tested rollback path

Syllabus

6 modules · 18 lessons · about 14 hours

Module 1: Inference Patterns and API Design

Matching the serving pattern to the use case and designing an API clients can depend on.
3 lessons
  1. Real-Time, Asynchronous, Batch and Streaming Inference
  2. Designing Inference APIs with FastAPI and Pydantic
  3. REST versus gRPC, Payload Formats and the Open Inference Protocol

Module 2: Model Formats and Serving Frameworks

Portable model formats and the frameworks that add batching, multi-model hosting and metrics.
3 lessons
  1. Serialization and Portability: Pickle Risks, safetensors and ONNX
  2. Faster Inference with ONNX Runtime Execution Providers
  3. BentoML, Dynamo-Triton and KServe: When a Framework Earns Its Place

Module 3: Containerizing Models

Building images that are small, reproducible, secure and able to see the GPU.
3 lessons
  1. Multi-Stage Dockerfiles, Slim Base Images and Model Weight Handling
  2. GPU Containers: CUDA Base Images, Driver Compatibility and the NVIDIA Container Toolkit
  3. Image Scanning, Signing and Startup Time

Module 4: Running Model Services on Kubernetes

Deployments, probes, resources and GPU scheduling for inference workloads.
3 lessons
  1. Deployments, Services, Readiness Probes and Graceful Shutdown for Model Servers
  2. Scheduling GPUs: Device Plugins, Time-Slicing and MIG
  3. Serving with KServe InferenceServices

Module 5: Autoscaling and Cold Starts

Scaling on signals that track demand, and keeping latency acceptable while capacity arrives.
3 lessons
  1. Choosing a Scaling Signal: CPU, Concurrency, Queue Depth and Latency
  2. HPA, KEDA and Knative Scale-to-Zero in Practice
  3. Cold Start Budgets: Image Pulls, Weight Loading and Warm Pools

Module 6: Releasing and Operating Model Services

Safe rollouts, rollback, observability and the managed alternatives.
3 lessons
  1. Canary, Blue-Green and Shadow Deployments for Models
  2. Service Metrics, Prometheus and Latency SLOs
  3. Managed Endpoints Compared: SageMaker AI, Cloud Run and Azure ML Online Endpoints

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 20 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 45 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by FastAPI, Pydantic, BentoML, ONNX Runtime, NVIDIA Dynamo-Triton, Docker, NVIDIA Container Toolkit, Kubernetes, KServe, KEDA, Knative, Prometheus, Amazon SageMaker AI, Google Cloud Run, or Azure Machine Learning. Product and company names are trademarks of their respective owners.

More in Inference, Serving and MLOps

All 6 courses in this track
srv-201 Intermediate Waitlist

MLOps Fundamentals

The working practices that make machine learning reproducible and operable: versioning data, code and models together, tracking experiments, building pipelines, gating releases with CI/CD, and watching models for drift after they ship. Covers both classical ML and the newer LLM-application lifecycle.

14 h · 18 lessons Join the waitlist
srv-310 Expert Available

LLM Inference and Serving: vLLM, SGLang, TensorRT-LLM and Triton

The engineering of LLM inference from first principles to production: why decode is memory-bound, how to size the KV cache, how continuous batching, PagedAttention, prefix caching, quantization and speculative decoding change the economics, and how to run vLLM, SGLang, TensorRT-LLM and Dynamo at scale. Ends with load testing, Kubernetes autoscaling and a defensible cost per million tokens.

22 h · 19 lessons See price at checkout
srv-320 Expert Waitlist

Quantization and Inference Optimization

The numerics and engineering of making models cheaper to run: number formats from FP16 to FP4, post-training quantization algorithms such as GPTQ, AWQ and SmoothQuant, quantization-aware training, KV cache quantization, and the compiler and kernel work (torch.compile, TensorRT, fused attention, CUDA graphs) that turns smaller numbers into real speed. Every technique is paired with a way to measure what it costs in quality.

20 h · 20 lessons Join the waitlist