Wirewalk Instituteof AI
srv-340 Expert Waitlist

Edge and On-Device AI

For engineers who must run models on phones, laptops, embedded boards and edge servers, and leave able to choose a runtime, compress a model to fit, ship it, and keep it updated in the field.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

On-device inference trades the data center's abundance for privacy, offline operation, low latency and zero per-token cost — but only if the model fits the device's memory, thermal and battery budget. This course teaches that trade-off and the engineering behind it. You start with a decision framework for on-device, edge-server and hybrid designs, then work through the compression and conversion path: quantizing and pruning for the target, exporting with torch.export, and lowering to ExecuTorch, LiteRT, Core ML, ONNX Runtime and OpenVINO, including how NPU delegates and execution providers decide which operators actually run on accelerated hardware.

Dedicated modules cover small language models on device — llama.cpp and GGUF, MLC LLM, MLX on Apple silicon, and the platform models exposed through Apple's Foundation Models framework and Android's Gemini Nano via ML Kit — and edge vision and sensor pipelines on NVIDIA Jetson with TensorRT and DeepStream. App-level integration in Swift and Kotlin is covered in dev-211; this course is about the model, runtime and fleet engineering beneath it. The course ends with operations: benchmarking on real devices, fleet model delivery and rollback, protecting model files, and hybrid routing between device and cloud. Platform facts are stated as of October 2026; several mobile AI APIs change at each annual OS release.

What you will be able to do

  • Decide whether a feature should run on device, on an edge server, in the cloud or as a hybrid, using latency, privacy, cost and power criteria
  • Estimate whether a model fits a target device's memory, compute and energy budget before converting it
  • Export and convert a PyTorch model to ExecuTorch, LiteRT, Core ML or ONNX and resolve unsupported-operator and delegate fallbacks
  • Select quantization and compression settings appropriate to a specific NPU, GPU or CPU target
  • Run a small language model locally with llama.cpp, MLC LLM or a platform model API and manage context, memory and thermal limits
  • Benchmark latency, throughput and energy on real hardware and interpret the results
  • Design a fleet update, rollback and model-protection process for deployed edge devices

Syllabus

6 modules · 20 lessons · about 18 hours

Module 1: Deciding What Runs at the Edge

The constraints and trade-offs that determine where inference should happen.
3 lessons
  1. On-Device, Edge Server, Cloud or Hybrid: A Decision Framework
  2. Budgets That Bind: Memory, Bandwidth, Thermals and Battery
  3. Estimating Fit Before You Convert: Parameters, Activations and Operators

Module 2: The Export and Conversion Path

Getting a model out of training code and into a portable, accelerator-aware representation.
3 lessons
  1. torch.export and Graph Capture Pitfalls
  2. Delegates, Execution Providers and Partitioning: Why Operators Fall Back to CPU
  3. Compression for the Target: Static INT8, Weight-Only INT4 and Pruning on NPUs

Module 3: Mobile and Desktop Runtimes

The major on-device runtimes and how to target each platform's accelerators.
4 lessons
  1. ExecuTorch: Backends for CPU, GPU and NPU Targets
  2. Android: LiteRT, the NNAPI Deprecation and Vendor NPU Acceleration
  3. Apple Platforms: Core ML Conversion, the Neural Engine and MLX
  4. Windows and PCs: ONNX Runtime, Windows ML and OpenVINO

Module 4: Small Language Models on Device

Running and integrating local LLMs within the limits of a phone or laptop.
3 lessons
  1. llama.cpp, GGUF and MLC LLM for Local Inference
  2. Platform Models: Apple Foundation Models Framework and Gemini Nano via ML Kit
  3. Context, KV Cache and Thermal Throttling on Mobile Hardware

Module 5: Edge Vision and Embedded Systems

Real-time perception pipelines on edge servers and embedded boards.
3 lessons
  1. Jetson and TensorRT: Building an Edge Vision Pipeline
  2. Video Analytics Pipelines with DeepStream and GStreamer
  3. Microcontrollers and TinyML: What Fits in Kilobytes

Module 6: Operating Edge AI in the Field

Measuring on real devices, shipping updates safely and protecting the model.
4 lessons
  1. Benchmarking on Real Devices: Latency, Energy and Device Farms
  2. Fleet Model Delivery, A/B Rollout and Rollback
  3. Protecting Model Files and Securing Edge Endpoints
  4. Hybrid Routing Between Device and Cloud

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by PyTorch ExecuTorch, Google LiteRT, Google ML Kit, Gemini Nano, Apple Core ML, Apple Foundation Models framework, Apple MLX, ONNX Runtime, Windows ML, Intel OpenVINO, Qualcomm AI Hub, NVIDIA Jetson, NVIDIA TensorRT, NVIDIA DeepStream, llama.cpp, MLC LLM, or Arm. Product and company names are trademarks of their respective owners.

More in Inference, Serving and MLOps

All 6 courses in this track
srv-201 Intermediate Waitlist

MLOps Fundamentals

The working practices that make machine learning reproducible and operable: versioning data, code and models together, tracking experiments, building pipelines, gating releases with CI/CD, and watching models for drift after they ship. Covers both classical ML and the newer LLM-application lifecycle.

14 h · 18 lessons Join the waitlist
srv-210 Intermediate Waitlist

Deploying Models: APIs, Containers and Autoscaling

A practical path from a model file to a production service: designing the inference API, packaging it in a lean GPU-ready container, running it on Kubernetes with KServe, autoscaling it on the right signal, and rolling out new versions safely. Focused on general ML models; LLM-specific engines are covered in srv-310.

14 h · 18 lessons Join the waitlist
srv-310 Expert Available

LLM Inference and Serving: vLLM, SGLang, TensorRT-LLM and Triton

The engineering of LLM inference from first principles to production: why decode is memory-bound, how to size the KV cache, how continuous batching, PagedAttention, prefix caching, quantization and speculative decoding change the economics, and how to run vLLM, SGLang, TensorRT-LLM and Dynamo at scale. Ends with load testing, Kubernetes autoscaling and a defensible cost per million tokens.

22 h · 19 lessons See price at checkout