For engineers who must run models on phones, laptops, embedded boards and edge servers, and leave able to choose a runtime, compress a model to fit, ship it, and keep it updated in the field.
The curriculum is being prepared and reviewed. Enrollment is not open yet.
About this course
On-device inference trades the data center's abundance for privacy, offline operation, low latency and zero per-token cost — but only if the model fits the device's memory, thermal and battery budget. This course teaches that trade-off and the engineering behind it. You start with a decision framework for on-device, edge-server and hybrid designs, then work through the compression and conversion path: quantizing and pruning for the target, exporting with torch.export, and lowering to ExecuTorch, LiteRT, Core ML, ONNX Runtime and OpenVINO, including how NPU delegates and execution providers decide which operators actually run on accelerated hardware.
Dedicated modules cover small language models on device — llama.cpp and GGUF, MLC LLM, MLX on Apple silicon, and the platform models exposed through Apple's Foundation Models framework and Android's Gemini Nano via ML Kit — and edge vision and sensor pipelines on NVIDIA Jetson with TensorRT and DeepStream. App-level integration in Swift and Kotlin is covered in dev-211; this course is about the model, runtime and fleet engineering beneath it. The course ends with operations: benchmarking on real devices, fleet model delivery and rollback, protecting model files, and hybrid routing between device and cloud. Platform facts are stated as of October 2026; several mobile AI APIs change at each annual OS release.
What you will be able to do
Decide whether a feature should run on device, on an edge server, in the cloud or as a hybrid, using latency, privacy, cost and power criteria
Estimate whether a model fits a target device's memory, compute and energy budget before converting it
Export and convert a PyTorch model to ExecuTorch, LiteRT, Core ML or ONNX and resolve unsupported-operator and delegate fallbacks
Select quantization and compression settings appropriate to a specific NPU, GPU or CPU target
Run a small language model locally with llama.cpp, MLC LLM or a platform model API and manage context, memory and thermal limits
Benchmark latency, throughput and energy on real hardware and interpret the results
Design a fleet update, rollback and model-protection process for deployed edge devices
Syllabus
6 modules · 20 lessons · about 18 hours
1
Module 1: Deciding What Runs at the Edge
The constraints and trade-offs that determine where inference should happen.3 lessons
On-Device, Edge Server, Cloud or Hybrid: A Decision Framework
Budgets That Bind: Memory, Bandwidth, Thermals and Battery
Estimating Fit Before You Convert: Parameters, Activations and Operators
2
Module 2: The Export and Conversion Path
Getting a model out of training code and into a portable, accelerator-aware representation.3 lessons
torch.export and Graph Capture Pitfalls
Delegates, Execution Providers and Partitioning: Why Operators Fall Back to CPU
Compression for the Target: Static INT8, Weight-Only INT4 and Pruning on NPUs
3
Module 3: Mobile and Desktop Runtimes
The major on-device runtimes and how to target each platform's accelerators.4 lessons
ExecuTorch: Backends for CPU, GPU and NPU Targets
Android: LiteRT, the NNAPI Deprecation and Vendor NPU Acceleration
Apple Platforms: Core ML Conversion, the Neural Engine and MLX
Windows and PCs: ONNX Runtime, Windows ML and OpenVINO
4
Module 4: Small Language Models on Device
Running and integrating local LLMs within the limits of a phone or laptop.3 lessons
llama.cpp, GGUF and MLC LLM for Local Inference
Platform Models: Apple Foundation Models Framework and Gemini Nano via ML Kit
Context, KV Cache and Thermal Throttling on Mobile Hardware
5
Module 5: Edge Vision and Embedded Systems
Real-time perception pipelines on edge servers and embedded boards.3 lessons
Jetson and TensorRT: Building an Edge Vision Pipeline
Video Analytics Pipelines with DeepStream and GStreamer
Microcontrollers and TinyML: What Fits in Kilobytes
6
Module 6: Operating Edge AI in the Field
Measuring on real devices, shipping updates safely and protecting the model.4 lessons
Benchmarking on Real Devices: Latency, Energy and Device Farms
Fleet Model Delivery, A/B Rollout and Rollback
Protecting Model Files and Securing Edge Endpoints
Hybrid Routing Between Device and Cloud
Exam and certificate
The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.
Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.
The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by PyTorch ExecuTorch, Google LiteRT, Google ML Kit, Gemini Nano, Apple Core ML, Apple Foundation Models framework, Apple MLX, ONNX Runtime, Windows ML, Intel OpenVINO, Qualcomm AI Hub, NVIDIA Jetson, NVIDIA TensorRT, NVIDIA DeepStream, llama.cpp, MLC LLM, or Arm. Product and company names are trademarks of their respective owners.
The working practices that make machine learning reproducible and operable: versioning data, code and models together, tracking experiments, building pipelines, gating releases with CI/CD, and watching models for drift after they ship. Covers both classical ML and the newer LLM-application lifecycle.
A practical path from a model file to a production service: designing the inference API, packaging it in a lean GPU-ready container, running it on Kubernetes with KServe, autoscaling it on the right signal, and rolling out new versions safely. Focused on general ML models; LLM-specific engines are covered in srv-310.
The engineering of LLM inference from first principles to production: why decode is memory-bound, how to size the KV cache, how continuous batching, PagedAttention, prefix caching, quantization and speculative decoding change the economics, and how to run vLLM, SGLang, TensorRT-LLM and Dynamo at scale. Ends with load testing, Kubernetes autoscaling and a defensible cost per million tokens.