Wirewalk Instituteof AI
bld-250 Intermediate Waitlist

Multimodal Applications: Vision, Documents and Audio

For engineers building features that read images, screenshots, PDFs, audio and video: send the right input to the right model, get structured results back, and know where multimodal models still fail.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

Frontier models from OpenAI, Anthropic and Google now accept images, PDFs and, in some cases, audio and video in the same request as text, and open-weight vision-language models such as Qwen-VL and Llama 4 can run on your own hardware. This course teaches how to use that capability in applications rather than demos. You will learn how each API tokenizes and prices images and pages, how resolution and cropping affect accuracy, how to extract structured data from photos, receipts and screenshots with a JSON schema, and how to ground answers in a region of an image.

Later modules cover document-level work (native PDF input, page images versus extracted text, tables and charts), audio (transcription with models such as OpenAI's gpt-transcribe, which replaces the deprecated gpt-4o-transcribe and whisper-1 API models, and open-weight Whisper run locally, diarization, and audio-native understanding in Gemini), and video (frame sampling versus native video input). The final module is about shipping: building a multimodal eval set, the characteristic failures (counting, small text, spatial reasoning, hallucinated reading of illegible content), privacy for images of people and documents, and cost control. Large-scale document ingestion pipelines are covered in dat-210. The course ends in an automated exam; passing earns a Wirewalk Institute of AI certificate of completion and assessment.

What you will be able to do

  • Estimate token cost and latency for image, PDF, audio and video inputs across major APIs and choose resolution or sampling accordingly
  • Extract validated structured data from photos, screenshots, receipts and forms using JSON schemas and confidence checks
  • Choose between native PDF input, page images and text extraction for a given document type and justify the choice
  • Build a transcription and audio-understanding pipeline with diarization and timestamped output
  • Analyze video by frame sampling or native video input and tie answers to timestamps
  • Compare frontier multimodal APIs with self-hosted open-weight vision-language models on accuracy, cost and data control
  • Build a multimodal evaluation set that targets known failure modes such as counting, small text and spatial reasoning

Syllabus

6 modules · 17 lessons · about 13 hours

Module 1: How Multimodal Models See and Hear

The mechanics that decide accuracy and cost for non-text inputs.
3 lessons
  1. Vision Encoders, Image Tokens and Why Resolution Matters
  2. Image Inputs Across OpenAI, Claude and Gemini: Limits, Detail Settings and Pricing
  3. Open-Weight Vision-Language Models: Qwen-VL, Llama 4 and Gemma

Module 2: Image Understanding in Applications

From a photo or screenshot to a validated, structured result.
3 lessons
  1. Structured Extraction from Photos, Receipts and Screenshots
  2. Charts, Diagrams and UI Screenshots: Reading Visual Structure
  3. Grounding and Localization: Bounding Boxes and Region References

Module 3: Documents as Multimodal Input

Getting answers from PDFs, scans and forms without losing layout.
3 lessons
  1. Native PDF Input vs Page Images vs Extracted Text
  2. Tables, Handwriting and Multi-Page Forms
  3. Dedicated OCR Models Alongside General Models

Module 4: Audio: Transcription and Understanding

Turning speech and sound into text, structure and answers.
3 lessons
  1. Speech-to-Text with gpt-transcribe, Open-Weight Whisper and Hosted Alternatives
  2. Diarization, Timestamps and Long Recordings
  3. Audio-Native Understanding: Tone, Events and Question Answering

Module 5: Video

Answering questions about what happens, and when.
2 lessons
  1. Frame Sampling Strategies and Their Trade-offs
  2. Native Video Input with Gemini and Timestamped Answers

Module 6: Shipping Multimodal Features

Evaluation, failure modes, privacy and cost.
3 lessons
  1. Characteristic Failures: Counting, Small Text, Spatial Reasoning and Invented Readings
  2. Building a Multimodal Eval Set and Scoring It
  3. Privacy for Faces, IDs and Documents; Cost Controls for Media Inputs

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 20 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 40 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by OpenAI, Anthropic, Google, Mistral AI, Meta, or Alibaba Qwen. Product and company names are trademarks of their respective owners.

More in Building AI Applications

All 14 courses in this track
bld-101 Basic Waitlist

No-Code AI Assistants: Custom GPTs, Projects, Gems and Copilot Studio

A hands-on course on building no-code assistants with the builders already in your workplace tools: ChatGPT projects, custom GPTs and their successor plugins, Claude Projects and Skills, Gemini Gems, and Microsoft 365 Copilot Agent Builder and Copilot Studio. You leave with a tested, documented assistant and a plan to share it safely.

7 h · 17 lessons Join the waitlist
bld-201 Intermediate Available

Building with LLM APIs Across Vendors

A hands-on, code-first course on the three major LLM APIs side by side: messages, streaming, structured outputs, tool calling, vision, cost, caching, batch, retries, abstraction layers, cloud routes and testing. You finish by building and hardening a small multi-vendor service.

14 h · 18 lessons See price at checkout
bld-210 Intermediate Available

Retrieval-Augmented Generation Systems

A practitioner's course on retrieval-augmented generation: when to use it, how to ingest, chunk, embed and index documents, how to combine BM25 and vector search with reranking, and how to measure, debug and operate the result. Every module includes working Python sketches.

14 h · 16 lessons See price at checkout