Frontier models from OpenAI, Anthropic and Google now accept images, PDFs and, in some cases, audio and video in the same request as text, and open-weight vision-language models such as Qwen-VL and Llama 4 can run on your own hardware. This course teaches how to use that capability in applications rather than demos. You will learn how each API tokenizes and prices images and pages, how resolution and cropping affect accuracy, how to extract structured data from photos, receipts and screenshots with a JSON schema, and how to ground answers in a region of an image.
Later modules cover document-level work (native PDF input, page images versus extracted text, tables and charts), audio (transcription with models such as OpenAI's gpt-transcribe, which replaces the deprecated gpt-4o-transcribe and whisper-1 API models, and open-weight Whisper run locally, diarization, and audio-native understanding in Gemini), and video (frame sampling versus native video input). The final module is about shipping: building a multimodal eval set, the characteristic failures (counting, small text, spatial reasoning, hallucinated reading of illegible content), privacy for images of people and documents, and cost control. Large-scale document ingestion pipelines are covered in dat-210. The course ends in an automated exam; passing earns a Wirewalk Institute of AI certificate of completion and assessment.