Wirewalk Instituteof AI
mod-385 Expert Waitlist

Training Multimodal Models

For ML engineers who build or adapt vision-language and other multimodal models and need to choose an architecture, assemble image-text data, run the pretraining and instruction-tuning stages, and evaluate the result.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

Most frontier and open-weight models now accept images, and many accept video and audio. Under the hood, nearly all of them combine a pretrained encoder for each modality with a language model, then train the combination in stages: align the representations, teach broad multimodal understanding, then instruction-tune for the tasks users care about. The design choices along the way, such as how images become tokens, which parameters are frozen, how resolution is handled and what data is used, decide both quality and cost.

This course walks the full pipeline with open tools and open recipes. You train and evaluate a contrastive image-text encoder and understand why SigLIP's sigmoid loss displaced softmax CLIP training at scale. You study connector architectures from LLaVA's projector to cross-attention and native-resolution designs in the Qwen-VL family, and contrast them with early-fusion models that tokenize images directly. You build an image-text data pipeline with DataComp-style filtering, recaptioning and dense-caption data such as PixMo, then run alignment pretraining and visual instruction tuning with Hugging Face Transformers and TRL, including LoRA fine-tuning of an open VLM on document and chart data. Later modules extend the recipe to video and speech, then cover evaluation with VLMEvalKit and lmms-eval, object-hallucination testing, and preference tuning for multimodal outputs. Image and video generation are covered in mod-350. Model and library facts are current as of October 2026.

What you will be able to do

  • Compare contrastive, connector-based and early-fusion multimodal architectures and pick one for a given budget and task
  • Train and evaluate a CLIP- or SigLIP-style contrastive encoder with OpenCLIP and zero-shot retrieval metrics
  • Design the vision-language connector, token budget and resolution strategy for a VLM
  • Build an image-text training corpus with filtering, deduplication, recaptioning and license review
  • Run staged alignment pretraining and visual instruction tuning and choose what to freeze at each stage
  • Fine-tune an open VLM with LoRA for documents, charts or domain imagery
  • Extend a vision-language recipe to video frames and audio inputs and manage their token cost
  • Evaluate a multimodal model for accuracy, grounding and hallucination and detect benchmark contamination

Syllabus

6 modules · 21 lessons · about 22 hours

Module 1: Multimodal architectures

The main ways to combine modalities and the trade-offs between them.
3 lessons
  1. Encoders, connectors and decoders: a map of multimodal model designs
  2. Images as tokens: patching, resolution, tiling and token budgets
  3. Early fusion and native multimodality: Chameleon-style tokenized images versus adapters

Module 2: Contrastive image-text pretraining

Training the vision encoders most VLMs are built on.
3 lessons
  1. CLIP and the InfoNCE objective: batch size, temperature and zero-shot transfer
  2. SigLIP and SigLIP 2: sigmoid loss, multilingual data and dense features
  3. Training and evaluating with OpenCLIP: retrieval, zero-shot classification and linear probes

Module 3: Data for multimodal models

Assembling, filtering and synthesizing the image-text data that sets the quality ceiling.
4 lessons
  1. Web-scale image-text pairs: LAION, DataComp and CLIP-score filtering
  2. Recaptioning and dense captions: synthetic and human-written data such as PixMo
  3. Documents, charts, OCR and interleaved image-text data
  4. Licensing, personal data and safety filtering for image datasets

Module 4: Building and training a vision-language model

The staged recipe from aligned components to an instruction-following VLM.
4 lessons
  1. Stage 1 alignment: training the projector with frozen encoder and LLM
  2. Stage 2 multimodal pretraining and stage 3 visual instruction tuning
  3. Native dynamic resolution and multimodal positional encoding in the Qwen-VL family
  4. LoRA fine-tuning an open VLM with Transformers and TRL on domain documents

Module 5: Beyond images: video, audio and any-to-any

Extending the recipe to temporal and acoustic inputs and to multimodal outputs.
3 lessons
  1. Video understanding: frame sampling, temporal pooling and long-video token budgets
  2. Speech and audio inputs: encoder choice and joint training with text
  3. Omni and any-to-any models: speech output and image generation heads

Module 6: Evaluating and aligning multimodal models

Measuring what the model really sees and reducing hallucination.
4 lessons
  1. Benchmark suites with VLMEvalKit and lmms-eval: MMMU, DocVQA, ChartQA and video QA
  2. Object hallucination and grounding tests: POPE-style probes and pointing
  3. Multimodal preference tuning: DPO with image-conditioned pairs
  4. Contamination, serving cost and a multimodal release checklist

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by OpenAI (CLIP), OpenCLIP, Google (SigLIP 2, PaliGemma), Qwen (Qwen-VL family), AllenAI (Molmo, PixMo), LLaVA, Hugging Face (Transformers, TRL), Meta (Chameleon), LAION / DataComp, VLMEvalKit, or lmms-eval. Product and company names are trademarks of their respective owners.

More in Model Development

All 16 courses in this track
mod-201 Intermediate Available

Deep Learning with PyTorch

A hands-on course in modern PyTorch: tensors and autograd, nn.Module design, data loading, a training loop you understand line by line, GPU and mixed-precision training, torch.compile, and exporting a model for deployment.

14 h · 19 lessons See price at checkout
mod-202 Intermediate Available

Transformers from the Ground Up

Build a decoder-only transformer piece by piece: tokenization, embeddings, scaled dot-product attention, multi-head and grouped-query attention, positional schemes such as RoPE, and the training and sampling loop, then map it onto real open-weight model architectures.

16 h · 19 lessons See price at checkout
mod-203 Intermediate Waitlist

Classical Machine Learning in Production

Gradient-boosted trees and linear models still win most tabular problems. This course covers leakage-proof pipelines, honest validation, tuning, calibration and explainability, then packaging, serving and monitoring for drift.

14 h · 19 lessons Join the waitlist