Wirewalk Instituteof AI
mod-320 Expert Waitlist

Alignment: RLHF, DPO and Preference Optimization

For ML engineers who post-train open-weight models and need to collect preference data, train reward models, run PPO-style RLHF or DPO-family methods, and prove the result is better and not just longer.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

Supervised fine-tuning teaches a model what good outputs look like; preference optimization teaches it which of two plausible outputs is better. This course treats that second step as an engineering discipline. You start with the objective itself, the KL-regularized reward maximization behind RLHF, and derive why DPO can optimize the same objective without an explicit reward model or online sampling.

The course then covers each stage with real tooling. You design preference-data collection, rubrics and annotator agreement, and use AI feedback and constitution-style principles where human labels are too expensive. You train and audit reward models, run PPO with TRL, OpenRLHF or verl, and compare it with DPO, KTO, ORPO and SimPO on cost, stability and data needs. The final modules are about knowing whether it worked: length bias, reward hacking, over-refusal, sycophancy and regressions on capability benchmarks, plus safety tuning and red-team data. Reinforcement learning from verifiable rewards and GRPO for reasoning are covered in depth in mod-370 and only introduced here. Library features are current as of October 2026.

What you will be able to do

  • Derive the KL-regularized RLHF objective and explain the role of the reference policy and the KL coefficient
  • Design a preference-data collection program with rubrics, agreement checks and AI-feedback augmentation
  • Train a Bradley-Terry reward model and audit it for length bias and exploitable shortcuts
  • Run PPO-based RLHF and diagnose instability from KL, reward and value-loss curves
  • Derive DPO from the RLHF objective and choose between DPO, IPO, KTO, ORPO and SimPO for a given dataset and budget
  • Evaluate an aligned model for helpfulness, harmlessness, over-refusal and capability regressions
  • Identify reward hacking, sycophancy and mode collapse and select mitigations

Syllabus

6 modules · 22 lessons · about 20 hours

Module 1: The alignment objective

What preference optimization is trying to do and where it sits after pretraining and SFT.
3 lessons
  1. From SFT to preferences: what supervised fine-tuning cannot teach
  2. The KL-regularized reward objective and the reference policy
  3. Helpful, honest, harmless: turning goals into a specification

Module 2: Preference data

Collecting the comparisons that everything downstream depends on.
4 lessons
  1. Pairwise, ranked and binary feedback: formats and trade-offs
  2. Annotation programs: rubrics, annotator agreement and quality control
  3. AI feedback, LLM judges and constitution-guided labeling
  4. Public preference datasets and their licenses and biases

Module 3: Reward models

Learning a scalar judge of quality and checking that it deserves trust.
3 lessons
  1. Bradley-Terry reward modeling and training with RewardTrainer
  2. Evaluating reward models: held-out accuracy and RewardBench-style suites
  3. Length bias, spurious features and reward-model ensembles

Module 4: RLHF with PPO

Online optimization against a reward model, and why it is hard to run.
4 lessons
  1. PPO for language models: policy, value head, advantages and the KL penalty
  2. Running RLHF with TRL, OpenRLHF and verl: generation, memory and vLLM rollouts
  3. Reading an RLHF run: reward, KL, entropy and value loss
  4. Beyond PPO: RLOO and group-relative baselines in brief

Module 5: Direct preference optimization and its variants

Offline methods that skip the explicit reward model.
4 lessons
  1. Deriving DPO: the implicit reward and the beta parameter
  2. IPO, KTO, ORPO and SimPO: what each changes and when to use it
  3. Online and iterative DPO with on-policy samples
  4. Practical DPO: data mixtures, LoRA and avoiding likelihood collapse

Module 6: Did alignment work?

Evaluation, failure modes and safety tuning.
4 lessons
  1. Win rates, LLM-as-judge pitfalls and length-controlled comparison
  2. Reward hacking, sycophancy and over-refusal
  3. Safety tuning with red-team data and refusal calibration
  4. Checking for capability regressions and an alignment release checklist

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by Hugging Face (TRL, PEFT), OpenRLHF, verl, NVIDIA NeMo RL, vLLM, Argilla, or AllenAI (Tulu recipes, RewardBench). Product and company names are trademarks of their respective owners.

More in Model Development

All 16 courses in this track
mod-201 Intermediate Available

Deep Learning with PyTorch

A hands-on course in modern PyTorch: tensors and autograd, nn.Module design, data loading, a training loop you understand line by line, GPU and mixed-precision training, torch.compile, and exporting a model for deployment.

14 h · 19 lessons See price at checkout
mod-202 Intermediate Available

Transformers from the Ground Up

Build a decoder-only transformer piece by piece: tokenization, embeddings, scaled dot-product attention, multi-head and grouped-query attention, positional schemes such as RoPE, and the training and sampling loop, then map it onto real open-weight model architectures.

16 h · 19 lessons See price at checkout
mod-203 Intermediate Waitlist

Classical Machine Learning in Production

Gradient-boosted trees and linear models still win most tabular problems. This course covers leakage-proof pipelines, honest validation, tuning, calibration and explainability, then packaging, serving and monitoring for drift.

14 h · 19 lessons Join the waitlist