Wirewalk Instituteof AI
mod-370 Expert Waitlist

Reinforcement Learning for LLM Reasoning: RLVR, GRPO and Reward Design

For ML engineers who want to train reasoning behavior into open-weight LLMs with verifiable rewards, run GRPO-family algorithms at scale, and tell genuine capability gains from reward hacking.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

Since DeepSeek-R1 showed that large-scale reinforcement learning against automatically checkable rewards could produce long chain-of-thought reasoning, RL from verifiable rewards (RLVR) has become a standard stage of LLM post-training. It is also a different engineering problem from preference-based RLHF: rewards come from programs rather than human judges, rollouts are long and expensive, and the most common failure is not instability but a model that learns to satisfy the checker instead of solving the problem.

This course treats reasoning RL as a system. You build verifiers and sandboxes for math answers, unit-tested code and multi-turn tool use; derive GRPO from PPO and study the fixes later work made to its normalization, clipping and length behavior; and run training with verl, TRL's GRPOTrainer, OpenRLHF or NeMo RL using vLLM or SGLang rollouts. You learn the cold-start and curriculum choices behind R1-style recipes, the infrastructure trade-offs of colocated versus disaggregated and asynchronous rollouts, and how to design rewards when only part of a task is verifiable. The last module is about proof: pass@k versus pass@1, contamination checks, format and length hacking, and when distilling a strong reasoner beats running RL at all. Preference-based alignment (RLHF, DPO) is covered in mod-320; RL foundations in mod-206. Library and model facts are current as of October 2026.

What you will be able to do

  • Explain where RLVR sits relative to SFT, RLHF and distillation, and when it is the right tool
  • Build reward functions and sandboxed verifiers for math, code and multi-turn tool-use tasks
  • Derive the GRPO objective from PPO and explain what Dr. GRPO, DAPO and GSPO change and why
  • Configure and run a GRPO training job with verl or TRL and size its rollout and training resources
  • Design an RLVR data pipeline with difficulty filtering, curricula and cold-start SFT
  • Design rewards for partially verifiable tasks using rubrics, generative reward models and process supervision
  • Detect reward hacking, length exploitation and benchmark contamination in a reasoning RL run
  • Evaluate reasoning gains with pass@1, pass@k and held-out suites and decide between RL and distillation

Syllabus

6 modules · 23 lessons · about 22 hours

Module 1: Why reinforcement learning for reasoning

What verifiable rewards add beyond SFT and preference optimization, and what the R1-era results do and do not show.
3 lessons
  1. From RLHF to RLVR: replacing a learned judge with a checker
  2. Reading the DeepSeek-R1 and R1-Zero recipes: cold start, RL stages and rejection sampling
  3. Test-time compute: long chain-of-thought, self-verification and when RL elicits versus teaches

Module 2: Verifiers and reward functions

Building the reward signal: answer checkers, code sandboxes, tool environments and format rewards.
4 lessons
  1. Math verifiers: answer extraction, symbolic equivalence and the Math-Verify approach
  2. Code rewards: unit tests, sandboxed execution and partial credit
  3. Multi-turn and tool-use environments: agent rollouts with real tool calls
  4. Format, length and language-consistency rewards and how they backfire

Module 3: The GRPO family

Group-relative policy optimization and the corrections that followed it.
4 lessons
  1. From PPO to GRPO: dropping the critic and using group-normalized advantages
  2. Bias in the objective: Dr. GRPO, length normalization and the standard-deviation term
  3. DAPO and clip-higher: entropy collapse, dynamic sampling and token-level loss
  4. Sequence-level ratios (GSPO), RLOO and REINFORCE++: choosing an estimator

Module 4: Rollout and training systems

Running reasoning RL efficiently on GPU clusters.
4 lessons
  1. Framework landscape: verl, TRL GRPOTrainer, OpenRLHF and NeMo RL
  2. Inference engines in the loop: vLLM and SGLang rollouts, weight sync and KV-cache memory
  3. Colocated, disaggregated and asynchronous RL: off-policy staleness and its corrections
  4. Sizing a run: group size, sequence length, rollout-to-train GPU ratio and cost

Module 5: Data, curriculum and reward design beyond pure verification

Choosing training problems and extending RL to tasks that are only partly checkable.
4 lessons
  1. Building RLVR datasets: difficulty filtering, pass-rate bands and deduplication
  2. Curricula, cold-start SFT and multi-stage training
  3. Rubric rewards, generative reward models and LLM judges for open-ended tasks
  4. Process reward models and step-level supervision: promise and pitfalls

Module 6: Proving it worked

Evaluation, failure analysis and the RL-versus-distillation decision.
4 lessons
  1. Measuring reasoning: pass@1, pass@k, majority voting and variance across seeds
  2. Reward hacking and verifier exploits: case patterns and audits
  3. Contamination, held-out suites and live benchmarks
  4. RL or distillation: choosing for small models and a reasoning-RL release checklist

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by Hugging Face (TRL), verl, OpenRLHF, NVIDIA NeMo RL, vLLM, SGLang, DeepSeek (R1, GRPO), Qwen, or AllenAI (Tulu 3, OLMo). Product and company names are trademarks of their respective owners.

More in Model Development

All 16 courses in this track
mod-201 Intermediate Available

Deep Learning with PyTorch

A hands-on course in modern PyTorch: tensors and autograd, nn.Module design, data loading, a training loop you understand line by line, GPU and mixed-precision training, torch.compile, and exporting a model for deployment.

14 h · 19 lessons See price at checkout
mod-202 Intermediate Available

Transformers from the Ground Up

Build a decoder-only transformer piece by piece: tokenization, embeddings, scaled dot-product attention, multi-head and grouped-query attention, positional schemes such as RoPE, and the training and sampling loop, then map it onto real open-weight model architectures.

16 h · 19 lessons See price at checkout
mod-203 Intermediate Waitlist

Classical Machine Learning in Production

Gradient-boosted trees and linear models still win most tabular problems. This course covers leakage-proof pipelines, honest validation, tuning, calibration and explainability, then packaging, serving and monitoring for drift.

14 h · 19 lessons Join the waitlist