Wirewalk Instituteof AI
mod-360 Expert Waitlist

Pretraining a Language Model

For research and ML engineers who need to plan and run a language-model pretraining run end to end: data, tokenizer, architecture, optimizer, stability, evaluation and handoff to post-training.

The curriculum is being prepared and reviewed. Enrollment is not open yet.

About this course

Pretraining is where a model's knowledge and most of its capability come from, and where most of its compute is spent. This course follows a pretraining project from decision to handoff. You start with the questions that decide whether to do it at all: pretraining from scratch versus continued pretraining of an open base model, the token and compute budget, and what open recipes such as AI2's OLMo and Hugging Face's FineWeb and SmolLM work have shown publicly.

The data module is the largest: extraction from Common Crawl, language identification, heuristic and model-based quality filtering, MinHash and exact deduplication, PII and toxicity handling, decontamination, and mixing with code, math and multilingual sources using datatrove. You then train a byte-level BPE tokenizer and make architecture and hyperparameter choices using small proxy models and maximal-update parametrization (muP) for transfer. The training modules cover AdamW and newer optimizers such as Muon, warmup-stable-decay schedules, loss spikes and their mitigations (z-loss, QK-norm, gradient clipping, rollback), annealing on high-quality data, and context-length extension. Parallelism, cluster operations and scaling-law compute planning are covered in trn-310, trn-330 and trn-320; this course uses them rather than re-teaching them. Facts are current as of October 2026.

What you will be able to do

  • Decide between pretraining from scratch, continued pretraining and fine-tuning given data, compute and goals
  • Build a web-scale data pipeline with extraction, language identification, quality filtering, deduplication and decontamination
  • Train and evaluate a tokenizer and quantify its effect on sequence length and multilingual cost
  • Choose architecture and hyperparameters using small proxy runs and muP-style transfer
  • Configure optimizers and learning-rate schedules, including warmup-stable-decay and annealing on high-quality data
  • Detect and recover from loss spikes, divergence and data-related instabilities during a long run
  • Evaluate base-model checkpoints during training and prepare a model for post-training handoff

Syllabus

6 modules · 20 lessons · about 24 hours

Module 1: Planning a pretraining run

Deciding what to train, on how many tokens, and whether to train at all.
3 lessons
  1. Scratch versus continued pretraining versus fine-tuning
  2. Token budgets, compute budgets and what open recipes report
  3. Training stacks compared: torchtitan, Megatron-Core and nanotron

Module 2: Building the pretraining corpus

Turning the raw web and curated sources into trillions of training tokens.
4 lessons
  1. From Common Crawl to text: extraction and language identification with datatrove
  2. Quality filtering: heuristics and classifier-based filters such as FineWeb-Edu
  3. Deduplication at scale: exact, MinHash and suffix-array methods
  4. PII, toxicity, licensing and benchmark decontamination

Module 3: Data mixtures and tokenization

Deciding what proportion of each source the model sees, and how text is split.
3 lessons
  1. Mixing web, code, math, books and multilingual data
  2. Training a byte-level BPE tokenizer and choosing vocabulary size
  3. Curricula, upsampling and mid-training data

Module 4: Architecture and hyperparameters at scale

Making decisions with small experiments that hold for the big run.
3 lessons
  1. Architecture choices for a base model: depth, width, GQA and MoE
  2. Proxy models and muP for hyperparameter transfer
  3. Batch size, sequence length and the critical batch size

Module 5: Running and stabilizing training

Optimizers, schedules and what to do when the loss misbehaves.
4 lessons
  1. Optimizers: AdamW settings, weight decay and Muon
  2. Schedules: cosine versus warmup-stable-decay and checkpoint annealing
  3. Loss spikes and divergence: z-loss, QK-norm, clipping and rollback
  4. Monitoring a run: loss, gradient norms, throughput and data-loader health

Module 6: Finishing the base model

Evaluating, extending and handing off what you trained.
3 lessons
  1. Evaluating base checkpoints: perplexity, few-shot suites and early signals
  2. Long-context extension: RoPE base scaling and long-document data
  3. Handoff to post-training and documenting the run in a model card

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 25 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 60 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by PyTorch (torchtitan), NVIDIA (Megatron-Core, NeMo), Hugging Face (datatrove, nanotron, FineWeb), Allen Institute for AI (OLMo, Dolma), EleutherAI (lm-evaluation-harness), or Weights & Biases. Product and company names are trademarks of their respective owners.

More in Model Development

All 16 courses in this track
mod-201 Intermediate Available

Deep Learning with PyTorch

A hands-on course in modern PyTorch: tensors and autograd, nn.Module design, data loading, a training loop you understand line by line, GPU and mixed-precision training, torch.compile, and exporting a model for deployment.

14 h · 19 lessons See price at checkout
mod-202 Intermediate Available

Transformers from the Ground Up

Build a decoder-only transformer piece by piece: tokenization, embeddings, scaled dot-product attention, multi-head and grouped-query attention, positional schemes such as RoPE, and the training and sampling loop, then map it onto real open-weight model architectures.

16 h · 19 lessons See price at checkout
mod-203 Intermediate Waitlist

Classical Machine Learning in Production

Gradient-boosted trees and linear models still win most tabular problems. This course covers leakage-proof pipelines, honest validation, tuning, calibration and explainability, then packaging, serving and monitoring for drift.

14 h · 19 lessons Join the waitlist