Wirewalk Instituteof AI
mod-202 Intermediate Available

Transformers from the Ground Up

For engineers who want to understand the transformer well enough to implement a GPT-style decoder from scratch and read modern model configs with confidence.

Level
Intermediate
Study time
16 hours
Lessons
19 in 6 modules
Exam
20 questions

About this course

Transformers are usually explained with diagrams that skip the parts that matter in practice: tensor shapes, masking, normalization placement and how positions are encoded. In this course you implement every component in PyTorch and check it against shape-level reasoning, then assemble a small GPT-style model and train it on a text corpus.

Once the core is clear, the course walks through the architectural choices that distinguish today's open-weight models: pre-norm and RMSNorm, SwiGLU feed-forward blocks, rotary embeddings, grouped-query and multi-head latent attention, mixture-of-experts layers, and the KV cache used at inference time. You finish by reading real Hugging Face config.json files and explaining what each field changes in compute, memory and behavior. Architecture details are current as of October 2026.

What you will be able to do

  • Explain byte-pair encoding tokenization and its effect on sequence length, cost and multilingual behavior
  • Implement scaled dot-product attention with causal masking and verify tensor shapes at every step
  • Build multi-head and grouped-query attention and compute their parameter and KV-cache cost
  • Compare absolute, learned and rotary positional encodings and explain how RoPE scaling extends context
  • Assemble, train and sample from a small decoder-only transformer using temperature, top-k and top-p
  • Explain mixture-of-experts routing, load balancing and why total and active parameters differ
  • Read an open-weight model's config.json and estimate its parameter count and memory footprint

Syllabus

6 modules · 19 lessons · about 16 hours

Module 1: Tokens and embeddings

How text becomes integer IDs and then vectors the network can process.
3 lessons
  1. Byte-pair encoding and training a tokenizer
  2. Token embeddings, the output head and weight tying
  3. Why tokenization explains odd model behavior

Module 2: Attention, line by line

The attention operation implemented and reasoned about at the level of shapes.
4 lessons
  1. Queries, keys and values: scaled dot-product attention
  2. Causal masking and padding masks
  3. Multi-head, multi-query and grouped-query attention
  4. FlashAttention and why attention is memory-bound

Module 3: Position and the transformer block

Encoding order and wiring attention and MLP layers into a stable residual block.
3 lessons
  1. Positional encodings: sinusoidal, learned and RoPE
  2. Residual streams, LayerNorm versus RMSNorm and pre-norm placement
  3. Feed-forward blocks: GELU MLPs and SwiGLU

Module 4: Building and training a mini GPT

Putting the pieces together and training a model you wrote yourself.
3 lessons
  1. Assembling the decoder-only model in PyTorch
  2. Next-token loss, batching and a training run on a text corpus
  3. Sampling: greedy, temperature, top-k and top-p

Module 5: Inference mechanics

What changes when the model generates one token at a time.
2 lessons
  1. The KV cache: what it stores and how big it gets
  2. Long context: RoPE scaling, sliding-window and sparse attention

Module 6: Modern architectures in the wild

Mapping your implementation onto the open-weight models practitioners actually use.
4 lessons
  1. Mixture-of-experts: routers, top-k experts and load balancing
  2. Multi-head latent attention and other KV-cache reductions
  3. Encoder-only, encoder-decoder and decoder-only: when each fits
  4. Reading a Hugging Face config.json and estimating parameters

Exam and certificate

The exam unlocks once you have completed every lesson. It draws 20 questions from a larger bank, spread across the modules, with the answer options shuffled for each attempt. You have 45 minutes and need 80% to pass. After a failed attempt you can try again after 24 hours, up to 3 attempts in total. Feedback is given per module, never as the answer key.

Passing earns a certificate of completion and assessment from the Wirewalk Institute of AI with a unique ID that anyone can check at institute.wirewalk.com/verify. It is not an accredited qualification or a vendor certification. How certification works.

The Wirewalk Institute of AI is independent. This course teaches publicly documented products and is not affiliated with or endorsed by PyTorch, Hugging Face (Transformers, Tokenizers), or OpenAI (tiktoken). Product and company names are trademarks of their respective owners.

More in Model Development

All 16 courses in this track
mod-201 Intermediate Available

Deep Learning with PyTorch

A hands-on course in modern PyTorch: tensors and autograd, nn.Module design, data loading, a training loop you understand line by line, GPU and mixed-precision training, torch.compile, and exporting a model for deployment.

14 h · 19 lessons See price at checkout
mod-203 Intermediate Waitlist

Classical Machine Learning in Production

Gradient-boosted trees and linear models still win most tabular problems. This course covers leakage-proof pipelines, honest validation, tuning, calibration and explainability, then packaging, serving and monitoring for drift.

14 h · 19 lessons Join the waitlist
mod-204 Intermediate Waitlist

Computer Vision Models

From CNNs and vision transformers to self-supervised backbones, real-time detectors and promptable segmentation: how modern vision models work, how to fine-tune them on your own labeled data, how to measure them with mAP and IoU, and how to deploy them fast.

15 h · 18 lessons Join the waitlist