Module 9 · 5 lessons · about 335 minutes
From LSTMs to today's models
Attention as a dot-product lookup, how frontier models are pretrained and post-trained, fine-tuning your own model, and how vision models find objects.
- Lessons
- 0/5
- Exercises
- 0/24
- Code
- 0/6
- Checkpoint
- Not taken
Already comfortable with this? Take the checkpoint cold. If you pass, move on and come back only for what you missed.

- 1AttentionAttention lets every position look directly at every other position. Score with dot products, turn the scores into weights with a softmax, and average the values. It is the operation that replaced recurrence and the core of every transformer.70 min
- 2Inside a transformerA transformer cuts text into tokens, turns each token into a vector, sends the vectors through a stack of identical blocks of attention and small networks, and turns the last vector into a probability for every possible next token. This lesson follows one sentence through every step, with the shapes and the numbers.75 min
- 3Pretraining and fine-tuningPretraining is the next-character loss you already wrote, run on trillions of tokens. Supervised fine-tuning keeps that loss and changes the data. This lesson writes out both, estimates what pretraining costs, and fine-tunes a model of your own.65 min
- 4Preferences, rewards and adaptersSupervised fine-tuning can only imitate. Preference methods learn from comparisons, reinforcement learning learns from answers a program can check, and LoRA adapts a model by training a small change to each weight matrix. This lesson writes out each objective and its gradient, trains an adapter, and ends with what a post-training company does all day.60 min
- 5Vision and what comes nextConvolution shares one small filter across a whole image, the way a recurrent network shares its weights across time. From there, the R-CNN family finds objects by sharing computation, segmentation labels every pixel, vision transformers read patches as tokens, and CLIP puts images and text in one space.65 min
- Checkpoint12 questions. Pass with 80% to finish the module.
Practice sets for this module
Fresh problems every time. Use them until the mechanics feel automatic.
A query attends over three keys , , , in dimensions. Find the attention weights . Give each to 3 decimal places.
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
A convolution slides a kernel over a input with padding 1 (1 row and column of zeros added on every side) and stride 1. How wide is the output?
Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.