ml.lab
Python sleeps until you run code

Part 1

From matrices to memory

Ten modules. The first half rebuilds the math you need: pictures first, then hand calculation, then code. The second half uses it to build networks, recurrent networks and LSTMs from scratch, and ends with attention, transformers and how today's models are trained. Each module ends with a checkpoint you can also take first, to skip what you already know.

In a hurry

The LLM fast track, about 17 hours

The shortest path to a working understanding of how a language model is built and trained. It skips the eigenvalue and determinant depth, the extra practice lessons and the LSTM module; come back to those when you want the full picture.

  1. 1The big pictureThe whole landscape, and what pretraining and post-training mean.
  2. 2The dot productThe one operation behind neurons, attention and embedding search.
  3. 3Matrix multiplicationHow a layer transforms a whole batch at once.
  4. 4Shapes, transposes and batchesReading shapes, the everyday skill of model code.
  5. 5GradientsWhich way is downhill for a loss.
  6. 6Gradient descentHow every model learns.
  7. 7Neurons and layersNeurons and layers as dot products and matrices.
  8. 8Softmax and cross-entropyScores to probabilities, and the loss every LLM minimizes.
  9. 9BackpropagationHow the gradient of every weight is computed.
  10. 10Training a networkThe loop you will write for every model.
  11. 11Predicting the next characterNext-token prediction, the pretraining objective itself.
  12. 12AttentionThe mechanism at the heart of transformers, number by number.
  13. 13Inside a transformerHow a sentence becomes next-token probabilities.
  14. 14Pretraining and fine-tuningPretraining at scale and supervised fine-tuning.
  15. 15Preferences, rewards and adaptersPreference tuning, RL with rewards and LoRA adapters.