ml.lab
Python sleeps until you run code

Reference

Glossary

Every term the course uses, in plain words, with a link to where it is taught.

166 terms

Activation function · also nonlinearity

The elementwise nonlinearity applied after a layer's matrix multiplication, such as ReLU, tanh, sigmoid or GELU. Without it, a stack of layers collapses into a single layer: one matrix and one bias.

Taught in Neural networks from scratch
Adam · also AdamW

An optimizer (Kingma and Ba; the paper appeared in 2014 and was published at ICLR in 2015) that scales each parameter's step using running averages of its gradient and its squared gradient, with a bias correction for the first steps. It is the usual default for training networks, often with decoupled weight decay (AdamW).

Taught in Neural networks from scratch
Attention · also self-attention

A weighted average of value vectors, with weights given by the softmax of dot-product scores between a query and each key. In self-attention the queries, keys and values all come from the same sequence, so every position can look directly at the others.

Taught in From LSTMs to today's models
Axis

One dimension of a numpy array. A reduction such as X.sum(axis=0) collapses the named axis: for an (n, d) array, axis=0 leaves one number per column and axis=1 one per row.

Taught in The map
Backpropagation · also backprop, reverse-mode differentiation

Computing the gradient of the loss with respect to every parameter by applying the chain rule backwards through the computational graph. The forward pass stores intermediate values; the backward pass multiplies local derivatives by the incoming gradient at each node, for a small multiple of the forward pass's cost.

Taught in Neural networks from scratch
Backpropagation through time · also BPTT, truncated BPTT

Backpropagation on an unrolled recurrent network. The gradient at each hidden state combines the contribution from that step's output with the one arriving from the next step, and each weight's gradient is a sum over time steps. Truncated BPTT stops the gradient after a fixed number of steps to bound the cost.

Taught in Recurrent networks
Base model · also pretrained model

The model that comes out of pretraining. It continues text in the style of its training data and knows a great deal, but it does not reliably follow instructions until it is post-trained.

Taught in From LSTMs to today's models
Basis

A set of linearly independent vectors that spans the space. Every vector is then exactly one combination of them, and its coefficients are the vector's coordinates in that basis.

Taught in Vectors and the dot product
Batch

A group of examples processed together. In code a batch is an array X of shape (batch, features) with one row per example, so a layer is X @ W + b; math in this course writes a single vector as a column instead.

Taught in Matrices are machines
Bias

A learned constant added after the weighted sum, which shifts where a neuron switches on. For a batch, the bias vector is broadcast and added to every row.

Taught in Neural networks from scratch
Bigram model

A language model that predicts each character or token from the one before it alone, using counted frequencies P(b∣a)P(b \mid a), smoothed so that no pair gets probability 0. It is the simplest model with context, and an honest baseline that a trained network has to beat.

Taught in Recurrent networks
Broadcasting

numpy's rule for combining arrays of different shapes: compare the shapes from the right; each pair of sizes must be equal or include a 1, and size-1 or missing dimensions are stretched to match. It is how one bias vector is added to every row of a batch.

Taught in Matrices are machines
Candidate · also candidate cell state

The vector gt\mathbf{g}_t of new content an LSTM proposes to write into its cell state, computed with tanh. Some sources call it c~t\tilde{\mathbf{c}}_t.

Taught in LSTMs
Catastrophic forgetting

Losing abilities a model had before when it is trained further on narrow data. Lower learning rates, mixing in general data and parameter-efficient methods such as LoRA reduce it.

Taught in From LSTMs to today's models
Cauchy-Schwarz inequality

∣a⋅b∣≤∣a∣ ∣b∣|\mathbf{a}\cdot\mathbf{b}| \le |\mathbf{a}|\,|\mathbf{b}|, with equality exactly when one vector is a multiple of the other. It keeps cosine similarity between −1-1 and 11, which is what lets an angle be defined between vectors in any number of dimensions.

Taught in Vectors and the dot product
Causal mask

Blocking each position from attending to later positions, by setting those scores to −∞-\infty before the softmax, so a language model predicts every token from earlier tokens only.

Taught in From LSTMs to today's models
Cell state

The LSTM's memory vector ct\mathbf{c}_t. It is updated by elementwise scaling and adding rather than by a matrix multiplication, so information and gradients can pass through many steps nearly unchanged.

Taught in LSTMs
Chain rule

The derivative of a composition is the product of the derivatives: (f∘g)′(x)=f′(g(x)) g′(x)(f \circ g)'(x) = f'(g(x))\,g'(x). For functions of vectors it becomes a product of Jacobians, and backpropagation applies it systematically.

Taught in Derivatives, gradients and Jacobians
Chain rule of probability

p(x1,…,xT)=∏t=1Tp(xt∣x1,…,xt−1)p(x_1,\allowbreak \ldots,\allowbreak x_T) = \prod_{t=1}^{T} p(x_t \mid x_1,\allowbreak \ldots,\allowbreak x_{t-1}): the probability of a sequence is the product of each token's probability given everything before it. It is why predicting the next token is enough to model whole texts.

Taught in The map
Characteristic polynomial · also characteristic equation

det⁡(A−λI)\det(\mathbf{A} - \lambda\mathbf{I}), whose roots are the eigenvalues. For a 2×22 \times 2 matrix it is λ2−(tr⁡A)λ+det⁡A\lambda^2 - (\operatorname{tr}\mathbf{A})\lambda + \det\mathbf{A}, where the trace tr⁡A\operatorname{tr}\mathbf{A} is the sum of the diagonal entries. The eigenvalues add up to the trace and multiply to the determinant.

Taught in Eigenvectors and eigenvalues
CLIP

A model that places images and text in one embedding space by training matching image and caption pairs to have high cosine similarity and mismatched pairs low similarity, so images and text can be compared directly.

Taught in From LSTMs to today's models
Column space · also range, image

Every output a matrix can produce: the span of its columns. Its dimension is the rank.

Taught in Determinants, inverses and rank
Computational graph

A computation drawn as nodes, each one simple operation, joined by edges that carry values. The forward pass fills in every node's value; backpropagation visits the nodes in reverse and multiplies each node's local derivatives by the gradient arriving from above.

Taught in Neural networks from scratch
Condition number

The ratio of a matrix's largest to smallest stretch, σmax⁡/σmin⁡\sigma_{\max}/\sigma_{\min}; for a quadratic loss, the ratio λmax⁡/λmin⁡\lambda_{\max}/\lambda_{\min} of its Hessian's eigenvalues. A large value makes solving systems sensitive to small errors and makes gradient descent zig-zag and converge slowly.

Taught in Determinants, inverses and rank
Convolution

Sliding a small grid of weights, the kernel, across an image and computing at each position the sum of elementwise products with the patch underneath; the outputs form a feature map. Deep learning does not flip the kernel, so strictly this is cross-correlation.

Taught in From LSTMs to today's models
Convolutional neural network · also CNN, convnet

A network built mainly from convolution layers, which share weights across space so a pattern detector works anywhere in the image. The dominant architecture for images from 2012 on, now often combined with or replaced by vision transformers.

Taught in From LSTMs to today's models
Cosine similarity

a⋅b∣a∣ ∣b∣\frac{\mathbf{a}\cdot\mathbf{b}}{|\mathbf{a}|\,|\mathbf{b}|}, the cosine of the angle between two vectors. The Cauchy-Schwarz inequality ∣a⋅b∣≤∣a∣ ∣b∣|\mathbf{a}\cdot\mathbf{b}| \le |\mathbf{a}|\,|\mathbf{b}| keeps it between −1-1 and 11. It ignores length, which is why embedding search uses it.

Taught in Vectors and the dot product
Covariance matrix

For data with dd features, the symmetric d×dd \times d matrix whose diagonal holds each feature's variance and whose entry (j,k)(j,\allowbreak k) measures how features jj and kk vary together. From centered data X~\tilde{\mathbf{X}} with nn rows it is X~⊤X~/(n−1)\tilde{\mathbf{X}}^\top\tilde{\mathbf{X}}/(n - 1), as np.cov computes it. Its eigenvectors are the principal components.

Taught in Eigenvectors and eigenvalues
Cross-entropy · also cross-entropy loss, negative log-likelihood, log loss

The loss −ln⁡p-\ln p, where pp is the probability the model gave to the correct class or next token, averaged over examples. A uniform guess over KK classes scores ln⁡K\ln K. Combined with softmax, its gradient with respect to the logits is p−y\mathbf{p} - \mathbf{y}.

Taught in Neural networks from scratch
Decision boundary

The set of inputs where a classifier switches from one class to another. For a single neuron it is the line (a hyperplane in more dimensions) w⋅x+b=0\mathbf{w}\cdot\mathbf{x} + b = 0, perpendicular to w\mathbf{w}; a hidden layer lets a network draw curved boundaries.

Taught in Neural networks from scratch
Dense layer · also fully connected layer, linear layer

A matrix multiplication plus a bias, Wx+b\mathbf{W}\mathbf{x} + \mathbf{b} for one column vector or X @ W + b for a batch in code, usually followed by a nonlinearity.

Taught in Matrices are machines
Derivative

The slope of the best straight-line approximation to a function at a point: f(x+h)≈f(x)+f′(x) hf(x + h) \approx f(x) + f'(x)\,h for small hh. That linear approximation is the idea behind every gradient in training.

Taught in Derivatives, gradients and Jacobians
Determinant

The factor by which a square matrix scales area (volume in 3D). A negative value means orientation is flipped; zero means space is squashed flat, so the matrix is singular and has no inverse. For (abcd)\begin{pmatrix} a & b \\ c & d \end{pmatrix} it is ad−bcad - bc.

Taught in Determinants, inverses and rank
Diagonalization

Writing A=PDP−1\mathbf{A} = \mathbf{P}\mathbf{D}\mathbf{P}^{-1} with eigenvectors as the columns of P\mathbf{P} and eigenvalues on the diagonal of D\mathbf{D}, which is possible when an n×nn \times n matrix has nn linearly independent eigenvectors. Then Ak=PDkP−1\mathbf{A}^k = \mathbf{P}\mathbf{D}^k\mathbf{P}^{-1}: in the eigenvector basis, a power only raises each eigenvalue to the kkth power.

Taught in Eigenvectors and eigenvalues
Distillation · also knowledge distillation

Training a smaller student model to match a larger teacher model's outputs or its full next-token probabilities, keeping much of the quality at a lower serving cost.

Taught in From LSTMs to today's models
Dot product · also inner product, scalar product

a⋅b=∑iaibi\mathbf{a}\cdot\mathbf{b} = \sum_i a_i b_i: multiply matching entries and add. It equals ∣a∣ ∣b∣cos⁡θ|\mathbf{a}|\,|\mathbf{b}|\cos\theta, so it is positive when two vectors roughly agree, zero when they are perpendicular (orthogonal) and negative when they disagree.

Taught in Vectors and the dot product
DPO · also direct preference optimization

Direct preference optimization: learning from preference pairs with one loss that raises the preferred response's probability relative to the rejected one, both measured against a frozen reference model. It needs no reward model and no reinforcement-learning loop.

Taught in From LSTMs to today's models
Dropout

Randomly setting a fraction of activations to zero during training and rescaling the rest, which discourages units from relying on each other and reduces overfitting. It is switched off at inference.

Taught in LSTMs
Early stopping

Keeping the weights from the step where the validation loss was lowest, instead of the ones at the end of training. It is a simple defense against overfitting.

Taught in Neural networks from scratch
Eigenvalue

The factor λ\lambda by which a matrix stretches one of its eigenvectors. A negative λ\lambda flips it. Complex eigenvalues, as for a rotation, mean no real direction stays on its line, though ∣λ∣|\lambda| still measures growth.

Taught in Eigenvectors and eigenvalues
Eigenvector

A nonzero vector v\mathbf{v} that a square matrix only stretches or flips, without turning it off its line: Av=λv\mathbf{A}\mathbf{v} = \lambda\mathbf{v}.

Taught in Eigenvectors and eigenvalues
Elementwise product · also Hadamard product

Multiplying two vectors or matrices of the same shape entry by entry, written ⊙\odot; in numpy it is *.

Taught in Derivatives, gradients and Jacobians
Embedding · also embedding table

A learned vector that represents a discrete item such as a token, typically with hundreds or thousands of entries. An embedding table stores one row per token, and looking a token up is the same as multiplying its one-hot vector by that table.

Taught in Recurrent networks
Epoch

One full pass through the training data.

Taught in Neural networks from scratch
Exploding gradient

A gradient that grows geometrically as it is passed back through many steps or layers, because each step multiplies it by factors larger than 1. Gradient clipping is the standard remedy.

Taught in Recurrent networks
Fast R-CNN

R-CNN's successor (2015): run the CNN once on the whole image, crop each proposed region's features from the shared feature map (RoI pooling), and classify them with a small network.

Taught in From LSTMs to today's models
Faster R-CNN

Fast R-CNN plus a small learned region proposal network that proposes boxes from the same shared feature map, replacing the slow hand-designed proposal step (2015).

Taught in From LSTMs to today's models
Finite difference · also central difference

Estimating a derivative from function values. The central difference f(x+h)−f(x−h)2h\frac{f(x + h) - f(x - h)}{2h} has error proportional to h2h^2, far smaller than the forward difference's error proportional to hh. It is the basis of gradient checking.

Taught in Derivatives, gradients and Jacobians
Forget gate

The gate ft\mathbf{f}_t that scales the previous cell state: near 1 keeps it, near 0 erases it.

Taught in LSTMs
Forget-gate bias

The bias of the LSTM's forget gate, commonly initialized to 1 or more so the gate starts mostly open and the cell remembers by default early in training.

Taught in LSTMs
Gate

A vector of values between 0 and 1, computed by a sigmoid from the current input and the previous hidden state, that multiplies another vector entry by entry to control how much of it passes.

Taught in LSTMs
GELU · also Gaussian error linear unit

A smooth ReLU-like activation, x Φ(x)x\,\Phi(x), where Φ\Phi is the cumulative distribution function of the standard normal distribution. It is used in many transformers.

Taught in Neural networks from scratch
Gradient

The vector ∇f\nabla f of partial derivatives ∂f/∂xi\partial f / \partial x_i, each the rate of change when only xix_i moves. It points in the direction of steepest increase, its length is that steepest slope, and the rate of change along a unit direction u\mathbf{u} is ∇f⋅u\nabla f \cdot \mathbf{u}.

Taught in Derivatives, gradients and Jacobians
Gradient checking · also gradient check

Testing a hand-written backward pass by comparing its gradients with finite-difference estimates, measured by relative error. A tiny relative error means the backward pass is almost certainly right; a large one points to a bug.

Taught in Neural networks from scratch
Gradient clipping · also clipping by norm

Rescaling the gradient when its norm exceeds a threshold cc: if ∣g∣>c|\mathbf{g}| > c, replace g\mathbf{g} with c g/∣g∣c\,\mathbf{g}/|\mathbf{g}|. This keeps the direction and caps the size of the step.

Taught in Recurrent networks
Gradient descent

Minimizing a loss by repeatedly stepping against its gradient, w←w−η∇L\mathbf{w} \leftarrow \mathbf{w} - \eta\nabla L. With a small enough step size, each step lowers the loss.

Taught in Derivatives, gradients and Jacobians
Greedy decoding

Generating text by always choosing the single most probable next token. It is deterministic and tends toward repetitive text; sampling with a temperature adds variety.

Taught in Recurrent networks
GRU · also gated recurrent unit

The gated recurrent unit, a simpler relative of the LSTM with two gates (update and reset) and no separate cell state. With the same sizes it has three quarters of an LSTM's weights.

Taught in LSTMs
Hessian

The matrix of second partial derivatives of a scalar function. Its eigenvalues are the function's curvatures along the Hessian's eigenvector directions.

Taught in Derivatives, gradients and Jacobians
Hidden state

The vector ht\mathbf{h}_t that an RNN or LSTM carries from one step to the next, summarizing what it has read so far.

Taught in Recurrent networks
Identity matrix

The square matrix I\mathbf{I} with ones on the diagonal and zeros everywhere else. It moves nothing, Ix=x\mathbf{I}\mathbf{x} = \mathbf{x}, so AI=IA=A\mathbf{A}\mathbf{I} = \mathbf{I}\mathbf{A} = \mathbf{A}.

Taught in Matrices are machines
Inference · also serving

Running a trained model to produce outputs, as opposed to training it. Serving a language model means generating tokens for many users at once, usually with batching, caching of earlier computation and quantized weights.

Taught in The map
Initialization · also weight initialization, Xavier initialization, Glorot initialization, He initialization

The choice of starting weights. They must be random to break the symmetry between units, and scaled so signals neither shrink nor blow up across layers, as in Xavier (Glorot) and He initialization.

Taught in Neural networks from scratch
Input gate

The gate it\mathbf{i}_t that scales how much of the candidate gt\mathbf{g}_t is written into the cell state.

Taught in LSTMs
Instance segmentation

Giving each object in an image its own pixel mask, so two objects of the same class, such as two people standing together, stay separate. Mask R-CNN does it by adding a mask branch to a detector.

Taught in From LSTMs to today's models
Inverse · also matrix inverse

The matrix A−1\mathbf{A}^{-1} that undoes A\mathbf{A}, so A−1A=I\mathbf{A}^{-1}\mathbf{A} = \mathbf{I}. It exists exactly when det⁡A≠0\det\mathbf{A} \ne 0. Numerical code solves Ax=b\mathbf{A}\mathbf{x} = \mathbf{b} directly instead of forming the inverse.

Taught in Determinants, inverses and rank
Jacobian

For a function from Rn\mathbb{R}^n to Rm\mathbb{R}^m, the m×nm \times n matrix of partial derivatives Jij=∂fi/∂xjJ_{ij} = \partial f_i / \partial x_j. Near a point the function acts like this matrix, and the chain rule multiplies Jacobians.

Taught in Derivatives, gradients and Jacobians
KL divergence · also Kullback-Leibler divergence

KL⁡(π ∥ πref)=∑yπ(y)ln⁡π(y)πref(y)\operatorname{KL}(\pi \,\|\, \pi_{\text{ref}}) = \sum_y \pi(y)\ln\frac{\pi(y)}{\pi_{\text{ref}}(y)}, a measure of how far one probability distribution is from another: zero when they agree and positive otherwise. RLHF subtracts it, times a coefficient β\beta, as a penalty for drifting from the reference model.

Taught in From LSTMs to today's models
KV cache

The keys and values of every earlier token, kept in memory while a model generates, so that attention for each new token needs only that token's own new query, key and value. It holds one key vector and one value vector per token, per head, per layer, so it grows linearly with the context.

Taught in From LSTMs to today's models
Language model · also LM

A function that takes a sequence of tokens and returns a probability for every token in the vocabulary being next, pθ(xt+1∣x1,…,xt)p_\theta(x_{t+1} \mid x_1,\allowbreak \ldots,\allowbreak x_t). It generates text by repeated sampling: draw a token, append it, and ask again.

Taught in The map
Layer normalization · also LayerNorm

Standardizing each vector on its own, (x−μ)/σ2+ε(\mathbf{x} - \mu)/\sqrt{\sigma^2 + \varepsilon} with that vector's mean and variance, then applying a learned per-feature scale and shift. It is used inside transformer blocks; RMSNorm is a variant that skips subtracting the mean.

Taught in The map
Learning rate · also step size

The step size η\eta in gradient descent. Too small and training crawls; too large and it overshoots or diverges. On a quadratic bowl, steps are stable only when η<2/λmax⁡\eta < 2/\lambda_{\max}, where λmax⁡\lambda_{\max} is the largest curvature.

Taught in Derivatives, gradients and Jacobians
Learning-rate schedule · also schedule, warmup

A plan for changing the learning rate during training. Lowering it late in training shrinks the jitter that mini-batch noise causes; large models typically use a short warmup from near zero followed by a slow decay.

Taught in Derivatives, gradients and Jacobians
Linear combination

A sum of scaled vectors, c1a1+c2a2+⋯c_1\mathbf{a}_1 + c_2\mathbf{a}_2 + \cdots, where the numbers cic_i are the coefficients.

Taught in Vectors and the dot product
Linear independence · also independent vectors

Vectors are linearly independent when none of them is a linear combination of the others. Equivalently, the only combination that gives the zero vector has every coefficient equal to zero.

Taught in Vectors and the dot product
Linear transformation · also linear map

A function that respects addition and scaling: f(x+y)=f(x)+f(y)f(\mathbf{x} + \mathbf{y}) = f(\mathbf{x}) + f(\mathbf{y}) and f(cx)=c f(x)f(c\mathbf{x}) = c\,f(\mathbf{x}). Pictured on a grid, it keeps lines straight, parallel lines parallel and evenly spaced, and the origin fixed. On vectors of numbers every such function is a matrix, whose columns are where the standard basis vectors land.

Taught in Matrices are machines
Log-sum-exp · also LSE, log-softmax

LSE⁡(z)=ln⁡∑jezj\operatorname{LSE}(\mathbf{z}) = \ln\sum_j e^{z_j}, computed stably as m+ln⁡∑jezj−mm + \ln\sum_j e^{z_j - m} with m=max⁡jzjm = \max_j z_j. Log-softmax is z−LSE⁡(z)\mathbf{z} - \operatorname{LSE}(\mathbf{z}) and the cross-entropy for class yy is LSE⁡(z)−zy\operatorname{LSE}(\mathbf{z}) - z_y, so code never takes the log of a probability.

Taught in Neural networks from scratch
Logistic regression

A single sigmoid neuron, p=σ(w⋅x+b)p = \sigma(\mathbf{w}\cdot\mathbf{x} + b), whose output is read as the probability of class 1. It is the classic linear classifier: its decision boundary is a straight line (a hyperplane).

Taught in Neural networks from scratch
Logits

The raw scores a model outputs for each class or token, before softmax turns them into probabilities.

Taught in Neural networks from scratch
LoRA · also low-rank adaptation

Low-rank adaptation: fine-tune a weight matrix by learning a low-rank update ΔW=BA\Delta\mathbf{W} = \mathbf{B}\mathbf{A} instead of changing all its entries. With inner dimension rr, a d×kd \times k matrix needs only r(d+k)r(d + k) new parameters.

Taught in From LSTMs to today's models
Loss · also loss function, objective

A single number that measures how wrong a model's predictions are on some data, written LL. Training adjusts the parameters to make it small. Language models use the cross-entropy loss; regression usually uses mean squared error.

Taught in The map
Loss masking

Counting the training loss only on some tokens, typically the response tokens of a fine-tuning example and not the prompt. The model still reads the masked tokens as context; it is just not trained to produce them.

Taught in From LSTMs to today's models
Low-rank approximation

The closest rank-kk matrix to a given matrix, found by keeping its kk largest singular values and their singular vectors. It stores an m×nm \times n matrix in k(m+n)k(m + n) numbers, with the singular values folded into one of the two factors.

Taught in Eigenvectors and eigenvalues
LSTM · also long short-term memory

Long short-term memory: a recurrent cell with a separate cell state updated additively, ct=ft⊙ct−1+it⊙gt\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \mathbf{g}_t, and sigmoid gates that decide what to forget, write and reveal. Introduced by Hochreiter and Schmidhuber (1997); the forget gate was added by Gers, Schmidhuber and Cummins (2000).

Taught in LSTMs
Mask R-CNN

Faster R-CNN with an added branch that predicts a pixel mask for each detected object (2017). This gives instance segmentation, which separates individual objects even when they belong to the same class.

Taught in From LSTMs to today's models
Matrix

A rectangular grid of numbers with mm rows and nn columns. Multiplying it by a vector with nn entries gives a vector with mm entries: each output entry is a row dotted with the input, and the whole output is a combination of the columns.

Taught in Matrices are machines
Matrix multiplication · also matmul

AB\mathbf{A}\mathbf{B} is the single transformation that applies B\mathbf{B} and then A\mathbf{A}. Entry (i,j)(i,\allowbreak j) is row ii of A\mathbf{A} dotted with column jj of B\mathbf{B}, and an m×nm \times n by n×pn \times p product takes mnpmnp multiply-adds. In general AB≠BA\mathbf{A}\mathbf{B} \ne \mathbf{B}\mathbf{A}.

Taught in Matrices are machines
Maximum likelihood · also maximum likelihood estimation, MLE

Choosing parameters that make the observed data as probable as possible under the model. For a language model, minimizing the average next-token cross-entropy is exactly this, because the per-token losses add up to −ln⁡-\ln of the probability of the whole text.

Taught in Neural networks from scratch
Mean squared error · also MSE

The average squared difference between predictions and targets, the standard loss for regression.

Taught in Neural networks from scratch
Momentum

A variant of gradient descent that steps along a velocity, a running sum of recent gradients in which each older one counts less: v←βv+∇L\mathbf{v} \leftarrow \beta\mathbf{v} + \nabla L, w←w−ηv\mathbf{w} \leftarrow \mathbf{w} - \eta\mathbf{v}. It damps zig-zagging across steep directions and builds speed along shallow ones.

Taught in Derivatives, gradients and Jacobians
Multi-head attention

Running hh attentions side by side, each with its own query, key and value projections of width dmodel/hd_{\text{model}}/h, then placing their outputs side by side and mixing them with one more learned matrix. Each head can learn its own pattern of where to look, for about the same work as one head of full width.

Taught in From LSTMs to today's models
Multilayer perceptron · also MLP, feed-forward network

A network of dense layers with nonlinearities between them. The feed-forward block inside each transformer layer is a small one.

Taught in Neural networks from scratch
Multiply-add

One multiplication together with the addition that accumulates it, the basic step of a dot product. An (m×n)(n×p)(m \times n)(n \times p) matrix product takes mnpmnp of them. Compute budgets are quoted in FLOPs (floating-point operations), where one multiply-add counts as two.

Taught in Matrices are machines
Neuron · also unit

A dot product plus a bias passed through a nonlinearity, ϕ(w⋅x+b)\phi(\mathbf{w}\cdot\mathbf{x} + b). A layer computes many neurons at once with one matrix multiplication.

Taught in Neural networks from scratch
Next-token prediction · also language modeling

The training task of language models: at every position, predict the token that comes next, scored with cross-entropy. It is the pretraining objective of GPT-style models and the objective of the character models in modules 7 and 8.

Taught in Recurrent networks
Norm · also length, L2 norm, L1 norm, magnitude

The length of a vector. The usual L2 (Euclidean) norm is ∣x∣=x12+⋯+xn2|\mathbf{x}| = \sqrt{x_1^2 + \cdots + x_n^2}, which many sources write ∥x∥\|\mathbf{x}\|. The L1 norm, ∑i∣xi∣\sum_i |x_i|, appears in regularization.

Taught in Vectors and the dot product
Null space

Every input a matrix sends to zero. It contains more than the zero vector exactly when the columns are linearly dependent, which is when the matrix loses information.

Taught in Determinants, inverses and rank
Object detection

Finding each object in an image: drawing a box around it and naming its class.

Taught in From LSTMs to today's models
One-hot vector

A vector of zeros with a single 1 at the position given by an item's id. Multiplying a one-hot vector by a matrix picks out one row or column, which is what an embedding lookup does.

Taught in Recurrent networks
Open-weight model

A model whose trained parameters are published for anyone to download, run and fine-tune. What else is released (training data, code) and the license terms vary from model to model.

Taught in The map
Orthogonal matrix

A square matrix Q\mathbf{Q} whose columns have length 1 and are perpendicular to each other, so Q⊤Q=I\mathbf{Q}^\top\mathbf{Q} = \mathbf{I} and its inverse is its transpose. It is a rotation, possibly with a reflection: it keeps every length and angle, and all of its singular values are 1.

Taught in Eigenvectors and eigenvalues
Outer product

uv⊤\mathbf{u}\mathbf{v}^\top, a column times a row, which gives a matrix with entries uivju_i v_j. The outer product of two nonzero vectors has rank one, and AB\mathbf{A}\mathbf{B} is the sum of the outer products of the columns of A\mathbf{A} with the matching rows of B\mathbf{B}.

Taught in Matrices are machines
Output gate

The gate ot\mathbf{o}_t that scales how much of the cell state is revealed as the hidden state: ht=ot⊙tanh⁡(ct)\mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{c}_t).

Taught in LSTMs
Overfitting

Fitting the quirks of the training data instead of the underlying pattern: training loss keeps falling while validation loss stalls or rises.

Taught in Neural networks from scratch
Parameter · also weight

A number the model learns in training, usually an entry of a weight matrix or a bias vector. A 70B model has about 70 billion of them.

Taught in The map
Partial derivative

The derivative of a function of several inputs with respect to one of them, with all the others held fixed, written ∂f/∂xi\partial f / \partial x_i. The gradient collects the partial derivatives into a vector.

Taught in Derivatives, gradients and Jacobians
Perplexity

The exponential of a language model's average cross-entropy in natural-log units, eLe^{L}: roughly the number of equally likely tokens the model is choosing between at each step. A uniform model over VV tokens has perplexity VV.

Taught in Neural networks from scratch
Policy gradient

The identity ∇θ Ey∼πθ[R(y)]=Ey∼πθ[R(y) ∇θln⁡πθ(y)]\nabla_\theta\,\mathbb{E}_{y \sim \pi_\theta}[R(y)] = \mathbb{E}_{y \sim \pi_\theta}[R(y)\,\nabla_\theta\ln\pi_\theta(y)], which trains a model on rewards that cannot be differentiated, such as a pass or fail check. In practice the reward is replaced by an advantage: the reward minus a baseline such as the average over a group of attempts.

Taught in From LSTMs to today's models
Pooling · also max pooling

Shrinking a feature map by summarizing each small block, most often by its largest value (max pooling). A 2×22 \times 2 window with stride 2 halves the height and the width.

Taught in From LSTMs to today's models
Post-training

Everything done to a pretrained model to make it useful and safe: supervised fine-tuning, preference optimization (RLHF, DPO), reinforcement learning with verifiable rewards, safety tuning and distillation. It uses far less data than pretraining and, when adapting a model to a domain, usually a small fraction of the compute.

Taught in From LSTMs to today's models
Power iteration

Finding the dominant eigenvector by multiplying a starting vector by the matrix again and again, rescaling each time. The component along the eigenvector with the largest ∣λ∣|\lambda| outgrows all the others.

Taught in Eigenvectors and eigenvalues
Pretraining

Training a model from scratch to predict the next token, with the cross-entropy loss, on an enormous amount of raw text (trillions of tokens). It takes enormous compute, usually the largest share of a model's training budget, and produces a base model.

Taught in From LSTMs to today's models
Principal component analysis · also PCA

Finding the directions in which data varies most: the eigenvectors of its covariance matrix, ordered by eigenvalue, where each eigenvalue is the variance of the data along its direction. Keeping the top few compresses the data.

Taught in Eigenvectors and eigenvalues
Projection

The shadow of a\mathbf{a} on the line through b\mathbf{b}, proj⁡ba=a⋅bb⋅b b\operatorname{proj}_{\mathbf{b}}\mathbf{a} = \frac{\mathbf{a}\cdot\mathbf{b}}{\mathbf{b}\cdot\mathbf{b}}\,\mathbf{b}. What is left over, a−proj⁡ba\mathbf{a} - \operatorname{proj}_{\mathbf{b}}\mathbf{a}, is perpendicular to b\mathbf{b}.

Taught in Vectors and the dot product
Quantization

Storing a model's numbers in fewer bits, for example 8 or 4 instead of 16, to cut memory and speed up inference at some cost in precision. An 8-billion-parameter model at 4 bits per weight takes about 4 GB.

Taught in The map
Query, key and value · also QKV

The three vectors attention computes for each position by multiplying its representation by learned matrices. The query is compared with every key by a dot product, and the resulting weights average the values.

Taught in From LSTMs to today's models
R-CNN

An object detector (2014) that proposed about 2,000 candidate regions per image with a classical algorithm and ran a CNN separately on each one. Accurate for its time, and slow.

Taught in From LSTMs to today's models
Rank

The number of dimensions that survive a matrix: the dimension of its column space, equal to its number of linearly independent columns. A rank-kk matrix can be written as a sum of kk outer products.

Taught in Determinants, inverses and rank
Receptive field

The patch of input pixels that one output of a convolutional network depends on. With stride 1, each layer of k×kk \times k kernels widens it by k−1k - 1, so LL layers see 1+L(k−1)1 + L(k - 1) pixels on a side; strides and pooling make it grow faster.

Taught in From LSTMs to today's models
Recurrent neural network · also RNN

A network that reads a sequence one step at a time, updating a hidden state with the same weights at every step: ht=tanh⁡(Whhht−1+Wxhxt+b)\mathbf{h}_t = \tanh(\mathbf{W}_{hh}\mathbf{h}_{t-1} + \mathbf{W}_{xh}\mathbf{x}_t + \mathbf{b}). Unrolled over time, it is a deep network with tied weights.

Taught in Recurrent networks
Reinforcement learning with verifiable rewards · also RLVR

Post-training in which the model makes many attempts at problems a program can check (math with known answers, code with tests), and reinforcement learning makes the successful attempts more likely. It is central to recent reasoning models.

Taught in From LSTMs to today's models
ReLU · also rectified linear unit

max⁡(0,x)\max(0,\allowbreak x): zero for negative inputs and the identity for positive ones. It is the default activation of deep feed-forward networks, partly because its slope is exactly 1 wherever it is active.

Taught in Neural networks from scratch
Residual connection · also skip connection, residual stream

Adding a block's input to its output, x+F(x)\mathbf{x} + F(\mathbf{x}), which gives information and gradients a direct additive path through a deep network, much like the LSTM's cell state.

Taught in From LSTMs to today's models
Reward model

A model trained to give a response a single score that predicts how much people would prefer it. RLHF optimizes the language model against it.

Taught in From LSTMs to today's models
RLHF · also reinforcement learning from human feedback

Reinforcement learning from human feedback: people compare pairs of responses, a reward model learns to predict their preferences, and reinforcement learning (classically PPO) pushes the language model toward high-reward responses, with a penalty for drifting far from where it started.

Taught in From LSTMs to today's models
Saddle point

A point where the gradient is zero but the surface curves up in some directions and down in others, so it is neither a minimum nor a maximum. Gradient descent can slow down near one.

Taught in Derivatives, gradients and Jacobians
Saturation

The state of a sigmoid or tanh unit whose input is far from zero, where the curve is almost flat. Its derivative is then near zero, so little gradient flows back through it.

Taught in Derivatives, gradients and Jacobians
Scaled dot-product attention

softmax(QK⊤/d) V\mathrm{softmax}(\mathbf{Q}\mathbf{K}^\top/\sqrt{d})\,\mathbf{V}. Dividing by d\sqrt{d} keeps the spread of the scores from growing with the dimension dd of the keys, which would otherwise push the softmax into saturation.

Taught in From LSTMs to today's models
Scaling laws

Empirical regularities in which a model's loss falls smoothly and predictably, roughly as a power law, as parameters, data and compute grow. They are used to choose model and data size for a compute budget.

Taught in From LSTMs to today's models
Semantic segmentation

Labeling every pixel of an image with a class, such as road, car or sky, without separating individual objects. Fully convolutional networks (FCN) and U-Net do it in one pass that outputs an image-sized map of class scores.

Taught in From LSTMs to today's models
Shape

The size of an array along each axis, such as (32, 10) for 32 examples with 10 features. A product (n, d) @ (d, k) has shape (n, k): the inner sizes must match and disappear. Checking shapes is the first step in debugging model code.

Taught in Matrices are machines
Sigmoid · also logistic function

σ(x)=11+e−x\sigma(x) = \frac{1}{1 + e^{-x}}, which squashes any number into the interval from 0 to 1, with derivative σ(x) (1−σ(x))\sigma(x)\,(1 - \sigma(x)). LSTM gates use it as a soft switch.

Taught in Derivatives, gradients and Jacobians
Singular matrix

A square matrix with determinant zero. It squashes space into fewer dimensions, sends some nonzero vector to zero and has no inverse. A square matrix that is not singular is called invertible.

Taught in Determinants, inverses and rank
Singular value

One of the stretch factors σ1≥σ2≥⋯≥0\sigma_1 \ge \sigma_2 \ge \cdots \ge 0 on the diagonal of Σ\boldsymbol{\Sigma} in the SVD. For a 2×22 \times 2 matrix they are the semi-axes of the ellipse it makes from the unit circle. The largest bounds growth, ∣Ax∣≤σ1∣x∣|\mathbf{A}\mathbf{x}| \le \sigma_1|\mathbf{x}|; many sources call σ1\sigma_1 the spectral norm and write it ∥A∥2\|\mathbf{A}\|_2.

Taught in Eigenvectors and eigenvalues
Singular value decomposition · also SVD

The factorization of any matrix as A=UΣV⊤\mathbf{A} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^\top: a rotation (possibly with a reflection), a stretch along the axes by the singular values, and another rotation.

Taught in Eigenvectors and eigenvalues
Softmax

Turns scores z\mathbf{z} into probabilities, pi=ezi/∑jezjp_i = e^{z_i} / \sum_j e^{z_j}. Subtracting the largest score first gives the same result and avoids overflow. The logarithm of the denominator is called log-sum-exp.

Taught in Neural networks from scratch
Span

The set of all linear combinations of some vectors. Two vectors in the plane that do not lie on a common line through the origin span the whole plane; two on a common line span only that line.

Taught in Vectors and the dot product
Spectral radius

The largest eigenvalue size, ρ(A)=max⁡i∣λi∣\rho(\mathbf{A}) = \max_i |\lambda_i|. Below 1, repeated multiplication by A\mathbf{A} eventually shrinks every vector toward zero; above 1, almost every vector eventually blows up.

Taught in Eigenvectors and eigenvalues
Stochastic gradient descent · also SGD, mini-batch gradient descent

Gradient descent that estimates the gradient from a mini-batch, a small random subset of the training data, at each step. Each step is noisier than with the full gradient but far cheaper.

Taught in Derivatives, gradients and Jacobians
Stride and padding · also stride, padding

The stride ss is how far a convolution's kernel moves between positions; padding pp adds rings of zeros around the border. Along an axis of length nn, a kernel of size kk fits at ⌊(n+2p−k)/s⌋+1\lfloor (n + 2p - k)/s \rfloor + 1 positions.

Taught in From LSTMs to today's models
Supervised fine-tuning · also SFT, instruction tuning

Further training of a pretrained model on curated demonstrations, each a prompt with an ideal response, using the same next-token cross-entropy loss. The loss is often counted only on the response tokens, which is called loss masking.

Taught in From LSTMs to today's models
Symmetric matrix

A square matrix equal to its transpose, A=A⊤\mathbf{A} = \mathbf{A}^\top. Its eigenvalues are real and its eigenvectors can be chosen perpendicular to each other. Covariance matrices and Hessians are symmetric.

Taught in Eigenvectors and eigenvalues
Tanh · also hyperbolic tangent

The hyperbolic tangent, which squashes numbers into the interval from −1-1 to 11 and is centered at zero, with derivative 1−tanh⁡2x1 - \tanh^2 x. It is the nonlinearity of plain RNNs and of the LSTM candidate.

Taught in Derivatives, gradients and Jacobians
Teacher forcing

Training a sequence model by feeding it the true previous token at every step rather than its own prediction. The targets are the inputs shifted by one position.

Taught in Recurrent networks
Temperature

A number T>0T > 0 that divides the logits before softmax. Below 1 it sharpens the distribution toward the top choice; above 1 it flattens it, so sampling becomes more varied. Module 7 writes it τ\tau, because there TT is the sequence length.

Taught in Neural networks from scratch
Token

The unit of text a model reads and writes: a common word, a piece of a rarer word, a punctuation mark or a space, depending on the tokenizer. The model sees each token as an integer id.

Taught in The map
Tokenizer · also tokenization

The fixed procedure that splits text into tokens from a vocabulary and maps them to integer ids and back. Most current tokenizers are subword schemes such as byte-pair encoding (BPE), built by repeatedly merging the most frequent adjacent pair of symbols into a new token.

Taught in The map
Top-k and top-p sampling · also top-k, top-p, nucleus sampling

Two ways of cutting off the unlikely tail before sampling the next token: top-k keeps the kk most probable tokens, top-p keeps the smallest set of most probable tokens whose probabilities add up to at least pp. The kept probabilities are renormalized to sum to 1.

Taught in Recurrent networks
Trace

The sum of the diagonal entries of a square matrix, tr⁡A\operatorname{tr}\mathbf{A}. It equals the sum of the eigenvalues, which makes it a quick check on eigenvalues found by hand.

Taught in Eigenvectors and eigenvalues
Training compute estimate · also 6ND

The rule of thumb that training a dense model with NN parameters on DD tokens takes about C≈6NDC \approx 6ND operations: about 2 per parameter per token for the forward pass and 4 for the backward pass.

Taught in From LSTMs to today's models
Transformer

The architecture introduced in 2017 in "Attention Is All You Need" that replaced recurrence with attention. Each block combines self-attention and an MLP with residual connections and normalization, and stacks of these blocks underlie today's large language models.

Taught in From LSTMs to today's models
Transpose

A⊤\mathbf{A}^\top swaps rows and columns, so entry (i,j)(i,\allowbreak j) becomes entry (j,i)(j,\allowbreak i). It reverses products, (AB)⊤=B⊤A⊤(\mathbf{A}\mathbf{B})^\top = \mathbf{B}^\top\mathbf{A}^\top, and writes the dot product as a⊤b\mathbf{a}^\top\mathbf{b}.

Taught in Matrices are machines
U-Net

A fully convolutional segmentation network (2015), designed for biomedical images: a path that shrinks the image into features, a path that expands back to full resolution, and skip connections between matching levels.

Taught in From LSTMs to today's models
Unit vector

A vector of length 1. Dividing a nonzero vector by its length, x/∣x∣\mathbf{x}/|\mathbf{x}|, gives the unit vector in its direction; this is called normalizing.

Taught in Vectors and the dot product
Universal approximation theorem

A network with one hidden layer and enough units can approximate any continuous function on a bounded region as closely as you like, for suitable activations such as sigmoid or ReLU. It says such a network exists, not that training will find it or that it is small.

Taught in Neural networks from scratch
Validation set · also held-out set

Examples held out from training to measure how well the model generalizes and to choose settings such as the learning rate.

Taught in Neural networks from scratch
Vanishing gradient

A gradient that shrinks geometrically as it is passed back through many steps or layers, because each step multiplies it by factors smaller than 1. It is the main reason plain RNNs struggle to learn dependencies more than a short distance back.

Taught in Recurrent networks
Variance · also standard deviation

The average squared distance from the mean μ\mu: σ2=1n∑i=1n(xi−μ)2\sigma^2 = \frac{1}{n}\sum_{i=1}^{n} (x_i - \mu)^2. Its square root σ\sigma is the standard deviation. This course, like np.var, divides by nn rather than n−1n - 1; the covariance matrix of module 4 follows np.cov and divides by n−1n - 1.

Taught in The map
Vector

An ordered list of numbers that can also be drawn as an arrow from the origin. Math in this course writes vectors in bold, v\mathbf{v}, and treats them as columns.

Taught in Vectors and the dot product
Vector-Jacobian product

A gradient vector multiplied by a Jacobian, J⊤g\mathbf{J}^\top\mathbf{g} in column form. Backpropagation passes gradients backwards as a chain of these products and never builds full Jacobians.

Taught in Derivatives, gradients and Jacobians
Vision transformer · also ViT

A transformer applied to images by cutting each image into patches and treating every patch as a token.

Taught in From LSTMs to today's models
Vocabulary

The fixed set of tokens (or characters) a model can read and produce. Its size VV is the number of outputs of the model's final softmax.

Taught in Recurrent networks
Weight decay

Shrinking every weight by a small fraction of itself at each step, a pull toward zero that discourages overfitting. With plain gradient descent it amounts to adding a squared-L2 penalty to the loss; AdamW applies it separately from Adam's per-parameter scaling.

Taught in Neural networks from scratch
Weight sharing · also tied weights

Using the same weights at many positions: across time in an RNN, across space in a CNN. One learned pattern then applies everywhere, and the parameter count does not grow with the input's length or size.

Taught in Recurrent networks
Zero-shot classification

Classifying into classes a model was never trained on, by describing each class in text. With CLIP, each image gets the class whose caption embedding has the highest cosine similarity with the image's embedding, so changing the classes means changing a list of captions.

Taught in From LSTMs to today's models