Reference
Glossary
Every term the course uses, in plain words, with a link to where it is taught.
166 terms
- Activation function · also nonlinearity
- Taught in Neural networks from scratch
The elementwise nonlinearity applied after a layer's matrix multiplication, such as ReLU, tanh, sigmoid or GELU. Without it, a stack of layers collapses into a single layer: one matrix and one bias.
- Adam · also AdamW
- Taught in Neural networks from scratch
An optimizer (Kingma and Ba; the paper appeared in 2014 and was published at ICLR in 2015) that scales each parameter's step using running averages of its gradient and its squared gradient, with a bias correction for the first steps. It is the usual default for training networks, often with decoupled weight decay (AdamW).
- Attention · also self-attention
- Taught in From LSTMs to today's models
A weighted average of value vectors, with weights given by the softmax of dot-product scores between a query and each key. In self-attention the queries, keys and values all come from the same sequence, so every position can look directly at the others.
- Axis
- Taught in The map
One dimension of a numpy array. A reduction such as
X.sum(axis=0)collapses the named axis: for an(n, d)array,axis=0leaves one number per column andaxis=1one per row. - Backpropagation · also backprop, reverse-mode differentiation
- Taught in Neural networks from scratch
Computing the gradient of the loss with respect to every parameter by applying the chain rule backwards through the computational graph. The forward pass stores intermediate values; the backward pass multiplies local derivatives by the incoming gradient at each node, for a small multiple of the forward pass's cost.
- Backpropagation through time · also BPTT, truncated BPTT
- Taught in Recurrent networks
Backpropagation on an unrolled recurrent network. The gradient at each hidden state combines the contribution from that step's output with the one arriving from the next step, and each weight's gradient is a sum over time steps. Truncated BPTT stops the gradient after a fixed number of steps to bound the cost.
- Base model · also pretrained model
- Taught in From LSTMs to today's models
The model that comes out of pretraining. It continues text in the style of its training data and knows a great deal, but it does not reliably follow instructions until it is post-trained.
- Basis
- Taught in Vectors and the dot product
A set of linearly independent vectors that spans the space. Every vector is then exactly one combination of them, and its coefficients are the vector's coordinates in that basis.
- Batch
- Taught in Matrices are machines
A group of examples processed together. In code a batch is an array
Xof shape(batch, features)with one row per example, so a layer isX @ W + b; math in this course writes a single vector as a column instead. - Bias
- Taught in Neural networks from scratch
A learned constant added after the weighted sum, which shifts where a neuron switches on. For a batch, the bias vector is broadcast and added to every row.
- Bigram model
- Taught in Recurrent networks
A language model that predicts each character or token from the one before it alone, using counted frequencies , smoothed so that no pair gets probability 0. It is the simplest model with context, and an honest baseline that a trained network has to beat.
- Broadcasting
- Taught in Matrices are machines
numpy's rule for combining arrays of different shapes: compare the shapes from the right; each pair of sizes must be equal or include a 1, and size-1 or missing dimensions are stretched to match. It is how one bias vector is added to every row of a batch.
- Candidate · also candidate cell state
- Taught in LSTMs
The vector of new content an LSTM proposes to write into its cell state, computed with tanh. Some sources call it .
- Catastrophic forgetting
- Taught in From LSTMs to today's models
Losing abilities a model had before when it is trained further on narrow data. Lower learning rates, mixing in general data and parameter-efficient methods such as LoRA reduce it.
- Cauchy-Schwarz inequality
- Taught in Vectors and the dot product
, with equality exactly when one vector is a multiple of the other. It keeps cosine similarity between and , which is what lets an angle be defined between vectors in any number of dimensions.
- Causal mask
- Taught in From LSTMs to today's models
Blocking each position from attending to later positions, by setting those scores to before the softmax, so a language model predicts every token from earlier tokens only.
- Cell state
- Taught in LSTMs
The LSTM's memory vector . It is updated by elementwise scaling and adding rather than by a matrix multiplication, so information and gradients can pass through many steps nearly unchanged.
- Chain rule
- Taught in Derivatives, gradients and Jacobians
The derivative of a composition is the product of the derivatives: . For functions of vectors it becomes a product of Jacobians, and backpropagation applies it systematically.
- Chain rule of probability
- Taught in The map
: the probability of a sequence is the product of each token's probability given everything before it. It is why predicting the next token is enough to model whole texts.
- Characteristic polynomial · also characteristic equation
- Taught in Eigenvectors and eigenvalues
, whose roots are the eigenvalues. For a matrix it is , where the trace is the sum of the diagonal entries. The eigenvalues add up to the trace and multiply to the determinant.
- CLIP
- Taught in From LSTMs to today's models
A model that places images and text in one embedding space by training matching image and caption pairs to have high cosine similarity and mismatched pairs low similarity, so images and text can be compared directly.
- Column space · also range, image
- Taught in Determinants, inverses and rank
Every output a matrix can produce: the span of its columns. Its dimension is the rank.
- Computational graph
- Taught in Neural networks from scratch
A computation drawn as nodes, each one simple operation, joined by edges that carry values. The forward pass fills in every node's value; backpropagation visits the nodes in reverse and multiplies each node's local derivatives by the gradient arriving from above.
- Condition number
- Taught in Determinants, inverses and rank
The ratio of a matrix's largest to smallest stretch, ; for a quadratic loss, the ratio of its Hessian's eigenvalues. A large value makes solving systems sensitive to small errors and makes gradient descent zig-zag and converge slowly.
- Constant error carousel
- Taught in LSTMs
The name the original 1997 LSTM paper gave to the cell's self-connection, which had a fixed weight of 1 so that error signals could flow back through many steps without shrinking or growing. With a forget gate, the direct path from to has Jacobian , so gradients pass along it nearly unchanged while the forget gates stay near 1.
- Convolution
- Taught in From LSTMs to today's models
Sliding a small grid of weights, the kernel, across an image and computing at each position the sum of elementwise products with the patch underneath; the outputs form a feature map. Deep learning does not flip the kernel, so strictly this is cross-correlation.
- Convolutional neural network · also CNN, convnet
- Taught in From LSTMs to today's models
A network built mainly from convolution layers, which share weights across space so a pattern detector works anywhere in the image. The dominant architecture for images from 2012 on, now often combined with or replaced by vision transformers.
- Cosine similarity
- Taught in Vectors and the dot product
, the cosine of the angle between two vectors. The Cauchy-Schwarz inequality keeps it between and . It ignores length, which is why embedding search uses it.
- Covariance matrix
- Taught in Eigenvectors and eigenvalues
For data with features, the symmetric matrix whose diagonal holds each feature's variance and whose entry measures how features and vary together. From centered data with rows it is , as
np.covcomputes it. Its eigenvectors are the principal components. - Cross-entropy · also cross-entropy loss, negative log-likelihood, log loss
- Taught in Neural networks from scratch
The loss , where is the probability the model gave to the correct class or next token, averaged over examples. A uniform guess over classes scores . Combined with softmax, its gradient with respect to the logits is .
- Decision boundary
- Taught in Neural networks from scratch
The set of inputs where a classifier switches from one class to another. For a single neuron it is the line (a hyperplane in more dimensions) , perpendicular to ; a hidden layer lets a network draw curved boundaries.
- Dense layer · also fully connected layer, linear layer
- Taught in Matrices are machines
A matrix multiplication plus a bias, for one column vector or
X @ W + bfor a batch in code, usually followed by a nonlinearity. - Derivative
- Taught in Derivatives, gradients and Jacobians
The slope of the best straight-line approximation to a function at a point: for small . That linear approximation is the idea behind every gradient in training.
- Determinant
- Taught in Determinants, inverses and rank
The factor by which a square matrix scales area (volume in 3D). A negative value means orientation is flipped; zero means space is squashed flat, so the matrix is singular and has no inverse. For it is .
- Diagonalization
- Taught in Eigenvectors and eigenvalues
Writing with eigenvectors as the columns of and eigenvalues on the diagonal of , which is possible when an matrix has linearly independent eigenvectors. Then : in the eigenvector basis, a power only raises each eigenvalue to the th power.
- Distillation · also knowledge distillation
- Taught in From LSTMs to today's models
Training a smaller student model to match a larger teacher model's outputs or its full next-token probabilities, keeping much of the quality at a lower serving cost.
- Dot product · also inner product, scalar product
- Taught in Vectors and the dot product
: multiply matching entries and add. It equals , so it is positive when two vectors roughly agree, zero when they are perpendicular (orthogonal) and negative when they disagree.
- DPO · also direct preference optimization
- Taught in From LSTMs to today's models
Direct preference optimization: learning from preference pairs with one loss that raises the preferred response's probability relative to the rejected one, both measured against a frozen reference model. It needs no reward model and no reinforcement-learning loop.
- Dropout
- Taught in LSTMs
Randomly setting a fraction of activations to zero during training and rescaling the rest, which discourages units from relying on each other and reduces overfitting. It is switched off at inference.
- Early stopping
- Taught in Neural networks from scratch
Keeping the weights from the step where the validation loss was lowest, instead of the ones at the end of training. It is a simple defense against overfitting.
- Eigenvalue
- Taught in Eigenvectors and eigenvalues
The factor by which a matrix stretches one of its eigenvectors. A negative flips it. Complex eigenvalues, as for a rotation, mean no real direction stays on its line, though still measures growth.
- Eigenvector
- Taught in Eigenvectors and eigenvalues
A nonzero vector that a square matrix only stretches or flips, without turning it off its line: .
- Elementwise product · also Hadamard product
- Taught in Derivatives, gradients and Jacobians
Multiplying two vectors or matrices of the same shape entry by entry, written ; in numpy it is
*. - Embedding · also embedding table
- Taught in Recurrent networks
A learned vector that represents a discrete item such as a token, typically with hundreds or thousands of entries. An embedding table stores one row per token, and looking a token up is the same as multiplying its one-hot vector by that table.
- Epoch
- Taught in Neural networks from scratch
One full pass through the training data.
- Exploding gradient
- Taught in Recurrent networks
A gradient that grows geometrically as it is passed back through many steps or layers, because each step multiplies it by factors larger than 1. Gradient clipping is the standard remedy.
- Fast R-CNN
- Taught in From LSTMs to today's models
R-CNN's successor (2015): run the CNN once on the whole image, crop each proposed region's features from the shared feature map (RoI pooling), and classify them with a small network.
- Faster R-CNN
- Taught in From LSTMs to today's models
Fast R-CNN plus a small learned region proposal network that proposes boxes from the same shared feature map, replacing the slow hand-designed proposal step (2015).
- Finite difference · also central difference
- Taught in Derivatives, gradients and Jacobians
Estimating a derivative from function values. The central difference has error proportional to , far smaller than the forward difference's error proportional to . It is the basis of gradient checking.
- Forget gate
- Taught in LSTMs
The gate that scales the previous cell state: near 1 keeps it, near 0 erases it.
- Forget-gate bias
- Taught in LSTMs
The bias of the LSTM's forget gate, commonly initialized to 1 or more so the gate starts mostly open and the cell remembers by default early in training.
- Gate
- Taught in LSTMs
A vector of values between 0 and 1, computed by a sigmoid from the current input and the previous hidden state, that multiplies another vector entry by entry to control how much of it passes.
- GELU · also Gaussian error linear unit
- Taught in Neural networks from scratch
A smooth ReLU-like activation, , where is the cumulative distribution function of the standard normal distribution. It is used in many transformers.
- Gradient
- Taught in Derivatives, gradients and Jacobians
The vector of partial derivatives , each the rate of change when only moves. It points in the direction of steepest increase, its length is that steepest slope, and the rate of change along a unit direction is .
- Gradient checking · also gradient check
- Taught in Neural networks from scratch
Testing a hand-written backward pass by comparing its gradients with finite-difference estimates, measured by relative error. A tiny relative error means the backward pass is almost certainly right; a large one points to a bug.
- Gradient clipping · also clipping by norm
- Taught in Recurrent networks
Rescaling the gradient when its norm exceeds a threshold : if , replace with . This keeps the direction and caps the size of the step.
- Gradient descent
- Taught in Derivatives, gradients and Jacobians
Minimizing a loss by repeatedly stepping against its gradient, . With a small enough step size, each step lowers the loss.
- Greedy decoding
- Taught in Recurrent networks
Generating text by always choosing the single most probable next token. It is deterministic and tends toward repetitive text; sampling with a temperature adds variety.
- GRU · also gated recurrent unit
- Taught in LSTMs
The gated recurrent unit, a simpler relative of the LSTM with two gates (update and reset) and no separate cell state. With the same sizes it has three quarters of an LSTM's weights.
- Hessian
- Taught in Derivatives, gradients and Jacobians
The matrix of second partial derivatives of a scalar function. Its eigenvalues are the function's curvatures along the Hessian's eigenvector directions.
- Identity matrix
- Taught in Matrices are machines
The square matrix with ones on the diagonal and zeros everywhere else. It moves nothing, , so .
- Inference · also serving
- Taught in The map
Running a trained model to produce outputs, as opposed to training it. Serving a language model means generating tokens for many users at once, usually with batching, caching of earlier computation and quantized weights.
- Initialization · also weight initialization, Xavier initialization, Glorot initialization, He initialization
- Taught in Neural networks from scratch
The choice of starting weights. They must be random to break the symmetry between units, and scaled so signals neither shrink nor blow up across layers, as in Xavier (Glorot) and He initialization.
- Input gate
- Taught in LSTMs
The gate that scales how much of the candidate is written into the cell state.
- Instance segmentation
- Taught in From LSTMs to today's models
Giving each object in an image its own pixel mask, so two objects of the same class, such as two people standing together, stay separate. Mask R-CNN does it by adding a mask branch to a detector.
- Inverse · also matrix inverse
- Taught in Determinants, inverses and rank
The matrix that undoes , so . It exists exactly when . Numerical code solves directly instead of forming the inverse.
- Jacobian
- Taught in Derivatives, gradients and Jacobians
For a function from to , the matrix of partial derivatives . Near a point the function acts like this matrix, and the chain rule multiplies Jacobians.
- KL divergence · also Kullback-Leibler divergence
- Taught in From LSTMs to today's models
, a measure of how far one probability distribution is from another: zero when they agree and positive otherwise. RLHF subtracts it, times a coefficient , as a penalty for drifting from the reference model.
- KV cache
- Taught in From LSTMs to today's models
The keys and values of every earlier token, kept in memory while a model generates, so that attention for each new token needs only that token's own new query, key and value. It holds one key vector and one value vector per token, per head, per layer, so it grows linearly with the context.
- Language model · also LM
- Taught in The map
A function that takes a sequence of tokens and returns a probability for every token in the vocabulary being next, . It generates text by repeated sampling: draw a token, append it, and ask again.
- Layer normalization · also LayerNorm
- Taught in The map
Standardizing each vector on its own, with that vector's mean and variance, then applying a learned per-feature scale and shift. It is used inside transformer blocks; RMSNorm is a variant that skips subtracting the mean.
- Learning rate · also step size
- Taught in Derivatives, gradients and Jacobians
The step size in gradient descent. Too small and training crawls; too large and it overshoots or diverges. On a quadratic bowl, steps are stable only when , where is the largest curvature.
- Learning-rate schedule · also schedule, warmup
- Taught in Derivatives, gradients and Jacobians
A plan for changing the learning rate during training. Lowering it late in training shrinks the jitter that mini-batch noise causes; large models typically use a short warmup from near zero followed by a slow decay.
- Linear combination
- Taught in Vectors and the dot product
A sum of scaled vectors, , where the numbers are the coefficients.
- Linear independence · also independent vectors
- Taught in Vectors and the dot product
Vectors are linearly independent when none of them is a linear combination of the others. Equivalently, the only combination that gives the zero vector has every coefficient equal to zero.
- Linear transformation · also linear map
- Taught in Matrices are machines
A function that respects addition and scaling: and . Pictured on a grid, it keeps lines straight, parallel lines parallel and evenly spaced, and the origin fixed. On vectors of numbers every such function is a matrix, whose columns are where the standard basis vectors land.
- Log-sum-exp · also LSE, log-softmax
- Taught in Neural networks from scratch
, computed stably as with . Log-softmax is and the cross-entropy for class is , so code never takes the log of a probability.
- Logistic regression
- Taught in Neural networks from scratch
A single sigmoid neuron, , whose output is read as the probability of class 1. It is the classic linear classifier: its decision boundary is a straight line (a hyperplane).
- Logits
- Taught in Neural networks from scratch
The raw scores a model outputs for each class or token, before softmax turns them into probabilities.
- LoRA · also low-rank adaptation
- Taught in From LSTMs to today's models
Low-rank adaptation: fine-tune a weight matrix by learning a low-rank update instead of changing all its entries. With inner dimension , a matrix needs only new parameters.
- Loss · also loss function, objective
- Taught in The map
A single number that measures how wrong a model's predictions are on some data, written . Training adjusts the parameters to make it small. Language models use the cross-entropy loss; regression usually uses mean squared error.
- Loss masking
- Taught in From LSTMs to today's models
Counting the training loss only on some tokens, typically the response tokens of a fine-tuning example and not the prompt. The model still reads the masked tokens as context; it is just not trained to produce them.
- Low-rank approximation
- Taught in Eigenvectors and eigenvalues
The closest rank- matrix to a given matrix, found by keeping its largest singular values and their singular vectors. It stores an matrix in numbers, with the singular values folded into one of the two factors.
- LSTM · also long short-term memory
- Taught in LSTMs
Long short-term memory: a recurrent cell with a separate cell state updated additively, , and sigmoid gates that decide what to forget, write and reveal. Introduced by Hochreiter and Schmidhuber (1997); the forget gate was added by Gers, Schmidhuber and Cummins (2000).
- Mask R-CNN
- Taught in From LSTMs to today's models
Faster R-CNN with an added branch that predicts a pixel mask for each detected object (2017). This gives instance segmentation, which separates individual objects even when they belong to the same class.
- Matrix
- Taught in Matrices are machines
A rectangular grid of numbers with rows and columns. Multiplying it by a vector with entries gives a vector with entries: each output entry is a row dotted with the input, and the whole output is a combination of the columns.
- Matrix multiplication · also matmul
- Taught in Matrices are machines
is the single transformation that applies and then . Entry is row of dotted with column of , and an by product takes multiply-adds. In general .
- Maximum likelihood · also maximum likelihood estimation, MLE
- Taught in Neural networks from scratch
Choosing parameters that make the observed data as probable as possible under the model. For a language model, minimizing the average next-token cross-entropy is exactly this, because the per-token losses add up to of the probability of the whole text.
- Mean squared error · also MSE
- Taught in Neural networks from scratch
The average squared difference between predictions and targets, the standard loss for regression.
- Momentum
- Taught in Derivatives, gradients and Jacobians
A variant of gradient descent that steps along a velocity, a running sum of recent gradients in which each older one counts less: , . It damps zig-zagging across steep directions and builds speed along shallow ones.
- Multi-head attention
- Taught in From LSTMs to today's models
Running attentions side by side, each with its own query, key and value projections of width , then placing their outputs side by side and mixing them with one more learned matrix. Each head can learn its own pattern of where to look, for about the same work as one head of full width.
- Multilayer perceptron · also MLP, feed-forward network
- Taught in Neural networks from scratch
A network of dense layers with nonlinearities between them. The feed-forward block inside each transformer layer is a small one.
- Multiply-add
- Taught in Matrices are machines
One multiplication together with the addition that accumulates it, the basic step of a dot product. An matrix product takes of them. Compute budgets are quoted in FLOPs (floating-point operations), where one multiply-add counts as two.
- Neuron · also unit
- Taught in Neural networks from scratch
A dot product plus a bias passed through a nonlinearity, . A layer computes many neurons at once with one matrix multiplication.
- Next-token prediction · also language modeling
- Taught in Recurrent networks
The training task of language models: at every position, predict the token that comes next, scored with cross-entropy. It is the pretraining objective of GPT-style models and the objective of the character models in modules 7 and 8.
- Norm · also length, L2 norm, L1 norm, magnitude
- Taught in Vectors and the dot product
The length of a vector. The usual L2 (Euclidean) norm is , which many sources write . The L1 norm, , appears in regularization.
- Null space
- Taught in Determinants, inverses and rank
Every input a matrix sends to zero. It contains more than the zero vector exactly when the columns are linearly dependent, which is when the matrix loses information.
- Object detection
- Taught in From LSTMs to today's models
Finding each object in an image: drawing a box around it and naming its class.
- One-hot vector
- Taught in Recurrent networks
A vector of zeros with a single 1 at the position given by an item's id. Multiplying a one-hot vector by a matrix picks out one row or column, which is what an embedding lookup does.
- Open-weight model
- Taught in The map
A model whose trained parameters are published for anyone to download, run and fine-tune. What else is released (training data, code) and the license terms vary from model to model.
- Orthogonal matrix
- Taught in Eigenvectors and eigenvalues
A square matrix whose columns have length 1 and are perpendicular to each other, so and its inverse is its transpose. It is a rotation, possibly with a reflection: it keeps every length and angle, and all of its singular values are 1.
- Outer product
- Taught in Matrices are machines
, a column times a row, which gives a matrix with entries . The outer product of two nonzero vectors has rank one, and is the sum of the outer products of the columns of with the matching rows of .
- Output gate
- Taught in LSTMs
The gate that scales how much of the cell state is revealed as the hidden state: .
- Overfitting
- Taught in Neural networks from scratch
Fitting the quirks of the training data instead of the underlying pattern: training loss keeps falling while validation loss stalls or rises.
- Parameter · also weight
- Taught in The map
A number the model learns in training, usually an entry of a weight matrix or a bias vector. A 70B model has about 70 billion of them.
- Partial derivative
- Taught in Derivatives, gradients and Jacobians
The derivative of a function of several inputs with respect to one of them, with all the others held fixed, written . The gradient collects the partial derivatives into a vector.
- Perplexity
- Taught in Neural networks from scratch
The exponential of a language model's average cross-entropy in natural-log units, : roughly the number of equally likely tokens the model is choosing between at each step. A uniform model over tokens has perplexity .
- Policy gradient
- Taught in From LSTMs to today's models
The identity , which trains a model on rewards that cannot be differentiated, such as a pass or fail check. In practice the reward is replaced by an advantage: the reward minus a baseline such as the average over a group of attempts.
- Pooling · also max pooling
- Taught in From LSTMs to today's models
Shrinking a feature map by summarizing each small block, most often by its largest value (max pooling). A window with stride 2 halves the height and the width.
- Post-training
- Taught in From LSTMs to today's models
Everything done to a pretrained model to make it useful and safe: supervised fine-tuning, preference optimization (RLHF, DPO), reinforcement learning with verifiable rewards, safety tuning and distillation. It uses far less data than pretraining and, when adapting a model to a domain, usually a small fraction of the compute.
- Power iteration
- Taught in Eigenvectors and eigenvalues
Finding the dominant eigenvector by multiplying a starting vector by the matrix again and again, rescaling each time. The component along the eigenvector with the largest outgrows all the others.
- Pretraining
- Taught in From LSTMs to today's models
Training a model from scratch to predict the next token, with the cross-entropy loss, on an enormous amount of raw text (trillions of tokens). It takes enormous compute, usually the largest share of a model's training budget, and produces a base model.
- Principal component analysis · also PCA
- Taught in Eigenvectors and eigenvalues
Finding the directions in which data varies most: the eigenvectors of its covariance matrix, ordered by eigenvalue, where each eigenvalue is the variance of the data along its direction. Keeping the top few compresses the data.
- Projection
- Taught in Vectors and the dot product
The shadow of on the line through , . What is left over, , is perpendicular to .
- Quantization
- Taught in The map
Storing a model's numbers in fewer bits, for example 8 or 4 instead of 16, to cut memory and speed up inference at some cost in precision. An 8-billion-parameter model at 4 bits per weight takes about 4 GB.
- Query, key and value · also QKV
- Taught in From LSTMs to today's models
The three vectors attention computes for each position by multiplying its representation by learned matrices. The query is compared with every key by a dot product, and the resulting weights average the values.
- R-CNN
- Taught in From LSTMs to today's models
An object detector (2014) that proposed about 2,000 candidate regions per image with a classical algorithm and ran a CNN separately on each one. Accurate for its time, and slow.
- Rank
- Taught in Determinants, inverses and rank
The number of dimensions that survive a matrix: the dimension of its column space, equal to its number of linearly independent columns. A rank- matrix can be written as a sum of outer products.
- Receptive field
- Taught in From LSTMs to today's models
The patch of input pixels that one output of a convolutional network depends on. With stride 1, each layer of kernels widens it by , so layers see pixels on a side; strides and pooling make it grow faster.
- Recurrent neural network · also RNN
- Taught in Recurrent networks
A network that reads a sequence one step at a time, updating a hidden state with the same weights at every step: . Unrolled over time, it is a deep network with tied weights.
- Reinforcement learning with verifiable rewards · also RLVR
- Taught in From LSTMs to today's models
Post-training in which the model makes many attempts at problems a program can check (math with known answers, code with tests), and reinforcement learning makes the successful attempts more likely. It is central to recent reasoning models.
- ReLU · also rectified linear unit
- Taught in Neural networks from scratch
: zero for negative inputs and the identity for positive ones. It is the default activation of deep feed-forward networks, partly because its slope is exactly 1 wherever it is active.
- Residual connection · also skip connection, residual stream
- Taught in From LSTMs to today's models
Adding a block's input to its output, , which gives information and gradients a direct additive path through a deep network, much like the LSTM's cell state.
- Reward model
- Taught in From LSTMs to today's models
A model trained to give a response a single score that predicts how much people would prefer it. RLHF optimizes the language model against it.
- RLHF · also reinforcement learning from human feedback
- Taught in From LSTMs to today's models
Reinforcement learning from human feedback: people compare pairs of responses, a reward model learns to predict their preferences, and reinforcement learning (classically PPO) pushes the language model toward high-reward responses, with a penalty for drifting far from where it started.
- Saddle point
- Taught in Derivatives, gradients and Jacobians
A point where the gradient is zero but the surface curves up in some directions and down in others, so it is neither a minimum nor a maximum. Gradient descent can slow down near one.
- Saturation
- Taught in Derivatives, gradients and Jacobians
The state of a sigmoid or tanh unit whose input is far from zero, where the curve is almost flat. Its derivative is then near zero, so little gradient flows back through it.
- Scaled dot-product attention
- Taught in From LSTMs to today's models
. Dividing by keeps the spread of the scores from growing with the dimension of the keys, which would otherwise push the softmax into saturation.
- Scaling laws
- Taught in From LSTMs to today's models
Empirical regularities in which a model's loss falls smoothly and predictably, roughly as a power law, as parameters, data and compute grow. They are used to choose model and data size for a compute budget.
- Semantic segmentation
- Taught in From LSTMs to today's models
Labeling every pixel of an image with a class, such as road, car or sky, without separating individual objects. Fully convolutional networks (FCN) and U-Net do it in one pass that outputs an image-sized map of class scores.
- Shape
- Taught in Matrices are machines
The size of an array along each axis, such as
(32, 10)for 32 examples with 10 features. A product(n, d) @ (d, k)has shape(n, k): the inner sizes must match and disappear. Checking shapes is the first step in debugging model code. - Sigmoid · also logistic function
- Taught in Derivatives, gradients and Jacobians
, which squashes any number into the interval from 0 to 1, with derivative . LSTM gates use it as a soft switch.
- Singular matrix
- Taught in Determinants, inverses and rank
A square matrix with determinant zero. It squashes space into fewer dimensions, sends some nonzero vector to zero and has no inverse. A square matrix that is not singular is called invertible.
- Singular value
- Taught in Eigenvectors and eigenvalues
One of the stretch factors on the diagonal of in the SVD. For a matrix they are the semi-axes of the ellipse it makes from the unit circle. The largest bounds growth, ; many sources call the spectral norm and write it .
- Singular value decomposition · also SVD
- Taught in Eigenvectors and eigenvalues
The factorization of any matrix as : a rotation (possibly with a reflection), a stretch along the axes by the singular values, and another rotation.
- Softmax
- Taught in Neural networks from scratch
Turns scores into probabilities, . Subtracting the largest score first gives the same result and avoids overflow. The logarithm of the denominator is called log-sum-exp.
- Span
- Taught in Vectors and the dot product
The set of all linear combinations of some vectors. Two vectors in the plane that do not lie on a common line through the origin span the whole plane; two on a common line span only that line.
- Spectral radius
- Taught in Eigenvectors and eigenvalues
The largest eigenvalue size, . Below 1, repeated multiplication by eventually shrinks every vector toward zero; above 1, almost every vector eventually blows up.
- Stochastic gradient descent · also SGD, mini-batch gradient descent
- Taught in Derivatives, gradients and Jacobians
Gradient descent that estimates the gradient from a mini-batch, a small random subset of the training data, at each step. Each step is noisier than with the full gradient but far cheaper.
- Stride and padding · also stride, padding
- Taught in From LSTMs to today's models
The stride is how far a convolution's kernel moves between positions; padding adds rings of zeros around the border. Along an axis of length , a kernel of size fits at positions.
- Supervised fine-tuning · also SFT, instruction tuning
- Taught in From LSTMs to today's models
Further training of a pretrained model on curated demonstrations, each a prompt with an ideal response, using the same next-token cross-entropy loss. The loss is often counted only on the response tokens, which is called loss masking.
- Symmetric matrix
- Taught in Eigenvectors and eigenvalues
A square matrix equal to its transpose, . Its eigenvalues are real and its eigenvectors can be chosen perpendicular to each other. Covariance matrices and Hessians are symmetric.
- Tanh · also hyperbolic tangent
- Taught in Derivatives, gradients and Jacobians
The hyperbolic tangent, which squashes numbers into the interval from to and is centered at zero, with derivative . It is the nonlinearity of plain RNNs and of the LSTM candidate.
- Teacher forcing
- Taught in Recurrent networks
Training a sequence model by feeding it the true previous token at every step rather than its own prediction. The targets are the inputs shifted by one position.
- Temperature
- Taught in Neural networks from scratch
A number that divides the logits before softmax. Below 1 it sharpens the distribution toward the top choice; above 1 it flattens it, so sampling becomes more varied. Module 7 writes it , because there is the sequence length.
- Token
- Taught in The map
The unit of text a model reads and writes: a common word, a piece of a rarer word, a punctuation mark or a space, depending on the tokenizer. The model sees each token as an integer id.
- Tokenizer · also tokenization
- Taught in The map
The fixed procedure that splits text into tokens from a vocabulary and maps them to integer ids and back. Most current tokenizers are subword schemes such as byte-pair encoding (BPE), built by repeatedly merging the most frequent adjacent pair of symbols into a new token.
- Top-k and top-p sampling · also top-k, top-p, nucleus sampling
- Taught in Recurrent networks
Two ways of cutting off the unlikely tail before sampling the next token: top-k keeps the most probable tokens, top-p keeps the smallest set of most probable tokens whose probabilities add up to at least . The kept probabilities are renormalized to sum to 1.
- Trace
- Taught in Eigenvectors and eigenvalues
The sum of the diagonal entries of a square matrix, . It equals the sum of the eigenvalues, which makes it a quick check on eigenvalues found by hand.
- Training compute estimate · also 6ND
- Taught in From LSTMs to today's models
The rule of thumb that training a dense model with parameters on tokens takes about operations: about 2 per parameter per token for the forward pass and 4 for the backward pass.
- Transformer
- Taught in From LSTMs to today's models
The architecture introduced in 2017 in "Attention Is All You Need" that replaced recurrence with attention. Each block combines self-attention and an MLP with residual connections and normalization, and stacks of these blocks underlie today's large language models.
- Transpose
- Taught in Matrices are machines
swaps rows and columns, so entry becomes entry . It reverses products, , and writes the dot product as .
- U-Net
- Taught in From LSTMs to today's models
A fully convolutional segmentation network (2015), designed for biomedical images: a path that shrinks the image into features, a path that expands back to full resolution, and skip connections between matching levels.
- Unit vector
- Taught in Vectors and the dot product
A vector of length 1. Dividing a nonzero vector by its length, , gives the unit vector in its direction; this is called normalizing.
- Universal approximation theorem
- Taught in Neural networks from scratch
A network with one hidden layer and enough units can approximate any continuous function on a bounded region as closely as you like, for suitable activations such as sigmoid or ReLU. It says such a network exists, not that training will find it or that it is small.
- Validation set · also held-out set
- Taught in Neural networks from scratch
Examples held out from training to measure how well the model generalizes and to choose settings such as the learning rate.
- Vanishing gradient
- Taught in Recurrent networks
A gradient that shrinks geometrically as it is passed back through many steps or layers, because each step multiplies it by factors smaller than 1. It is the main reason plain RNNs struggle to learn dependencies more than a short distance back.
- Variance · also standard deviation
- Taught in The map
The average squared distance from the mean : . Its square root is the standard deviation. This course, like
np.var, divides by rather than ; the covariance matrix of module 4 followsnp.covand divides by . - Vector
- Taught in Vectors and the dot product
An ordered list of numbers that can also be drawn as an arrow from the origin. Math in this course writes vectors in bold, , and treats them as columns.
- Vector-Jacobian product
- Taught in Derivatives, gradients and Jacobians
A gradient vector multiplied by a Jacobian, in column form. Backpropagation passes gradients backwards as a chain of these products and never builds full Jacobians.
- Vision transformer · also ViT
- Taught in From LSTMs to today's models
A transformer applied to images by cutting each image into patches and treating every patch as a token.
- Vocabulary
- Taught in Recurrent networks
The fixed set of tokens (or characters) a model can read and produce. Its size is the number of outputs of the model's final softmax.
- Weight decay
- Taught in Neural networks from scratch
Shrinking every weight by a small fraction of itself at each step, a pull toward zero that discourages overfitting. With plain gradient descent it amounts to adding a squared-L2 penalty to the loss; AdamW applies it separately from Adam's per-parameter scaling.
- Weight sharing · also tied weights
- Taught in Recurrent networks
Using the same weights at many positions: across time in an RNN, across space in a CNN. One learned pattern then applies everywhere, and the parameter count does not grow with the input's length or size.
- Zero-shot classification
- Taught in From LSTMs to today's models
Classifying into classes a model was never trained on, by describing each class in text. With CLIP, each image gets the class whose caption embedding has the highest cosine similarity with the image's embedding, so changing the classes means changing a list of captions.