Backpropagation computes the gradient of the loss with respect to every parameter in one backward sweep, by passing a single gradient vector from the loss back through the network and multiplying it by each step's local derivatives. This lesson derives it for real layers with batches and builds the layers yourself.
About 85 minutes
By the end you can
Run a forward and a backward pass through a computational graph by hand, adding the gradients where a value is used twice.
Derive ∂L/∂W=X⊤δ, ∂L/∂X=δW⊤ and ∂L/∂b=∑nδn for a linear layer with a batch, entry by entry and then as matrices.
Backpropagate through elementwise nonlinearities and write the full backward pass of a two-layer network.
Gradient-check any backward pass and interpret the relative error.
Implement Linear, Tanh and ReLU layers with forward and backward methods.
The small network you will train in the training lesson has 387 parameters (it is a 2-64-3 network, so (2⋅64+64)+(64⋅3+3)=192+195=387). GPT-2 small has about 124 million. Training needs the derivative of the loss with respect to every one of them, at every step.
The obvious way to get a derivative is to measure it: nudge one parameter, run the network again, and see how much the loss moved. That costs two forward passes per parameter, so 248 million forward passes for one step of GPT-2. Backpropagation gets all 124 million derivatives exactly, from one forward pass and one backward pass that costs about twice as much. That difference, a factor of tens of millions here, is the reason neural networks can be trained at all.
You have already met the idea twice. Matrix multiplication showed that composing linear maps multiplies their matrices. Jacobians and the chain rule showed that the same holds for derivatives: a network is a composition of layers, so its derivative is a product of the layers' Jacobians. Its last section showed why backprop evaluates that product from the loss end. The loss is a single number, so the product starts as one row of derivatives, and every step multiplies a vector by a matrix instead of a matrix by a matrix: backprop passes one gradient vector backwards and multiplies it by each layer's local derivative, never building a Jacobian. This lesson makes it concrete. First on graphs small enough to do by hand, then for the layers real networks are made of, with batches, and finally with the test that tells you whether your backward pass is right.
Write any computation as a computational graph: each node is one simple operation, and the edges carry values from one operation to the next. Take f=(x+y)z, with an intermediate node q=x+y, so that f=qz.
The forward pass computes and stores every node's value. With x=2, y=1, z=−3: q=3 and f=−9.
The backward pass starts at the output with ∂f∂f=1 and works back, one node at a time. Each node knows its local derivatives, the derivatives of its output with respect to its own inputs, and it multiplies them by the gradient arriving from above, the upstream gradient:
At f=qz: the local derivatives are ∂q∂f=z=−3 and ∂z∂f=q=3. Times the upstream gradient 1, those are the gradients of q and z.
At q=x+y: the local derivatives are both 1, so x and y each receive the upstream gradient −3 unchanged: ∂x∂f=∂y∂f=−3.
Check one: raising x by 0.01 raises q by 0.01 and changes f by 0.01×(−3)=−0.03, which is −3 times the nudge. Every step was the chain rule, ∂x∂f=∂q∂f∂x∂q, done in the order that lets each node reuse the gradient already computed above it.
The widget below runs exactly this computation, one step at a time. The graph flows left to right: inputs x, y and z, the sum node q, the product node f. White numbers are values from the forward pass; amber numbers are gradients from the backward pass, each written on the wire it travels along. The arrow buttons (or the arrow keys, once you click the graph) move one step; the panel writes out the current step's formula with its numbers, and a live note says what the step does. The readouts give each input's final gradient, with a "nudge check" underneath: the same derivative measured by nudging that input.
A few node types cover most of what you will see, and each has a recognizable effect on the gradient:
node
forward
what it sends back to each input
add, q=a+b
sum
the upstream gradient, unchanged, to both
multiply, q=ab
product
upstream times the other input
tanh, q=tanh(a)
squash
upstream times 1−q2
σ, q=σ(a)
squash
upstream times q(1−q)
ReLU, q=max(0,a)
clip
upstream if a>0, else 0
The multiply row needs the inputs, and the last three rows need the output (or, for ReLU, whether it was positive): this is why the forward pass must store values. The backward pass reads them.
Now a neuron, a=σ(w1x1+w2x2+b), with w1=1.5, x1=2, w2=−1, x2=1 and b=−1. Name the two products u1=w1x1 and u2=w2x2, and the sum s=u1+u2+b.
Forward.u1=(1.5)(2)=3, u2=(−1)(1)=−1, s=3+(−1)+(−1)=1 and a=σ(1)≈0.7311.
Backward through the sigmoid. Its local derivative is a(1−a)≈0.7311×0.2689≈0.1966.
Through the sum. It passes 0.1966 unchanged to each of its three inputs, u1, u2 and b.
Through the products. Each multiply node turns 0.1966 into the other factor times 0.1966:
The gradient with respect to a weight is the input it multiplies, times the signal from above. The gradients lesson found the same thing for w⋅x: its gradient with respect to w is x.
The widget shows this neuron with the numbers from the text. It opens with the forward pass already done, so every step you take is part of the backward pass.
Now do a full forward and backward pass by hand, through a small network with a loss at the end.
On paperA full backward pass by hand
A tiny network with one input, one ReLU hidden unit and one output computes
z=w1x+b1,h=ReLU(z),y^=w2h+b2,L=21(y^−t)2,
with x=2, w1=1, b1=−1, w2=3, b2=−1 and target t=4.
Run the forward pass: z, h, y^ and L.
Run the backward pass node by node and find ∂L/∂w1, ∂L/∂b1, ∂L/∂w2 and ∂L/∂b2.
Check ∂L/∂w1 with a central difference: compute L at w1=1.01 and at w1=0.99.
Now change b1 to −3 and redo the backward pass. Which gradients become zero, and why?
To check part 2 before reading the steps, enter the four gradients as a column in the order w1, b1, w2, b2.
Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.
∂L/∂w1, ∂L/∂b1, ∂L/∂w2, ∂L/∂b2
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
The general rule is short. Visit the nodes in reverse order of the forward pass. For each node u,
∂u∂L=v uses u∑∂v∂L∂u∂v,
a sum over every node v that takes u as an input. Reverse order guarantees that every ∂v∂L is finished before it is needed.
The sum is not optional. When a value feeds two places, it influences the loss along two paths, and the chain rule for many variables adds the paths. Take a=xy, s=x+y and L=as, with x=2 and y=3. Forward: a=6, s=5, L=30. Backward: ∂a∂L=s=5 and ∂s∂L=a=6. Now x feeds both a and s:
∂x∂L=5×3=15∂a∂L∂x∂a+6×1=6∂s∂L∂x∂s=21.
Check by expanding: L=as=xy(x+y)=x2y+xy2, so ∂x∂L=2xy+y2=12+9=21. Forgetting one path is the most common backprop bug. In code, a value used twice either has its gradients added explicitly, or accumulated with +=.
For vectors the rule is the one from module 5: a node that computes v=g(u) with Jacobian J turns the upstream gradient ∇vL into ∇uL=J⊤∇vL, a vector-Jacobian product, and each kind of layer computes its own without forming J.
The sum over uses matters most in a design every transformer relies on: the residual connection.
Quick checkThe gradient of a residual connection
Transformers and many deep networks use residual connections: a block computes y=h+F(h), adding its input to the output of a sublayer F. Backprop has delivered g=∂L/∂y. Writing JF for the Jacobian of F at h, what is ∂L/∂h?
Choose one answer, then check.
To recap: the backward pass visits the nodes in reverse, multiplies each upstream gradient by a local derivative, and adds the contributions wherever a value was used more than once.
Now the layer that holds almost all of a network's parameters. In the row convention of code, a batch of N inputs X (shape N×n) goes through
Y=XW+b,
with W of shape n×m, b with m entries added to every row, and Y of shape N×m. Backprop has already delivered δ=∂L/∂Y, the gradient of the scalar loss with respect to every entry of Y, so δ has shape N×m too. We want three gradients, each with the shape of what it is for: ∂L/∂W (n×m), ∂L/∂b (m entries) and ∂L/∂X (N×n), the last one to pass further back.
Work entry by entry, where everything is ordinary scalar calculus. Entry (r,j) of Y, output j of example number r, is
Yrj=k=1∑nXrkWkj+bj.
The weights. Which entries of Y does one weight Wkj touch? It appears in output j of every example, Y1j,…,YNj, each time multiplied by that example's input Xrk. So ∂Wkj∂Yrj=Xrk, and the paths add over the batch:
The bias.bj is added to output j of every example with local derivative 1, so ∂bj∂L=∑rδrj: the column sums of δ.
The inputs.Xrk appears in every output of its own example r, Yr1,…,Yrm, each time multiplied by Wkj. So
∂Xrk∂L=j=1∑mδrjWkj=(δW⊤)rk.Go slower: The weight gradient with every index written out
Step 1, the loss depends on Wkj only through Y. By the chain rule over every entry of Y:
∂Wkj∂L=∑r=1N∑j′=1m∂Yrj′∂L∂Wkj∂Yrj′.
Step 2, the local derivative.Yrj′=∑k′Xrk′Wk′j′+bj′. The weight Wkj appears in this sum only when j′=j and k′=k, with coefficient Xrk. So ∂Wkj∂Yrj′=Xrk if j′=j, and 0 otherwise.
Step 3, collapse the sum over j′. Only j′=j survives: ∂Wkj∂L=∑r=1NδrjXrk.
Step 4, recognize a matrix product.Xrk is entry (k,r) of X⊤. So the sum is ∑r(X⊤)krδrj, which is the row-times-column rule for entry (k,j) of X⊤δ.
The bias and the input, the same way.∂bj∂Yrj′ is 1 when j′=j and 0 otherwise, which leaves ∑rδrj. ∂Xrk∂Yr′j is Wkj when r′=r and 0 otherwise, which leaves ∑jδrjWkj=∑jδrj(W⊤)jk, entry (r,k) of δW⊤.
A worked example with N=2 examples, n=2 inputs and m=2 outputs:
X=(102−1),W=(12−10),δ=(123−1).
The weight gradient transposes X and multiplies row by column:
The same result as a sum over examples.Matrix multiplication showed that a product can be read as a sum of outer products. Read X⊤δ that way and it is ∑rxr⊤δr, where xr and δr are row r of X and of δ: one outer product per example, (its input, as a column) times (its error signal, as a row). In the example, (12)(1,3)+(0−1)(2,−1)=(1236)+(0−201)=(1037). A weight gets a large gradient when its input and the error signal at its output are large in the same examples. And a gradient from N examples is a sum of N rank-one matrices, so its rank is at most N (rank).
In the column convention of the math, with y=Wx+b and W of shape m×n, the same results read ∂W∂L=δx⊤ (an outer product) and ∂x∂L=W⊤δ, the vector-Jacobian product from module 5. Same numbers, transposed bookkeeping.
Now derive one weight gradient entry by entry yourself, and check it against the matrix formula.
On paperDerive a weight gradient entry by entry
A linear layer computes Y=XW+b on a batch of two examples, with
X=(132−1),W=(1−102),b=(0,1),
and the loss is L=21∑r,jYrj2, half the sum of the squares of all four outputs.
Compute Y, and show that δ=∂L/∂Y equals Y.
Derive ∂L/∂W21 entry by entry: which entries of Y does W21 affect, and with what local derivatives?
Compute the whole ∂L/∂W=X⊤δ and check its entry (2,1) against part 2.
Compute ∂L/∂b and ∂L/∂X.
To check part 3 before reading the steps, enter ∂L/∂W row by row.
Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.
∂L/∂W
One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
With the formula in hand, a batch of three is one matrix product.
Work it outA weight gradient from a batch of three
A linear layer Y=XW+b processed a batch of three examples with two inputs each, and backprop delivered δ=∂L/∂Y:
X=12−1013,δ=201−111.
Find ∂L/∂W and enter the 2×2 matrix row by row, as whole numbers.
∂L/∂W
One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
In code, the formula has to survive the row convention. Pick the right line.
Quick checkWhich line computes dW
A layer has X of shape (32, 100), W of shape (100, 10) and computes Y = X @ W + b. The loss is a mean over the batch, and backprop has delivered delta=∂L/∂Y of shape (32, 10), already including the 1/32 from the mean. Which line computes the weight gradient?
Choose one answer, then check.
To recap: a linear layer's backward pass is three matrix products, each the only one with the right shape. The weight gradient sums over the batch, one outer product per example.
An elementwise nonlinearity H=ϕ(Z) is the easy case. Each output entry depends only on the input entry in the same position, so the backward pass multiplies entry by entry by the slope:
∂Z∂L=∂H∂L⊙ϕ′(Z),
where ⊙ is the elementwise product. For tanh the slope is 1−H2, computed from the stored output. For the sigmoid it is H⊙(1−H). For ReLU it is a mask: 1 where Z>0 and 0 elsewhere, so the gradient passes through the units that were on and stops at the units that were off.
Put the pieces together for the network of the next lesson: a tanh hidden layer and a softmax cross-entropy output, trained on a batch of N examples with integer labels y. P holds the softmax probabilities, one row per example, and Y here holds the one-hot labels.
forward
backward
Z1=XW1+b1
∂L/∂W1=X⊤∂L/∂Z1, ∂L/∂b1= column sums of ∂L/∂Z1
Read the backward column from the bottom up: that is the order in which it runs. It starts from the previous lesson'sp−y in the bottom row, and every line above it is one of the two rules of this lesson. Nothing else is needed for a network of any depth, which is why code organizes it as layers, each with a forward that stores what it needs and a backward that applies its rule and hands the result to the layer below.
The diagram below runs both columns on the smallest network that has every piece: two inputs, two tanh hidden units and one output, on a single example. To keep the numbers few, its output has no activation and its loss is the squared error L=(y^−y)2, so the gradient it starts from is 2(y^−y) instead of p−y; every rule above it is the same. Each edge carries its weight, and once computed, its gradient in amber underneath. On wide screens the weights also appear below the network as the matrices W1 (2×2) and W2 (2×1), each weight above its gradient. Press Next to move through the ten steps.
The widget shows that product of slopes on a chain of two tanh units, h2=tanh(w2tanh(w1x)), starting at x=1 and w1=w2=2. Like the neuron above, it opens with the forward pass done.
A backward pass can be wrong in ways that still train a little, so a falling loss is no proof. The test that cannot share your mistakes is the definition of the derivative, which the derivatives lesson turned into the central difference:
∂θ∂L≈2hL(θ+h)−L(θ−h),
for one parameter θ at a time, with a small step h and every other parameter held fixed (code usually calls the step eps). Compare it with your analytic gradient using the relative error
rel(a,n)=max(∣a∣+∣n∣,tiny)∣a−n∣,
where a is the analytic value, n the numerical one, and "tiny" (such as 10−8) avoids dividing by zero when both are zero. A relative measure is needed because gradients range over many orders of magnitude: an absolute difference of 10−6 is excellent for a gradient of size 10 and useless for one of size 10−6.
How small should it be? In 64-bit floats with h=10−5, the central difference has a truncation error on the order of h2=10−10, plus rounding. So:
Below about 10−7: the gradient is right.
Between about 10−7 and 10−4: look closer before you trust it. Two innocent causes land here. A ReLU's kink at 0 makes the central difference wrong whenever a pre-activation lies within h of 0. And an entry whose true gradient is tiny compared with the loss (say 10−6 against a loss near 2) loses digits to rounding, because the two nudged losses agree in almost every digit. Check which entries are off: a real bug usually shows up across a whole parameter, not in one stray entry.
Above about 10−4: almost certainly a bug.
Practical rules: check in float64, on a small network and a small batch, a few random entries of every parameter (all entries when there are few), and with any randomness in the forward pass switched off. The code below plants two classic bugs in the backward pass of a two-layer network and shows what the check reports.
⌘+Enter runs · edit freelyPython sleeps until you run code
The correct backward pass agrees to about eight digits or better, everywhere. Forgetting the tanh slope corrupts both first-layer gradients but leaves W2 perfect, which points straight at the bug. The mean-instead-of-sum bug divides ∂L/∂b1 by N a second time (the 1/N is already inside δ), and only b1 fails. Notice that both buggy versions would still train somewhat: the first-layer gradients point in roughly sensible directions. Only the check catches them.
Now build the layers yourself. They are the ones the rest of Part 1 uses. The reference versions in mlref/nn.py have the same forward and backward, plus two small methods, parameters() and grads(), that list a layer's arrays and their gradients, so that the training loop of the next lesson can update them.
Code itLinear, Tanh and ReLU layers
Build the three layers the rest of Part 1 runs on, each with forward(X), which remembers what the backward pass needs, and backward(dY), which takes ∂L/∂Y and returns ∂L/∂X.
Linear(n_in, n_out, rng): Xavier initialization, W of shape (n_in, n_out) with standard deviation 2/(nin+nout) and b zero. backward sets self.dW and self.db (overwriting any old values) and returns dX.
Tanh and ReLU, elementwise.
The last lines chain your layers into a small network and gradient-check the first layer's dW. The reference versions in mlref/nn.py have the same forward and backward (plus parameters() and grads() for optimizers), and one test compares yours with them.
⌘+Enter runsPython sleeps until you run code
Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.
A few more backward passes, with fresh numbers each time.
Practice setBackprop reps
3 correct in a row completes the set. A miss starts the count again.
A linear layer computes Y=XW+b for a batch, one example per row. Backprop has delivered δ=∂L/∂Y. With W=−12212−2 and δ=021−12−1, find ∂L/∂X (the gradient passed back to the previous layer).
∂L/∂X
One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Finally, explain why one backward pass is enough, and what it costs.
In your own wordsExplain backprop's cost
Explain to a colleague who knows the chain rule how backpropagation computes the gradient of the loss with respect to millions of parameters in one backward pass, why that costs only a small multiple of a forward pass, and why training needs much more memory than running the model.