ml.lab
Python sleeps until you run code
06 Neural networks from scratch

Lesson 4 of 6

Backpropagation

Backpropagation computes the gradient of the loss with respect to every parameter in one backward sweep, by passing a single gradient vector from the loss back through the network and multiplying it by each step's local derivatives. This lesson derives it for real layers with batches and builds the layers yourself.

About 85 minutes
By the end you can
  • Run a forward and a backward pass through a computational graph by hand, adding the gradients where a value is used twice.

  • Derive ∂L/∂W=X⊤δ\partial L/\partial\mathbf{W} = \mathbf{X}^\top\boldsymbol{\delta}, ∂L/∂X=δW⊤\partial L/\partial\mathbf{X} = \boldsymbol{\delta}\mathbf{W}^\top and ∂L/∂b=∑nδn\partial L/\partial\mathbf{b} = \sum_n \boldsymbol{\delta}_n for a linear layer with a batch, entry by entry and then as matrices.

  • Backpropagate through elementwise nonlinearities and write the full backward pass of a two-layer network.

  • Gradient-check any backward pass and interpret the relative error.

  • Implement Linear, Tanh and ReLU layers with forward and backward methods.

The small network you will train in the training lesson has 387 parameters (it is a 2-64-3 network, so (2⋅64+64)+(64⋅3+3)=192+195=387(2 \cdot 64 + 64) + (64 \cdot 3 + 3) = 192 + 195 = 387). GPT-2 small has about 124 million. Training needs the derivative of the loss with respect to every one of them, at every step.

The obvious way to get a derivative is to measure it: nudge one parameter, run the network again, and see how much the loss moved. That costs two forward passes per parameter, so 248 million forward passes for one step of GPT-2. Backpropagation gets all 124 million derivatives exactly, from one forward pass and one backward pass that costs about twice as much. That difference, a factor of tens of millions here, is the reason neural networks can be trained at all.

You have already met the idea twice. Matrix multiplication showed that composing linear maps multiplies their matrices. Jacobians and the chain rule showed that the same holds for derivatives: a network is a composition of layers, so its derivative is a product of the layers' Jacobians. Its last section showed why backprop evaluates that product from the loss end. The loss is a single number, so the product starts as one row of derivatives, and every step multiplies a vector by a matrix instead of a matrix by a matrix: backprop passes one gradient vector backwards and multiplies it by each layer's local derivative, never building a Jacobian. This lesson makes it concrete. First on graphs small enough to do by hand, then for the layers real networks are made of, with batches, and finally with the test that tells you whether your backward pass is right.

Forward values, backward gradients

Write any computation as a computational graph: each node is one simple operation, and the edges carry values from one operation to the next. Take f=(x+y) zf = (x + y)\,z, with an intermediate node q=x+yq = x + y, so that f=q zf = q\,z.

The forward pass computes and stores every node's value. With x=2x = 2, y=1y = 1, z=−3z = -3: q=3q = 3 and f=−9f = -9.

The backward pass starts at the output with ∂f∂f=1\frac{\partial f}{\partial f} = 1 and works back, one node at a time. Each node knows its local derivatives, the derivatives of its output with respect to its own inputs, and it multiplies them by the gradient arriving from above, the upstream gradient:

  • At f=q zf = q\,z: the local derivatives are ∂f∂q=z=−3\frac{\partial f}{\partial q} = z = -3 and ∂f∂z=q=3\frac{\partial f}{\partial z} = q = 3. Times the upstream gradient 1, those are the gradients of qq and zz.
  • At q=x+yq = x + y: the local derivatives are both 1, so xx and yy each receive the upstream gradient −3-3 unchanged: ∂f∂x=∂f∂y=−3\frac{\partial f}{\partial x} = \frac{\partial f}{\partial y} = -3.

Check one: raising xx by 0.010.01 raises qq by 0.010.01 and changes ff by 0.01×(−3)=−0.030.01 \times (-3) = -0.03, which is −3-3 times the nudge. Every step was the chain rule, ∂f∂x=∂f∂q ∂q∂x\frac{\partial f}{\partial x} = \frac{\partial f}{\partial q}\,\frac{\partial q}{\partial x}, done in the order that lets each node reuse the gradient already computed above it.

The widget below runs exactly this computation, one step at a time. The graph flows left to right: inputs xx, yy and zz, the sum node qq, the product node ff. White numbers are values from the forward pass; amber numbers are gradients from the backward pass, each written on the wire it travels along. The arrow buttons (or the arrow keys, once you click the graph) move one step; the panel writes out the current step's formula with its numbers, and a live note says what the step does. The readouts give each input's final gradient, with a "nudge check" underneath: the same derivative measured by nudging that input.

A few node types cover most of what you will see, and each has a recognizable effect on the gradient:

nodeforwardwhat it sends back to each input
add, q=a+bq = a + bsumthe upstream gradient, unchanged, to both
multiply, q=a bq = a\,bproductupstream times the other input
tanh⁡\tanh, q=tanh⁡(a)q = \tanh(a)squashupstream times 1−q21 - q^2
σ\sigma, q=σ(a)q = \sigma(a)squashupstream times q(1−q)q(1 - q)
ReLU, q=max⁡(0,a)q = \max(0,\allowbreak a)clipupstream if a>0a > 0, else 0

The multiply row needs the inputs, and the last three rows need the output (or, for ReLU, whether it was positive): this is why the forward pass must store values. The backward pass reads them.

Now a neuron, a=σ(w1x1+w2x2+b)a = \sigma(w_1 x_1 + w_2 x_2 + b), with w1=1.5w_1 = 1.5, x1=2x_1 = 2, w2=−1w_2 = -1, x2=1x_2 = 1 and b=−1b = -1. Name the two products u1=w1x1u_1 = w_1x_1 and u2=w2x2u_2 = w_2x_2, and the sum s=u1+u2+bs = u_1 + u_2 + b.

  • Forward. u1=(1.5)(2)=3u_1 = (1.5)(2) = 3, u2=(−1)(1)=−1u_2 = (-1)(1) = -1, s=3+(−1)+(−1)=1s = 3 + (-1) + (-1) = 1 and a=σ(1)≈0.7311a = \sigma(1) \approx 0.7311.
  • Backward through the sigmoid. Its local derivative is a(1−a)≈0.7311×0.2689≈0.1966a(1 - a) \approx 0.7311 \times 0.2689 \approx 0.1966.
  • Through the sum. It passes 0.19660.1966 unchanged to each of its three inputs, u1u_1, u2u_2 and bb.
  • Through the products. Each multiply node turns 0.19660.1966 into the other factor times 0.19660.1966:
∂a∂w1≈0.1966×2≈0.393,∂a∂x1≈0.1966×1.5≈0.295,∂a∂b≈0.197.\frac{\partial a}{\partial w_1} \approx 0.1966 \times 2 \approx 0.393, \qquad \frac{\partial a}{\partial x_1} \approx 0.1966 \times 1.5 \approx 0.295, \qquad \frac{\partial a}{\partial b} \approx 0.197 .

The gradient with respect to a weight is the input it multiplies, times the signal from above. The gradients lesson found the same thing for w⋅x\mathbf{w}\cdot\mathbf{x}: its gradient with respect to w\mathbf{w} is x\mathbf{x}.

The widget shows this neuron with the numbers from the text. It opens with the forward pass already done, so every step you take is part of the backward pass.

Now do a full forward and backward pass by hand, through a small network with a loss at the end.

On paperA full backward pass by hand

A tiny network with one input, one ReLU hidden unit and one output computes

z=w1x+b1,h=ReLU⁡(z),y^=w2h+b2,L=12(y^−t)2,z = w_1 x + b_1, \qquad h = \operatorname{ReLU}(z), \qquad \hat{y} = w_2 h + b_2, \qquad L = \tfrac{1}{2}(\hat{y} - t)^2,

with x=2x = 2, w1=1w_1 = 1, b1=−1b_1 = -1, w2=3w_2 = 3, b2=−1b_2 = -1 and target t=4t = 4.

  1. Run the forward pass: zz, hh, y^\hat{y} and LL.
  2. Run the backward pass node by node and find ∂L/∂w1\partial L/\partial w_1, ∂L/∂b1\partial L/\partial b_1, ∂L/∂w2\partial L/\partial w_2 and ∂L/∂b2\partial L/\partial b_2.
  3. Check ∂L/∂w1\partial L/\partial w_1 with a central difference: compute LL at w1=1.01w_1 = 1.01 and at w1=0.99w_1 = 0.99.
  4. Now change b1b_1 to −3-3 and redo the backward pass. Which gradients become zero, and why?

To check part 2 before reading the steps, enter the four gradients as a column in the order w1w_1, b1b_1, w2w_2, b2b_2.

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

∂L/∂w1, ∂L/∂b1, ∂L/∂w2, ∂L/∂b2

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

One backward pass for every gradient

The general rule is short. Visit the nodes in reverse order of the forward pass. For each node uu,

∂L∂u=∑v uses u∂L∂v ∂v∂u,\frac{\partial L}{\partial u} = \sum_{v \text{ uses } u} \frac{\partial L}{\partial v}\,\frac{\partial v}{\partial u},

a sum over every node vv that takes uu as an input. Reverse order guarantees that every ∂L∂v\frac{\partial L}{\partial v} is finished before it is needed.

The sum is not optional. When a value feeds two places, it influences the loss along two paths, and the chain rule for many variables adds the paths. Take a=xya = xy, s=x+ys = x + y and L=a sL = a\,s, with x=2x = 2 and y=3y = 3. Forward: a=6a = 6, s=5s = 5, L=30L = 30. Backward: ∂L∂a=s=5\frac{\partial L}{\partial a} = s = 5 and ∂L∂s=a=6\frac{\partial L}{\partial s} = a = 6. Now xx feeds both aa and ss:

∂L∂x=∂L∂a ∂a∂x⏟5×3=15+∂L∂s ∂s∂x⏟6×1=6=21.\frac{\partial L}{\partial x} = \underbrace{\frac{\partial L}{\partial a}\,\frac{\partial a}{\partial x}}_{5 \times 3 = 15} + \underbrace{\frac{\partial L}{\partial s}\,\frac{\partial s}{\partial x}}_{6 \times 1 = 6} = 21 .

Check by expanding: L=a s=xy(x+y)=x2y+xy2L = a\,s = xy(x + y) = x^2 y + x y^2, so ∂L∂x=2xy+y2=12+9=21\frac{\partial L}{\partial x} = 2xy + y^2 = 12 + 9 = 21. Forgetting one path is the most common backprop bug. In code, a value used twice either has its gradients added explicitly, or accumulated with +=.

For vectors the rule is the one from module 5: a node that computes v=g(u)\mathbf{v} = g(\mathbf{u}) with Jacobian J\mathbf{J} turns the upstream gradient ∇vL\nabla_{\mathbf{v}}L into ∇uL=J⊤∇vL\nabla_{\mathbf{u}}L = \mathbf{J}^\top\nabla_{\mathbf{v}}L, a vector-Jacobian product, and each kind of layer computes its own without forming J\mathbf{J}.

The sum over uses matters most in a design every transformer relies on: the residual connection.

Quick checkThe gradient of a residual connection

Transformers and many deep networks use residual connections: a block computes y=h+F(h)\mathbf{y} = \mathbf{h} + F(\mathbf{h}), adding its input to the output of a sublayer FF. Backprop has delivered g=∂L/∂y\mathbf{g} = \partial L/\partial\mathbf{y}. Writing JF\mathbf{J}_F for the Jacobian of FF at h\mathbf{h}, what is ∂L/∂h\partial L/\partial\mathbf{h}?

Choose one answer, then check.

To recap: the backward pass visits the nodes in reverse, multiplies each upstream gradient by a local derivative, and adds the contributions wherever a value was used more than once.

The linear layer, entry by entry

Now the layer that holds almost all of a network's parameters. In the row convention of code, a batch of NN inputs X\mathbf{X} (shape N×nN \times n) goes through

Y=XW+b,\mathbf{Y} = \mathbf{X}\mathbf{W} + \mathbf{b},

with W\mathbf{W} of shape n×mn \times m, b\mathbf{b} with mm entries added to every row, and Y\mathbf{Y} of shape N×mN \times m. Backprop has already delivered δ=∂L/∂Y\boldsymbol{\delta} = \partial L/\partial\mathbf{Y}, the gradient of the scalar loss with respect to every entry of Y\mathbf{Y}, so δ\boldsymbol{\delta} has shape N×mN \times m too. We want three gradients, each with the shape of what it is for: ∂L/∂W\partial L/\partial\mathbf{W} (n×mn \times m), ∂L/∂b\partial L/\partial\mathbf{b} (mm entries) and ∂L/∂X\partial L/\partial\mathbf{X} (N×nN \times n), the last one to pass further back.

Work entry by entry, where everything is ordinary scalar calculus. Entry (r,j)(r,\allowbreak j) of Y\mathbf{Y}, output jj of example number rr, is

Yrj=∑k=1nXrk Wkj+bj.Y_{rj} = \sum_{k=1}^{n} X_{rk}\,W_{kj} + b_j .

The weights. Which entries of Y\mathbf{Y} does one weight WkjW_{kj} touch? It appears in output jj of every example, Y1j,…,YNjY_{1j},\allowbreak \ldots,\allowbreak Y_{Nj}, each time multiplied by that example's input XrkX_{rk}. So ∂Yrj∂Wkj=Xrk\frac{\partial Y_{rj}}{\partial W_{kj}} = X_{rk}, and the paths add over the batch:

∂L∂Wkj=∑r=1Nδrj Xrk=∑r=1N(X⊤)kr δrj=(X⊤δ)kj.\frac{\partial L}{\partial W_{kj}} = \sum_{r=1}^{N} \delta_{rj}\,X_{rk} = \sum_{r=1}^{N} (\mathbf{X}^\top)_{kr}\,\delta_{rj} = (\mathbf{X}^\top\boldsymbol{\delta})_{kj} .

The bias. bjb_j is added to output jj of every example with local derivative 1, so ∂L∂bj=∑rδrj\frac{\partial L}{\partial b_j} = \sum_{r} \delta_{rj}: the column sums of δ\boldsymbol{\delta}.

The inputs. XrkX_{rk} appears in every output of its own example rr, Yr1,…,YrmY_{r1},\allowbreak \ldots,\allowbreak Y_{rm}, each time multiplied by WkjW_{kj}. So

∂L∂Xrk=∑j=1mδrj Wkj=(δW⊤)rk.\frac{\partial L}{\partial X_{rk}} = \sum_{j=1}^{m} \delta_{rj}\,W_{kj} = (\boldsymbol{\delta}\mathbf{W}^\top)_{rk} .
Go slower: The weight gradient with every index written out

Step 1, the loss depends on WkjW_{kj} only through Y\mathbf{Y}. By the chain rule over every entry of Y\mathbf{Y}: ∂L∂Wkj=∑r=1N∑j′=1m∂L∂Yrj′ ∂Yrj′∂Wkj.\frac{\partial L}{\partial W_{kj}} = \sum_{r=1}^{N}\sum_{j'=1}^{m} \frac{\partial L}{\partial Y_{rj'}}\,\frac{\partial Y_{rj'}}{\partial W_{kj}} .

Step 2, the local derivative. Yrj′=∑k′Xrk′Wk′j′+bj′Y_{rj'} = \sum_{k'} X_{rk'}W_{k'j'} + b_{j'}. The weight WkjW_{kj} appears in this sum only when j′=jj' = j and k′=kk' = k, with coefficient XrkX_{rk}. So ∂Yrj′∂Wkj=Xrk\frac{\partial Y_{rj'}}{\partial W_{kj}} = X_{rk} if j′=jj' = j, and 0 otherwise.

Step 3, collapse the sum over j′j'. Only j′=jj' = j survives: ∂L∂Wkj=∑r=1Nδrj Xrk\frac{\partial L}{\partial W_{kj}} = \sum_{r=1}^{N} \delta_{rj}\,X_{rk}.

Step 4, recognize a matrix product. XrkX_{rk} is entry (k,r)(k,\allowbreak r) of X⊤\mathbf{X}^\top. So the sum is ∑r(X⊤)kr δrj\sum_r (\mathbf{X}^\top)_{kr}\,\delta_{rj}, which is the row-times-column rule for entry (k,j)(k,\allowbreak j) of X⊤δ\mathbf{X}^\top\boldsymbol{\delta}.

The bias and the input, the same way. ∂Yrj′∂bj\frac{\partial Y_{rj'}}{\partial b_j} is 1 when j′=jj' = j and 0 otherwise, which leaves ∑rδrj\sum_r \delta_{rj}. ∂Yr′j∂Xrk\frac{\partial Y_{r'j}}{\partial X_{rk}} is WkjW_{kj} when r′=rr' = r and 0 otherwise, which leaves ∑jδrjWkj=∑jδrj(W⊤)jk\sum_j \delta_{rj}W_{kj} = \sum_j \delta_{rj}(\mathbf{W}^\top)_{jk}, entry (r,k)(r,\allowbreak k) of δW⊤\boldsymbol{\delta}\mathbf{W}^\top.

A worked example with N=2N = 2 examples, n=2n = 2 inputs and m=2m = 2 outputs:

X=(120−1),W=(1−120),δ=(132−1).\mathbf{X} = \begin{pmatrix} 1 & 2 \\ 0 & -1 \end{pmatrix}, \quad \mathbf{W} = \begin{pmatrix} 1 & -1 \\ 2 & 0 \end{pmatrix}, \quad \boldsymbol{\delta} = \begin{pmatrix} 1 & 3 \\ 2 & -1 \end{pmatrix}.

The weight gradient transposes X\mathbf{X} and multiplies row by column:

∂L∂W=X⊤δ=(102−1)(132−1)=((1)(1)+(0)(2)(1)(3)+(0)(−1)(2)(1)+(−1)(2)(2)(3)+(−1)(−1))=(1307).\frac{\partial L}{\partial\mathbf{W}} = \mathbf{X}^\top\boldsymbol{\delta} = \begin{pmatrix} 1 & 0 \\ 2 & -1 \end{pmatrix}\begin{pmatrix} 1 & 3 \\ 2 & -1 \end{pmatrix} = \begin{pmatrix} (1)(1) + (0)(2) & (1)(3) + (0)(-1) \\ (2)(1) + (-1)(2) & (2)(3) + (-1)(-1) \end{pmatrix} = \begin{pmatrix} 1 & 3 \\ 0 & 7 \end{pmatrix}.

The bias gradient adds each column of δ\boldsymbol{\delta}: ∂L∂b=(1+2, 3−1)=(3,2)\frac{\partial L}{\partial\mathbf{b}} = (1 + 2,\allowbreak \ 3 - 1) = (3,\allowbreak 2). The input gradient multiplies by W⊤\mathbf{W}^\top:

∂L∂X=δW⊤=(132−1)(12−10)=((1)(1)+(3)(−1)(1)(2)+(3)(0)(2)(1)+(−1)(−1)(2)(2)+(−1)(0))=(−2234).\frac{\partial L}{\partial\mathbf{X}} = \boldsymbol{\delta}\mathbf{W}^\top = \begin{pmatrix} 1 & 3 \\ 2 & -1 \end{pmatrix}\begin{pmatrix} 1 & 2 \\ -1 & 0 \end{pmatrix} = \begin{pmatrix} (1)(1) + (3)(-1) & (1)(2) + (3)(0) \\ (2)(1) + (-1)(-1) & (2)(2) + (-1)(0) \end{pmatrix} = \begin{pmatrix} -2 & 2 \\ 3 & 4 \end{pmatrix}.

The same result as a sum over examples. Matrix multiplication showed that a product can be read as a sum of outer products. Read X⊤δ\mathbf{X}^\top\boldsymbol{\delta} that way and it is ∑rxr⊤δr\sum_r \mathbf{x}_r^\top\boldsymbol{\delta}_r, where xr\mathbf{x}_r and δr\boldsymbol{\delta}_r are row rr of X\mathbf{X} and of δ\boldsymbol{\delta}: one outer product per example, (its input, as a column) times (its error signal, as a row). In the example, (12)(1,3)+(0−1)(2,−1)=(1326)+(00−21)=(1307)\begin{pmatrix} 1 \\ 2 \end{pmatrix}(1,\allowbreak 3) + \begin{pmatrix} 0 \\ -1 \end{pmatrix}(2,\allowbreak -1) = \begin{pmatrix} 1 & 3 \\ 2 & 6 \end{pmatrix} + \begin{pmatrix} 0 & 0 \\ -2 & 1 \end{pmatrix} = \begin{pmatrix} 1 & 3 \\ 0 & 7 \end{pmatrix}. A weight gets a large gradient when its input and the error signal at its output are large in the same examples. And a gradient from NN examples is a sum of NN rank-one matrices, so its rank is at most NN (rank).

In the column convention of the math, with y=Wx+b\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{b} and W\mathbf{W} of shape m×nm \times n, the same results read ∂L∂W=δ x⊤\frac{\partial L}{\partial\mathbf{W}} = \boldsymbol{\delta}\,\mathbf{x}^\top (an outer product) and ∂L∂x=W⊤δ\frac{\partial L}{\partial\mathbf{x}} = \mathbf{W}^\top\boldsymbol{\delta}, the vector-Jacobian product from module 5. Same numbers, transposed bookkeeping.

Now derive one weight gradient entry by entry yourself, and check it against the matrix formula.

On paperDerive a weight gradient entry by entry

A linear layer computes Y=XW+b\mathbf{Y} = \mathbf{X}\mathbf{W} + \mathbf{b} on a batch of two examples, with

X=(123−1),W=(10−12),b=(0,1),\mathbf{X} = \begin{pmatrix} 1 & 2 \\ 3 & -1 \end{pmatrix}, \qquad \mathbf{W} = \begin{pmatrix} 1 & 0 \\ -1 & 2 \end{pmatrix}, \qquad \mathbf{b} = (0, 1),

and the loss is L=12∑r,jYrj2L = \frac{1}{2}\sum_{r,j} Y_{rj}^2, half the sum of the squares of all four outputs.

  1. Compute Y\mathbf{Y}, and show that δ=∂L/∂Y\boldsymbol{\delta} = \partial L/\partial\mathbf{Y} equals Y\mathbf{Y}.
  2. Derive ∂L/∂W21\partial L/\partial W_{21} entry by entry: which entries of Y\mathbf{Y} does W21W_{21} affect, and with what local derivatives?
  3. Compute the whole ∂L/∂W=X⊤δ\partial L/\partial\mathbf{W} = \mathbf{X}^\top\boldsymbol{\delta} and check its entry (2,1)(2,\allowbreak 1) against part 2.
  4. Compute ∂L/∂b\partial L/\partial\mathbf{b} and ∂L/∂X\partial L/\partial\mathbf{X}.

To check part 3 before reading the steps, enter ∂L/∂W\partial L/\partial\mathbf{W} row by row.

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

∂L/∂W

One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

With the formula in hand, a batch of three is one matrix product.

Work it outA weight gradient from a batch of three

A linear layer Y=XW+b\mathbf{Y} = \mathbf{X}\mathbf{W} + \mathbf{b} processed a batch of three examples with two inputs each, and backprop delivered δ=∂L/∂Y\boldsymbol{\delta} = \partial L/\partial\mathbf{Y}:

X=(1021−13),δ=(2−10111).\mathbf{X} = \begin{pmatrix} 1 & 0 \\ 2 & 1 \\ -1 & 3 \end{pmatrix}, \qquad \boldsymbol{\delta} = \begin{pmatrix} 2 & -1 \\ 0 & 1 \\ 1 & 1 \end{pmatrix}.

Find ∂L/∂W\partial L/\partial\mathbf{W} and enter the 2×22 \times 2 matrix row by row, as whole numbers.

∂L/∂W

One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

In code, the formula has to survive the row convention. Pick the right line.

Quick checkWhich line computes dW

A layer has X of shape (32, 100), W of shape (100, 10) and computes Y = X @ W + b. The loss is a mean over the batch, and backprop has delivered delta =∂L/∂Y= \partial L/\partial\mathbf{Y} of shape (32, 10), already including the 1/321/32 from the mean. Which line computes the weight gradient?

Choose one answer, then check.

To recap: a linear layer's backward pass is three matrix products, each the only one with the right shape. The weight gradient sums over the batch, one outer product per example.

Nonlinearities and a whole network

An elementwise nonlinearity H=ϕ(Z)\mathbf{H} = \phi(\mathbf{Z}) is the easy case. Each output entry depends only on the input entry in the same position, so the backward pass multiplies entry by entry by the slope:

∂L∂Z=∂L∂H⊙ϕ′(Z),\frac{\partial L}{\partial\mathbf{Z}} = \frac{\partial L}{\partial\mathbf{H}} \odot \phi'(\mathbf{Z}),

where ⊙\odot is the elementwise product. For tanh the slope is 1−H21 - \mathbf{H}^2, computed from the stored output. For the sigmoid it is H⊙(1−H)\mathbf{H}\odot(1 - \mathbf{H}). For ReLU it is a mask: 1 where Z>0\mathbf{Z} > 0 and 0 elsewhere, so the gradient passes through the units that were on and stops at the units that were off.

Put the pieces together for the network of the next lesson: a tanh hidden layer and a softmax cross-entropy output, trained on a batch of NN examples with integer labels y\mathbf{y}. P\mathbf{P} holds the softmax probabilities, one row per example, and Y\mathbf{Y} here holds the one-hot labels.

forwardbackward
Z1=XW1+b1\mathbf{Z}_1 = \mathbf{X}\mathbf{W}_1 + \mathbf{b}_1∂L/∂W1=X⊤ ∂L/∂Z1\partial L/\partial\mathbf{W}_1 = \mathbf{X}^\top\,\partial L/\partial\mathbf{Z}_1,  ∂L/∂b1=\ \partial L/\partial\mathbf{b}_1 = column sums of ∂L/∂Z1\partial L/\partial\mathbf{Z}_1
H=tanh⁡(Z1)\mathbf{H} = \tanh(\mathbf{Z}_1)∂L/∂Z1=∂L/∂H⊙(1−H2)\partial L/\partial\mathbf{Z}_1 = \partial L/\partial\mathbf{H} \odot (1 - \mathbf{H}^2)
Z2=HW2+b2\mathbf{Z}_2 = \mathbf{H}\mathbf{W}_2 + \mathbf{b}_2∂L/∂W2=H⊤ ∂L/∂Z2\partial L/\partial\mathbf{W}_2 = \mathbf{H}^\top\,\partial L/\partial\mathbf{Z}_2,  ∂L/∂b2=\ \partial L/\partial\mathbf{b}_2 = column sums,  ∂L/∂H=∂L/∂Z2 W2⊤\ \partial L/\partial\mathbf{H} = \partial L/\partial\mathbf{Z}_2\,\mathbf{W}_2^\top
L=L = mean cross-entropy of softmax⁡(Z2)\operatorname{softmax}(\mathbf{Z}_2)∂L/∂Z2=(P−Y)/N\partial L/\partial\mathbf{Z}_2 = (\mathbf{P} - \mathbf{Y})/N

Read the backward column from the bottom up: that is the order in which it runs. It starts from the previous lesson's p−y\mathbf{p} - \mathbf{y} in the bottom row, and every line above it is one of the two rules of this lesson. Nothing else is needed for a network of any depth, which is why code organizes it as layers, each with a forward that stores what it needs and a backward that applies its rule and hands the result to the layer below.

The diagram below runs both columns on the smallest network that has every piece: two inputs, two tanh hidden units and one output, on a single example. To keep the numbers few, its output has no activation and its loss is the squared error L=(y^−y)2L = (\hat{y} - y)^2, so the gradient it starts from is 2(y^−y)2(\hat{y} - y) instead of p−y\mathbf{p} - \mathbf{y}; every rule above it is the same. Each edge carries its weight, and once computed, its gradient in amber underneath. On wide screens the weights also appear below the network as the matrices W1\mathbf{W}_1 (2×22 \times 2) and W2\mathbf{W}_2 (2×12 \times 1), each weight above its gradient. Press Next to move through the ten steps.

The widget shows that product of slopes on a chain of two tanh units, h2=tanh⁡(w2tanh⁡(w1x))h_2 = \tanh(w_2\tanh(w_1x)), starting at x=1x = 1 and w1=w2=2w_1 = w_2 = 2. Like the neuron above, it opens with the forward pass done.

Gradient checking: trust, but verify

A backward pass can be wrong in ways that still train a little, so a falling loss is no proof. The test that cannot share your mistakes is the definition of the derivative, which the derivatives lesson turned into the central difference:

∂L∂θ≈L(θ+h)−L(θ−h)2h,\frac{\partial L}{\partial \theta} \approx \frac{L(\theta + h) - L(\theta - h)}{2h},

for one parameter θ\theta at a time, with a small step hh and every other parameter held fixed (code usually calls the step eps). Compare it with your analytic gradient using the relative error

rel(a,n)=∣a−n∣max⁡(∣a∣+∣n∣, tiny),\text{rel}(a, n) = \frac{|a - n|}{\max(|a| + |n|,\ \text{tiny})},

where aa is the analytic value, nn the numerical one, and "tiny" (such as 10−810^{-8}) avoids dividing by zero when both are zero. A relative measure is needed because gradients range over many orders of magnitude: an absolute difference of 10−610^{-6} is excellent for a gradient of size 10 and useless for one of size 10−610^{-6}.

How small should it be? In 64-bit floats with h=10−5h = 10^{-5}, the central difference has a truncation error on the order of h2=10−10h^2 = 10^{-10}, plus rounding. So:

  • Below about 10−710^{-7}: the gradient is right.
  • Between about 10−710^{-7} and 10−410^{-4}: look closer before you trust it. Two innocent causes land here. A ReLU's kink at 0 makes the central difference wrong whenever a pre-activation lies within hh of 0. And an entry whose true gradient is tiny compared with the loss (say 10−610^{-6} against a loss near 2) loses digits to rounding, because the two nudged losses agree in almost every digit. Check which entries are off: a real bug usually shows up across a whole parameter, not in one stray entry.
  • Above about 10−410^{-4}: almost certainly a bug.

Practical rules: check in float64, on a small network and a small batch, a few random entries of every parameter (all entries when there are few), and with any randomness in the forward pass switched off. The code below plants two classic bugs in the backward pass of a two-layer network and shows what the check reports.

⌘+Enter runs · edit freelyPython sleeps until you run code

The correct backward pass agrees to about eight digits or better, everywhere. Forgetting the tanh slope corrupts both first-layer gradients but leaves W2\mathbf{W}_2 perfect, which points straight at the bug. The mean-instead-of-sum bug divides ∂L/∂b1\partial L/\partial\mathbf{b}_1 by NN a second time (the 1/N1/N is already inside δ\boldsymbol{\delta}), and only b1\mathbf{b}_1 fails. Notice that both buggy versions would still train somewhat: the first-layer gradients point in roughly sensible directions. Only the check catches them.

Now build the layers yourself. They are the ones the rest of Part 1 uses. The reference versions in mlref/nn.py have the same forward and backward, plus two small methods, parameters() and grads(), that list a layer's arrays and their gradients, so that the training loop of the next lesson can update them.

Code itLinear, Tanh and ReLU layers

Build the three layers the rest of Part 1 runs on, each with forward(X), which remembers what the backward pass needs, and backward(dY), which takes ∂L/∂Y\partial L/\partial\mathbf{Y} and returns ∂L/∂X\partial L/\partial\mathbf{X}.

  1. Linear(n_in, n_out, rng): Xavier initialization, W of shape (n_in, n_out) with standard deviation 2/(nin+nout)\sqrt{2/(n_{\text{in}} + n_{\text{out}})} and b zero. backward sets self.dW and self.db (overwriting any old values) and returns dX.
  2. Tanh and ReLU, elementwise.

The last lines chain your layers into a small network and gradient-check the first layer's dW. The reference versions in mlref/nn.py have the same forward and backward (plus parameters() and grads() for optimizers), and one test compares yours with them.

⌘+Enter runsPython sleeps until you run code
Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.

A few more backward passes, with fresh numbers each time.

Practice setBackprop reps
3 correct in a row completes the set. A miss starts the count again.

A linear layer computes Y=XW+b\mathbf{Y} = \mathbf{X}\mathbf{W} + \mathbf{b} for a batch, one example per row. Backprop has delivered δ=∂L/∂Y\boldsymbol{\delta} = \partial L/\partial\mathbf{Y}. With W=[−11222−2]\mathbf{W} = \begin{bmatrix}-1 & 1 \\ 2 & 2 \\ 2 & -2\end{bmatrix} and δ=[0−1221−1]\boldsymbol{\delta} = \begin{bmatrix}0 & -1 \\ 2 & 2 \\ 1 & -1\end{bmatrix}, find ∂L/∂X\partial L/\partial\mathbf{X} (the gradient passed back to the previous layer).

∂L/∂X

One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Finally, explain why one backward pass is enough, and what it costs.

In your own wordsExplain backprop's cost

Explain to a colleague who knows the chain rule how backpropagation computes the gradient of the loss with respect to millions of parameters in one backward pass, why that costs only a small multiple of a forward pass, and why training needs much more memory than running the model.

A few sentences first: 0 of 60 characters.

Saved in this browser as you type.

Next: Training a network