ml.lab
Python sleeps until you run code
05 Derivatives, gradients and Jacobians

Lesson 4 of 6

Jacobians and the chain rule

When a function takes a vector and returns a vector, its derivative is a matrix, the Jacobian. Composing functions multiplies their Jacobians, and backprop is that product computed from the loss end, one vector at a time.

About 80 minutes
By the end you can
  • Build the Jacobian of a map from Rn\mathbb{R}^n to Rm\mathbb{R}^m, read its rows and columns, and predict its shape.

  • Derive the chain rule Jf∘g=Jf Jg\mathbf{J}_{f\circ g} = \mathbf{J}_f\,\mathbf{J}_g from linear approximations and apply it.

  • Write down the Jacobians of a linear layer, an elementwise nonlinearity and softmax, deriving diag(p)−pp⊤\mathrm{diag}(\mathbf{p}) - \mathbf{p}\mathbf{p}^\top.

  • Explain why backprop passes a vector backwards through vector-Jacobian products instead of building Jacobians, and implement one.

A layer of a network takes a vector and returns a vector. Take one with three inputs and two outputs, say a=tanh⁡(Wx+b)\mathbf{a} = \tanh(\mathbf{W}\mathbf{x} + \mathbf{b}) with W\mathbf{W} of shape 2×32 \times 3. Nudge the first input a little: both outputs move, each by its own amount. Nudge the second input: both move again, differently. There are 2×3=62 \times 3 = 6 sensitivities, one for every pair of an output and an input, and the natural place to keep six numbers indexed by an output and an input is a 2×32 \times 3 matrix.

That matrix is the Jacobian. A network is a chain of such functions, and this lesson shows that the chain rule for them is ordinary matrix multiplication. Then it shows the idea that makes training large networks possible at all: backprop computes that product without ever writing down a Jacobian.

The lesson goes in five steps: build the Jacobian and read it, predict its shape, multiply Jacobians along a chain, learn the three Jacobians every network is made of, and see how backprop uses them without building them.

The matrix of all partial derivatives

The gradient held one partial derivative per input because there was one output. With several outputs, each output gets its own gradient, and stacking them gives a matrix.

Let ff take nn inputs and return mm outputs, written f:Rn→Rmf: \mathbb{R}^n \to \mathbb{R}^m (here Rn\mathbb{R}^n is the set of vectors with nn real entries). Its outputs f1,…,fmf_1,\allowbreak \ldots,\allowbreak f_m are each a function of x=(x1,…,xn)\mathbf{x} = (x_1,\allowbreak \ldots,\allowbreak x_n). The Jacobian J\mathbf{J} holds one partial derivative for every output and input:

Jij=∂fi∂xj,J=[∂f1∂x1⋯∂f1∂xn⋮⋮∂fm∂x1⋯∂fm∂xn].J_{ij} = \frac{\partial f_i}{\partial x_j}, \qquad \mathbf{J} = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \cdots & \frac{\partial f_1}{\partial x_n} \\ \vdots & & \vdots \\ \frac{\partial f_m}{\partial x_1} & \cdots & \frac{\partial f_m}{\partial x_n} \end{bmatrix}.

Read it two ways.

  • Row ii holds the partial derivatives of output ii with respect to every input. It is the gradient of fif_i from the previous lesson, laid on its side.
  • Column jj says how every output moves when you nudge input jj alone. This is the column view from module 2: a matrix's columns are where the basis vectors land. Here they are where small nudges along each input axis land.

The linear approximation keeps its familiar form, with the derivative now a matrix:

f(x+h)≈f(x)+J(x) h.f(\mathbf{x} + \mathbf{h}) \approx f(\mathbf{x}) + \mathbf{J}(\mathbf{x})\,\mathbf{h}.

Row ii of this is the gradient forecast for output ii; the matrix does all mm of them at once. Matrices are exactly the linear maps, so the Jacobian is the best linear map approximating ff near x\mathbf{x}. Zoom in far enough on any smooth map and it looks like a matrix, just as any smooth curve looked like a line.

A worked example: polar coordinates. The map f(r,θ)=(rcos⁡θ, rsin⁡θ)f(r,\allowbreak \theta) = (r\cos\theta,\allowbreak \ r\sin\theta) turns a radius rr and an angle θ\theta into a point (x,y)(x,\allowbreak y) in the plane. Its four partial derivatives, each with the other input frozen:

  • x=rcos⁡θx = r\cos\theta: with θ\theta fixed it is rr times a constant, so ∂x∂r=cos⁡θ\frac{\partial x}{\partial r} = \cos\theta; with rr fixed, ∂x∂θ=−rsin⁡θ\frac{\partial x}{\partial \theta} = -r\sin\theta.
  • y=rsin⁡θy = r\sin\theta: ∂y∂r=sin⁡θ\frac{\partial y}{\partial r} = \sin\theta and ∂y∂θ=rcos⁡θ\frac{\partial y}{\partial \theta} = r\cos\theta.

Put them in the matrix, outputs down and inputs across:

J=[∂x∂r∂x∂θ∂y∂r∂y∂θ]=[cos⁡θ−rsin⁡θsin⁡θrcos⁡θ].\mathbf{J} = \begin{bmatrix} \frac{\partial x}{\partial r} & \frac{\partial x}{\partial \theta} \\ \frac{\partial y}{\partial r} & \frac{\partial y}{\partial \theta} \end{bmatrix} = \begin{bmatrix} \cos\theta & -r\sin\theta \\ \sin\theta & r\cos\theta \end{bmatrix}.

At r=2r = 2, θ=0\theta = 0, where cos⁡0=1\cos 0 = 1 and sin⁡0=0\sin 0 = 0, this is [1002]\begin{bmatrix} 1 & 0 \\ 0 & 2 \end{bmatrix}. Column 1: nudging the radius moves the point outward along xx at rate 1. Column 2: nudging the angle moves the point along yy at rate 2, because a point on a circle of radius 2 travels 2 units per radian of turn.

Check the approximation with the step h=(0.1,0.1)\mathbf{h} = (0.1,\allowbreak 0.1). First Jh=(1×0.1+0×0.1, 0×0.1+2×0.1)=(0.1,0.2)\mathbf{J}\mathbf{h} = (1 \times 0.1 + 0 \times 0.1,\allowbreak \ 0 \times 0.1 + 2 \times 0.1) = (0.1,\allowbreak 0.2), so the forecast is (2,0)+(0.1,0.2)=(2.1,0.2)(2,\allowbreak 0) + (0.1,\allowbreak 0.2) = (2.1,\allowbreak 0.2). The true point is (2.1cos⁡0.1, 2.1sin⁡0.1)≈(2.0895,0.2097)(2.1\cos 0.1,\allowbreak \ 2.1\sin 0.1) \approx (2.0895,\allowbreak 0.2097). The forecast is off by about 0.010.01 in each coordinate, the size of the step squared, just as the one-dimensional forecast was.

The Jacobian's determinant has a meaning too. In module 3, det⁡A\det\mathbf{A} was the factor by which a matrix scales areas. The Jacobian is the matrix the map looks like near a point, so det⁡J\det\mathbf{J} is the factor by which the map scales small areas there. For polar coordinates, compute it one move at a time:

det⁡J=(cos⁡θ)(rcos⁡θ)−(−rsin⁡θ)(sin⁡θ)=rcos⁡2θ+rsin⁡2θ=r(cos⁡2θ+sin⁡2θ)=r.\det\mathbf{J} = (\cos\theta)(r\cos\theta) - (-r\sin\theta)(\sin\theta) = r\cos^2\theta + r\sin^2\theta = r(\cos^2\theta + \sin^2\theta) = r.

The last step uses cos⁡2θ+sin⁡2θ=1\cos^2\theta + \sin^2\theta = 1. So a small rectangle of parameter values, with sides Δr\Delta r and Δθ\Delta\theta (small changes in radius and angle), lands as a patch of area about r Δr Δθr\,\Delta r\,\Delta\theta. The same change of angle sweeps out more area far from the center.

The widget shows this for the polar map. The first panel (on the left, or on top on a phone) is the input plane, rr across and θ\theta up, with a blue point p\mathbf{p} and a shaded square of side ss around it. The second panel is the output plane: the curved image of the input grid, the square's true image as a white patch, and in amber the parallelogram that J\mathbf{J} predicts, with its two sides drawn as a blue and a coral arrow. A magnifier in the corner enlarges both shapes. The readouts give p\mathbf{p}, J(p)\mathbf{J}(\mathbf{p}), det⁡J\det\mathbf{J}, the gap between white and amber as a share of the side, and the true area ratio (white area divided by square area).

Compute a Jacobian and its determinant by hand for a different map.

On paperA Jacobian by hand

Let f(x,y)=(x2−y2, 2xy)f(x,\allowbreak y) = (x^2 - y^2,\allowbreak \ 2xy).

  1. Write the Jacobian J(x,y)\mathbf{J}(x,\allowbreak y) in general.
  2. Evaluate it at (1,2)(1,\allowbreak 2).
  3. Compute det⁡J\det\mathbf{J} there. By what factor does ff scale small areas near (1,2)(1,\allowbreak 2)?

To check your work, enter J\mathbf{J} at (1,2)(1,\allowbreak 2) as a 2×22 \times 2 matrix: row 1 for the output x2−y2x^2 - y^2, row 2 for 2xy2xy, columns in the order xx, yy.

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

J at (1, 2)

One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Shapes: outputs down, inputs across

Before multiplying Jacobians, it pays to predict their shapes, because shapes catch most mistakes. A map f:Rn→Rmf: \mathbb{R}^n \to \mathbb{R}^m has an m×nm \times n Jacobian: one row per output, one column per input. Three cases cover everything in this course.

functionJacobian shape
number to number, R→R\mathbb{R} \to \mathbb{R}1×11 \times 1: the ordinary derivative
vector to number, such as a loss L:Rn→RL: \mathbb{R}^n \to \mathbb{R}1×n1 \times n: the gradient, as a row
vector to vector, such as a layer Rn→Rm\mathbb{R}^n \to \mathbb{R}^mm×nm \times n

A layer that maps a 768-dimensional vector to a 3072-dimensional one (the expansion inside each feed-forward block of the smallest GPT-2) has a 3072×7683072 \times 768 Jacobian with 3072×768=2,359,2963072 \times 768 = 2{,}359{,}296 entries, for a single token. Keep that number in mind for the last section.

One piece of bookkeeping. The Jacobian of a scalar function is a row, 1×n1 \times n. This course, like almost all code, writes the gradient ∇L\nabla L as a column with the same shape as the input. They hold the same numbers, arranged differently: ∇L=J⊤\nabla L = \mathbf{J}^\top. When a formula below needs the column version, you will see a transpose.

The chain rule is matrix multiplication

A network is one function applied after another, so we need the Jacobian of a composition. Compose two maps: let g:Rn→Rkg: \mathbb{R}^n \to \mathbb{R}^k and f:Rk→Rmf: \mathbb{R}^k \to \mathbb{R}^m, and apply gg first. The composition f∘gf\circ g goes from Rn\mathbb{R}^n to Rm\mathbb{R}^m, and its Jacobian is a product:

Jf∘g(x)=Jf(g(x)) Jg(x).\mathbf{J}_{f\circ g}(\mathbf{x}) = \mathbf{J}_f\bigl(g(\mathbf{x})\bigr)\,\mathbf{J}_g(\mathbf{x}).

The shapes fit: (m×k)(k×n)(m \times k)(k \times n) gives m×nm \times n, and the shared dimension kk is the size of the vector passed from gg to ff. This is the one-dimensional chain rule with slopes replaced by matrices, and the reason is the same. Near x\mathbf{x}, gg acts like the matrix Jg\mathbf{J}_g. Near g(x)g(\mathbf{x}), ff acts like Jf\mathbf{J}_f. Applying one linear map after another is multiplying their matrices.

Here is the rule at work on a concrete chain before we prove it. The diagram follows one point through two functions: gg turns (x,y)(x,\allowbreak y) into three numbers (u,v,w)=(x+y, xy, x−y)(u,\allowbreak v,\allowbreak w) = (x + y,\allowbreak \ xy,\allowbreak \ x - y), and ff turns those three into one number, f(u,v,w)=uv+w2f(u,\allowbreak v,\allowbreak w) = uv + w^2. The figure has three parts: the chain of vectors, with their shapes written underneath; the Jacobians Jf\mathbf{J}_f (1×31 \times 3) and Jg\mathbf{J}_g (3×23 \times 2) with their product; and, in the last step, a small table that checks the product by nudging. On a wide screen the chain sits to the left of the Jacobians, on a phone above them. Lime marks what each step computes or uses; step through with Next.

Why does the product give the right answer? The slow version below is the one-dimensional proof from the first lesson with every slope replaced by a matrix.

Go slower: The chain rule for Jacobians, step by step

Step 1, the inner map. For a small step h\mathbf{h}, g(x+h)≈g(x)+Jg(x) hg(\mathbf{x} + \mathbf{h}) \approx g(\mathbf{x}) + \mathbf{J}_g(\mathbf{x})\,\mathbf{h}. Write u=g(x)\mathbf{u} = g(\mathbf{x}) and δ=Jg(x) h\boldsymbol{\delta} = \mathbf{J}_g(\mathbf{x})\,\mathbf{h}, the nudge that arrives at ff: a vector with kk entries.

Step 2, the outer map. For that small nudge, f(u+δ)≈f(u)+Jf(u) δf(\mathbf{u} + \boldsymbol{\delta}) \approx f(\mathbf{u}) + \mathbf{J}_f(\mathbf{u})\,\boldsymbol{\delta}.

Step 3, substitute. Replace u\mathbf{u} by g(x)g(\mathbf{x}) and δ\boldsymbol{\delta} by Jg(x) h\mathbf{J}_g(\mathbf{x})\,\mathbf{h}:

f(g(x+h))≈f(g(x))+Jf(g(x)) Jg(x) h.f\bigl(g(\mathbf{x} + \mathbf{h})\bigr) \approx f\bigl(g(\mathbf{x})\bigr) + \mathbf{J}_f\bigl(g(\mathbf{x})\bigr)\,\mathbf{J}_g(\mathbf{x})\,\mathbf{h}.

Step 4, read off the matrix. The Jacobian of f∘gf\circ g is whatever matrix multiplies h\mathbf{h} in its linear approximation. So Jf∘g(x)=Jf(g(x)) Jg(x)\mathbf{J}_{f\circ g}(\mathbf{x}) = \mathbf{J}_f(g(\mathbf{x}))\,\mathbf{J}_g(\mathbf{x}).

The same thing entry by entry. Row ii of the first matrix times column jj of the second is

∂fi∂xj=∑q=1k∂fi∂uq ∂uq∂xj.\frac{\partial f_i}{\partial x_j} = \sum_{q=1}^{k} \frac{\partial f_i}{\partial u_q}\,\frac{\partial u_q}{\partial x_j}.

Read it as paths. Input xjx_j influences output fif_i through each intermediate value uqu_q. Each path contributes the product of its two local slopes, and the paths add. In the diagram, xx reached ff through uu (contributing 22), through vv (66) and through ww (−2-2), for a total of 66. This sum over paths is also how backprop handles a value that feeds into several places: the gradients arriving along the different paths are added.

This is the same fact you met in module 2, where the row-times-column rule came from composing two linear maps. Differentiation turns every smooth map into a linear one near a point, so composing smooth maps becomes multiplying their Jacobians.

A second worked example, where both functions return vectors. Let g(x,y)=(x+y, xy)g(x,\allowbreak y) = (x + y,\allowbreak \ xy), the diagram's gg without its third output, and f(u,v)=(u2, uv)f(u,\allowbreak v) = (u^2,\allowbreak \ uv). Find the Jacobian of f∘gf\circ g at (x,y)=(1,2)(x,\allowbreak y) = (1,\allowbreak 2).

  1. Where gg sends the point: g(1,2)=(1+2, 1×2)=(3,2)g(1,\allowbreak 2) = (1 + 2,\allowbreak \ 1 \times 2) = (3,\allowbreak 2), so ff receives (u,v)=(3,2)(u,\allowbreak v) = (3,\allowbreak 2).

  2. Inner Jacobian at (1,2)(1,\allowbreak 2): Jg=[11yx]=[1121]\mathbf{J}_g = \begin{bmatrix} 1 & 1 \\ y & x \end{bmatrix} = \begin{bmatrix} 1 & 1 \\ 2 & 1 \end{bmatrix}.

  3. Outer Jacobian at (3,2)(3,\allowbreak 2): u2u^2 has partials (2u,0)(2u,\allowbreak 0) and uvuv has partials (v,u)(v,\allowbreak u), so Jf=[2u0vu]=[6023]\mathbf{J}_f = \begin{bmatrix} 2u & 0 \\ v & u \end{bmatrix} = \begin{bmatrix} 6 & 0 \\ 2 & 3 \end{bmatrix}.

  4. Multiply, outer on the left, one row-times-column product per entry: 6×1+0×2=66 \times 1 + 0 \times 2 = 6, 6×1+0×1=66 \times 1 + 0 \times 1 = 6, 2×1+3×2=82 \times 1 + 3 \times 2 = 8 and 2×1+3×1=52 \times 1 + 3 \times 1 = 5:

    [6023][1121]=[6685].\begin{bmatrix} 6 & 0 \\ 2 & 3 \end{bmatrix}\begin{bmatrix} 1 & 1 \\ 2 & 1 \end{bmatrix} = \begin{bmatrix} 6 & 6 \\ 8 & 5 \end{bmatrix}.

Check directly: f(g(x,y))=((x+y)2, (x+y)xy)=((x+y)2, x2y+xy2)f(g(x,\allowbreak y)) = \bigl((x + y)^2,\allowbreak \ (x + y)xy\bigr) = \bigl((x + y)^2,\allowbreak \ x^2 y + x y^2\bigr). At (1,2)(1,\allowbreak 2) the first output has partial derivatives 2(x+y)=62(x + y) = 6 and 2(x+y)=62(x + y) = 6; the second has 2xy+y2=4+4=82xy + y^2 = 4 + 4 = 8 and x2+2xy=1+4=5x^2 + 2xy = 1 + 4 = 5.

Use that alarm on a chain of your own: predict the shape of its Jacobian and the order of the product.

Quick checkPredict the shapes

A map gg takes vectors with 4 entries to vectors with 3 entries, and ff takes vectors with 3 entries to vectors with 2 entries. Which statement about the Jacobian of f∘gf\circ g is right?

Choose one answer, then check.

A network with KK layers is a composition of KK functions, so its Jacobian is a product of KK matrices, JK⋯J2J1\mathbf{J}_K\cdots\mathbf{J}_2\mathbf{J}_1. In a recurrent network (module 7) the chain has one factor per time step, and nearly the same matrix appears again and again. Repeated multiplication by one matrix is exactly what module 4 studied: the eigenvalues decide whether the product dies away or explodes. That is the vanishing and exploding gradient problem, and it is the reason LSTMs exist.

So far. Each function in a chain has a Jacobian, evaluated where that function receives its input. The chain's Jacobian is their product, outer on the left, and each entry of the product is a sum over paths.

Now run the rule yourself on a chain whose Jacobians are both 2×22 \times 2.

Work it outThe chain rule as a product

Let g(x,y)=(xy, x−y)g(x,\allowbreak y) = (xy,\allowbreak \ x - y) and f(u,v)=(u2+v, uv)f(u,\allowbreak v) = (u^2 + v,\allowbreak \ uv). Compute the Jacobian of f∘gf\circ g at (x,y)=(3,1)(x,\allowbreak y) = (3,\allowbreak 1) as the product Jf(g(3,1)) Jg(3,1)\mathbf{J}_f\bigl(g(3,\allowbreak 1)\bigr)\,\mathbf{J}_g(3,\allowbreak 1). Enter the 2×22 \times 2 result with row ii for output ii of ff and columns in the order xx, yy.

Jacobian of f∘g at (3, 1)

One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Three Jacobians you will use constantly

Every network in this course is built from three kinds of steps: linear layers, elementwise nonlinearities and a softmax at the end. Knowing their three Jacobians is enough to differentiate the whole network with the chain rule.

A linear layer. In the column convention a linear layer computes z=Wx+b\mathbf{z} = \mathbf{W}\mathbf{x} + \mathbf{b}. Output ii is zi=∑jWijxj+biz_i = \sum_j W_{ij}x_j + b_i. Only the term WijxjW_{ij}x_j involves xjx_j, so ∂zi∂xj=Wij\frac{\partial z_i}{\partial x_j} = W_{ij}, and

J=W.\mathbf{J} = \mathbf{W}.

For example, with W=[1200−13]\mathbf{W} = \begin{bmatrix} 1 & 2 & 0 \\ 0 & -1 & 3 \end{bmatrix}, output 1 is x1+2x2+b1x_1 + 2x_2 + b_1, which changes at rate 2 when x2x_2 moves: entry (1,2)(1,\allowbreak 2) of W\mathbf{W}. The Jacobian of a linear map is the matrix itself, at every point: the best linear approximation of a linear map is the map. (The output also depends on W\mathbf{W} and b\mathbf{b}; the gradients with respect to those are derived in module 6.)

In code the convention flips. With a batch in rows, a layer is X @ W + b with W of shape (inputs, outputs), so one example is a row x⊤\mathbf{x}^\top and its output row is x⊤W\mathbf{x}^\top\mathbf{W}. That is the column formula with the matrix W⊤\mathbf{W}^\top, so in this convention the Jacobian (outputs down, inputs across) is W⊤\mathbf{W}^\top. Shapes settle it every time: the Jacobian of a layer from 3 inputs to 2 outputs is 2×32 \times 3, whichever way the code stores its weights.

An elementwise nonlinearity. Apply a function ϕ\phi, such as tanh⁡\tanh or σ\sigma, to each entry: ai=ϕ(zi)a_i = \phi(z_i). Output aia_i depends only on ziz_i, so every partial derivative off the diagonal is zero:

∂ai∂zj={ϕ′(zi)if i=j,0if i≠j,soJ=diag(ϕ′(z)),\frac{\partial a_i}{\partial z_j} = \begin{cases} \phi'(z_i) & \text{if } i = j, \\ 0 & \text{if } i \ne j, \end{cases} \qquad\text{so}\qquad \mathbf{J} = \mathrm{diag}\bigl(\phi'(\mathbf{z})\bigr),

the square matrix with the slopes ϕ′(zi)\phi'(z_i) on its diagonal and zeros elsewhere. For tanh⁡\tanh at z=(0,2)\mathbf{z} = (0,\allowbreak 2), the slopes 1−tanh⁡21 - \tanh^2 are 11 and 0.07070.0707, so J=[1000.0707]\mathbf{J} = \begin{bmatrix} 1 & 0 \\ 0 & 0.0707 \end{bmatrix}. Multiplying a vector by a diagonal matrix is the same as multiplying entry by entry: diag(d) v=d⊙v\mathrm{diag}(\mathbf{d})\,\mathbf{v} = \mathbf{d}\odot\mathbf{v}, where ⊙\odot is the elementwise product. So code never builds this matrix. For tanh⁡\tanh, the diagonal is 1−a21 - \mathbf{a}^2, squared entry by entry, from lesson 1.

The two combine by the chain rule: a whole layer a=tanh⁡(Wx+b)\mathbf{a} = \tanh(\mathbf{W}\mathbf{x} + \mathbf{b}) has Jacobian diag(1−a2) W\mathrm{diag}(1 - \mathbf{a}^2)\,\mathbf{W} with respect to x\mathbf{x}, the nonlinearity's Jacobian on the left because it is applied last. Its row ii is row ii of W\mathbf{W} scaled by that unit's slope, so a saturated unit (slope near zero) contributes a row of near zeros.

Check what the diagonal structure means for a layer of 4,096 units.

Quick checkThe Jacobian of an elementwise function

A layer applies tanh⁡\tanh to each entry of a vector z\mathbf{z} with 4,096 entries, giving a\mathbf{a}. Which statement about the Jacobian of this step is right?

Choose one answer, then check.

Softmax. Softmax turns a vector of KK scores z\mathbf{z} (called logits) into probabilities:

pi=eziS,S=∑k=1Kezk.p_i = \frac{e^{z_i}}{S}, \qquad S = \sum_{k=1}^{K} e^{z_k}.

Every output depends on every input through the shared sum SS, so this Jacobian is full. Its entries are

∂pi∂zj=pi (δij−pj),soJ=diag(p)−pp⊤,\frac{\partial p_i}{\partial z_j} = p_i\,(\delta_{ij} - p_j), \qquad\text{so}\qquad \mathbf{J} = \mathrm{diag}(\mathbf{p}) - \mathbf{p}\mathbf{p}^\top,

where δij\delta_{ij} (the Kronecker delta) is 1 when i=ji = j and 0 otherwise, and pp⊤\mathbf{p}\mathbf{p}^\top is the outer product: the K×KK \times K matrix whose entry (i,j)(i,\allowbreak j) is pipjp_i p_j.

Go slower: Deriving the softmax Jacobian

Step 1, write pip_i as a product. pi=ezi S−1p_i = e^{z_i}\,S^{-1}, with S=∑kezkS = \sum_k e^{z_k}.

Step 2, differentiate the first factor. ezie^{z_i} depends on zjz_j only when i=ji = j, so ∂ezi∂zj=δij ezi\frac{\partial e^{z_i}}{\partial z_j} = \delta_{ij}\,e^{z_i}.

Step 3, differentiate the second factor. Only one term of SS involves zjz_j, so ∂S∂zj=ezj\frac{\partial S}{\partial z_j} = e^{z_j}. By the chain rule (outer S−1S^{-1} with slope −S−2-S^{-2}), ∂S−1∂zj=−S−2 ezj\frac{\partial S^{-1}}{\partial z_j} = -S^{-2}\,e^{z_j}.

Step 4, product rule.

∂pi∂zj=δij ezi S−1+ezi(−S−2 ezj)=δij eziS−eziS⋅ezjS.\frac{\partial p_i}{\partial z_j} = \delta_{ij}\,e^{z_i}\,S^{-1} + e^{z_i}\bigl(-S^{-2}\,e^{z_j}\bigr) = \delta_{ij}\,\frac{e^{z_i}}{S} - \frac{e^{z_i}}{S}\cdot\frac{e^{z_j}}{S}.

Step 5, recognize the probabilities. eziS=pi\frac{e^{z_i}}{S} = p_i and ezjS=pj\frac{e^{z_j}}{S} = p_j, so ∂pi∂zj=δij pi−pi pj=pi(δij−pj)\frac{\partial p_i}{\partial z_j} = \delta_{ij}\,p_i - p_i\,p_j = p_i(\delta_{ij} - p_j).

Step 6, write it as a matrix. The term δij pi\delta_{ij}\,p_i is nonzero only on the diagonal, where it is pip_i: that is diag(p)\mathrm{diag}(\mathbf{p}). The term pipjp_i p_j is entry (i,j)(i,\allowbreak j) of pp⊤\mathbf{p}\mathbf{p}^\top. So J=diag(p)−pp⊤\mathbf{J} = \mathrm{diag}(\mathbf{p}) - \mathbf{p}\mathbf{p}^\top.

A worked example with two classes, z=(0,ln⁡3)\mathbf{z} = (0,\allowbreak \ln 3).

  1. Exponentiate. ez=(e0,eln⁡3)=(1,3)e^{\mathbf{z}} = (e^0,\allowbreak e^{\ln 3}) = (1,\allowbreak 3).

  2. Sum and divide. S=1+3=4S = 1 + 3 = 4, so p=(14,34)=(0.25,0.75)\mathbf{p} = (\tfrac{1}{4},\allowbreak \tfrac{3}{4}) = (0.25,\allowbreak 0.75).

  3. The outer product. Its entries are 0.25×0.25=0.06250.25 \times 0.25 = 0.0625, 0.25×0.75=0.18750.25 \times 0.75 = 0.1875 (twice) and 0.75×0.75=0.56250.75 \times 0.75 = 0.5625.

  4. Subtract it from diag(p)\mathrm{diag}(\mathbf{p}):

    J=[0.25000.75]−[0.06250.18750.18750.5625]=[0.1875−0.1875−0.18750.1875].\mathbf{J} = \begin{bmatrix} 0.25 & 0 \\ 0 & 0.75 \end{bmatrix} - \begin{bmatrix} 0.0625 & 0.1875 \\ 0.1875 & 0.5625 \end{bmatrix} = \begin{bmatrix} 0.1875 & -0.1875 \\ -0.1875 & 0.1875 \end{bmatrix}.

Three properties hold for every softmax, and each has a reason:

  • It is symmetric, because pp⊤\mathbf{p}\mathbf{p}^\top is.
  • Every column sums to zero (and so does every row, by symmetry). The probabilities always add up to 1, so a nudge to any logit cannot change their total: the changes it causes cancel.
  • Adding the same number to every logit changes nothing. That nudge points along 1=(1,1,…,1)\mathbf{1} = (1,\allowbreak 1,\allowbreak \ldots,\allowbreak 1), and J1\mathbf{J}\mathbf{1} is the vector of row sums, which is zero. The algebra agrees exactly, not only to first order: ezi+c=ec ezie^{z_i + c} = e^{c}\,e^{z_i}, and the factor ece^{c} cancels between the top and the bottom of pip_i. This is why the standard trick of subtracting the largest logit before exponentiating, done to avoid overflow, does not change the answer.

With two classes softmax is a sigmoid in disguise. Divide the top and bottom of p2p_2 by ez2e^{z_2}:

p2=ez2ez1+ez2=1ez1−z2+1=11+e−(z2−z1)=σ(z2−z1).p_2 = \frac{e^{z_2}}{e^{z_1} + e^{z_2}} = \frac{1}{e^{z_1 - z_2} + 1} = \frac{1}{1 + e^{-(z_2 - z_1)}} = \sigma(z_2 - z_1).

Here z2−z1=ln⁡3z_2 - z_1 = \ln 3, and the entry 0.1875=3160.1875 = \frac{3}{16} is exactly σ′(ln⁡3)=34×14\sigma'(\ln 3) = \tfrac{3}{4} \times \tfrac{1}{4}. In module 6 this Jacobian meets the cross-entropy loss, and the product collapses to the simple gradient p−y\mathbf{p} - \mathbf{y}.

The three in one line each. A linear layer's Jacobian is its matrix W\mathbf{W}. An elementwise function's is the diagonal matrix of its slopes. Softmax's is diag(p)−pp⊤\mathrm{diag}(\mathbf{p}) - \mathbf{p}\mathbf{p}^\top, symmetric with rows that sum to zero.

Derive the two-class softmax Jacobian yourself, through the sigmoid.

On paperThe softmax Jacobian for two classes

With two classes, softmax gives p1=ez1ez1+ez2p_1 = \frac{e^{z_1}}{e^{z_1} + e^{z_2}} and p2=1−p1p_2 = 1 - p_1.

  1. Show that p1=σ(z1−z2)p_1 = \sigma(z_1 - z_2).
  2. Use σ′=σ(1−σ)\sigma' = \sigma(1 - \sigma) and the chain rule to find all four entries of the Jacobian, and confirm that it equals diag(p)−pp⊤\mathrm{diag}(\mathbf{p}) - \mathbf{p}\mathbf{p}^\top.
  3. Evaluate it at z=(ln⁡4,0)\mathbf{z} = (\ln 4,\allowbreak 0).

To check your work, enter J\mathbf{J} at z=(ln⁡4,0)\mathbf{z} = (\ln 4,\allowbreak 0) as a 2×22 \times 2 matrix of decimals, row ii for pip_i and column jj for zjz_j.

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

J at z = (ln 4, 0)

One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Backprop never builds a Jacobian

The chain rule says the gradient of the loss is a long product of Jacobians. This section shows the cheap way to compute that product, which is what makes training big networks affordable.

Put the pieces together. A network computes a1=f1(x)\mathbf{a}_1 = f_1(\mathbf{x}), then a2=f2(a1)\mathbf{a}_2 = f_2(\mathbf{a}_1), and so on up to aK\mathbf{a}_K, and finally a scalar loss L(aK)L(\mathbf{a}_K). By the chain rule, the derivative of the loss with respect to the input is a product of Jacobians with the loss's 1×n1 \times n row at the far left:

∂L∂x=∂L∂aK⏟1×n JK JK−1⋯J1.\frac{\partial L}{\partial \mathbf{x}} = \underbrace{\frac{\partial L}{\partial \mathbf{a}_K}}_{1 \times n}\,\mathbf{J}_K\,\mathbf{J}_{K-1}\cdots\mathbf{J}_1 .

Matrix multiplication is associative (module 2), so we may group this product however we like, and the grouping decides the cost. Suppose every layer has width nn, so every Jk\mathbf{J}_k is n×nn \times n.

  • Right to left: compute J2J1\mathbf{J}_2\mathbf{J}_1, then J3(J2J1)\mathbf{J}_3(\mathbf{J}_2\mathbf{J}_1), and so on. Each step multiplies two n×nn \times n matrices: n2n^2 entries, each a dot product of length nn, so about n3n^3 multiply-adds.
  • Left to right: multiply the row ∂L∂aK\frac{\partial L}{\partial \mathbf{a}_K} by JK\mathbf{J}_K, which gives another row; multiply that by JK−1\mathbf{J}_{K-1}; and so on. Each step multiplies a row vector by a matrix: nn entries, each a dot product of length nn, so about n2n^2 multiply-adds.

With n=1000n = 1000 and K=10K = 10, left to right costs 1010 steps of 10610^6, about 10710^7 multiply-adds. Right to left needs 99 matrix-matrix products of 10910^9 each, about 9×1099 \times 10^9, nearly a thousand times more. Left to right starts at the loss and works back toward the input, the reverse of the order in which the forward pass ran, so it is called reverse mode. It wins whenever there is one output (the loss) and many inputs (the parameters). That is exactly the situation in training. (The opposite, forward mode, pushes a nudge of one input forward through J1\mathbf{J}_1, then J2\mathbf{J}_2, and so on; it wins when there are few inputs and many outputs.)

Why the transpose? The row times J\mathbf{J} from the left-to-right product, g⊤J\mathbf{g}^\top\mathbf{J}, is the same set of numbers as the column J⊤g\mathbf{J}^\top\mathbf{g}, just stood upright, and code keeps gradients as columns shaped like the values they belong to. The three Jacobians above give three VJP rules:

stepforwardbackward (VJP)cost
linearz=Wx+b\mathbf{z} = \mathbf{W}\mathbf{x} + \mathbf{b}∇xL=W⊤g\nabla_{\mathbf{x}} L = \mathbf{W}^\top\mathbf{g}one matrix-vector product
elementwisea=ϕ(z)\mathbf{a} = \phi(\mathbf{z})∇zL=ϕ′(z)⊙g\nabla_{\mathbf{z}} L = \phi'(\mathbf{z})\odot\mathbf{g}nn multiplications
softmaxp=softmax(z)\mathbf{p} = \mathrm{softmax}(\mathbf{z})∇zL=p⊙g−(p⋅g) p\nabla_{\mathbf{z}} L = \mathbf{p}\odot\mathbf{g} - (\mathbf{p}\cdot\mathbf{g})\,\mathbf{p}a few passes over KK numbers

The softmax row comes from expanding the product. This Jacobian is symmetric, so J⊤=J\mathbf{J}^\top = \mathbf{J}, and (diag(p)−pp⊤) g=diag(p) g−p (p⊤g)=p⊙g−(p⋅g) p(\mathrm{diag}(\mathbf{p}) - \mathbf{p}\mathbf{p}^\top)\,\mathbf{g} = \mathrm{diag}(\mathbf{p})\,\mathbf{g} - \mathbf{p}\,(\mathbf{p}^\top\mathbf{g}) = \mathbf{p}\odot\mathbf{g} - (\mathbf{p}\cdot\mathbf{g})\,\mathbf{p}. The full matrix would have K2K^2 entries: for a vocabulary of 50,000 tokens that is 2.5 billion numbers, while the VJP needs a few passes over 50,000.

The table uses the column convention. In row-batch code, where a layer computes Z = X @ W + b, the linear rule reads dX = dZ @ W.T, with dZ holding the gradient with respect to each example's output as a row. It is the same product with every vector turned into a row and the rows stacked, one per example. Module 6 derives it with shapes as the guide.

Here is the difference in practice. The code builds a six-layer tanh network of width 300, computes the gradient of the loss with respect to the input both ways, and checks three entries against finite differences.

⌘+Enter runs · edit freelyPython sleeps until you run code

All three routes agree. The vector route does n=300n = 300 times less arithmetic: six matrix-vector products of about n2n^2 multiply-adds each, against six matrix-matrix products of about n3n^3. The measured times differ by less than that, because small operations carry a fixed overhead, but the gap grows with the width. Notice what route 2 needed from the forward pass: the activations ak\mathbf{a}_k of every layer, to form the slopes 1−ak21 - \mathbf{a}_k^2. That is why training stores activations, and why training uses much more memory than running a model.

In short. The gradient is a product of Jacobians, and multiplying it from the loss end keeps every step a vector times a matrix. Each layer turns the incoming gradient g\mathbf{g} into J⊤g\mathbf{J}^\top\mathbf{g} with its own cheap rule, and no Jacobian is ever built.

The table's three rules, plus the weight gradients, are all of backprop for a multilayer network. Module 6 derives the weight gradients and builds layers with forward and backward methods; module 7 and module 8 run the same vector-Jacobian products backwards through time.

Build the tools: a numerical Jacobian, a numerical check of the chain rule, and a softmax VJP that never builds the matrix.

Code itJacobians, the chain rule and a softmax VJP

Build the Jacobian tools from the lesson.

  • num_jacobian(f, x, h=1e-5): the central-difference Jacobian of a vector function, with shape (m, n) for m outputs and n inputs.
  • softmax(z), stable for large logits.
  • softmax_jacobian(z): diag(p)−pp⊤\mathrm{diag}(\mathbf{p}) - \mathbf{p}\mathbf{p}^\top.
  • softmax_vjp(z, g): the vector-Jacobian product J⊤g\mathbf{J}^\top\mathbf{g} with O(K)O(K) work, never building J\mathbf{J}.

One test uses your num_jacobian to verify the chain rule Jf∘g=Jf Jg\mathbf{J}_{f\circ g} = \mathbf{J}_f\,\mathbf{J}_g on a small two-layer function.

⌘+Enter runsPython sleeps until you run code
Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.

For more practice, work through a few generated Jacobians: small maps by hand, linear layers, elementwise functions, softmax and a two-step chain.

Practice setJacobian reps
3 correct in a row completes the set. A miss starts the count again.

Let f(x,y)=(−2xy+x2, xy−2x2)\mathbf{f}(x,\allowbreak y) = \big(-2xy + x^2,\allowbreak \ xy - 2x^2\big). Find the Jacobian J\mathbf{J} at (x,y)=(1,2)(x,\allowbreak y) = (1,\allowbreak 2).

J

One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Finally, explain the core idea of this section to a colleague who proposes the expensive route.

In your own wordsWhy backprop passes a vector

A colleague proposes: to get the gradient of the loss with respect to a network's input, compute each layer's Jacobian, multiply them all together, then multiply by the gradient of the loss. Explain why no framework does this, and what backprop does instead.

A few sentences first: 0 of 60 characters.

Saved in this browser as you type.

Next: Gradient descent