When a function takes a vector and returns a vector, its derivative is a matrix, the Jacobian. Composing functions multiplies their Jacobians, and backprop is that product computed from the loss end, one vector at a time.
About 80 minutes
By the end you can
Build the Jacobian of a map from Rn to Rm, read its rows and columns, and predict its shape.
Derive the chain rule Jf∘g=JfJg from linear approximations and apply it.
Write down the Jacobians of a linear layer, an elementwise nonlinearity and softmax, deriving diag(p)−pp⊤.
Explain why backprop passes a vector backwards through vector-Jacobian products instead of building Jacobians, and implement one.
A layer of a network takes a vector and returns a vector. Take one with three inputs and two outputs, say a=tanh(Wx+b) with W of shape 2×3. Nudge the first input a little: both outputs move, each by its own amount. Nudge the second input: both move again, differently. There are 2×3=6 sensitivities, one for every pair of an output and an input, and the natural place to keep six numbers indexed by an output and an input is a 2×3 matrix.
That matrix is the Jacobian. A network is a chain of such functions, and this lesson shows that the chain rule for them is ordinary matrix multiplication. Then it shows the idea that makes training large networks possible at all: backprop computes that product without ever writing down a Jacobian.
The lesson goes in five steps: build the Jacobian and read it, predict its shape, multiply Jacobians along a chain, learn the three Jacobians every network is made of, and see how backprop uses them without building them.
The gradient held one partial derivative per input because there was one output. With several outputs, each output gets its own gradient, and stacking them gives a matrix.
Let f take n inputs and return m outputs, written f:Rn→Rm (here Rn is the set of vectors with n real entries). Its outputs f1,…,fm are each a function of x=(x1,…,xn). The Jacobian J holds one partial derivative for every output and input:
Row i holds the partial derivatives of output i with respect to every input. It is the gradient of fi from the previous lesson, laid on its side.
Column j says how every output moves when you nudge input j alone. This is the column view from module 2: a matrix's columns are where the basis vectors land. Here they are where small nudges along each input axis land.
The linear approximation keeps its familiar form, with the derivative now a matrix:
f(x+h)≈f(x)+J(x)h.
Row i of this is the gradient forecast for output i; the matrix does all m of them at once. Matrices are exactly the linear maps, so the Jacobian is the best linear map approximating f near x. Zoom in far enough on any smooth map and it looks like a matrix, just as any smooth curve looked like a line.
A worked example: polar coordinates. The map f(r,θ)=(rcosθ,rsinθ) turns a radius r and an angle θ into a point (x,y) in the plane. Its four partial derivatives, each with the other input frozen:
x=rcosθ: with θ fixed it is r times a constant, so ∂r∂x=cosθ; with r fixed, ∂θ∂x=−rsinθ.
y=rsinθ: ∂r∂y=sinθ and ∂θ∂y=rcosθ.
Put them in the matrix, outputs down and inputs across:
At r=2, θ=0, where cos0=1 and sin0=0, this is [1002]. Column 1: nudging the radius moves the point outward along x at rate 1. Column 2: nudging the angle moves the point along y at rate 2, because a point on a circle of radius 2 travels 2 units per radian of turn.
Check the approximation with the step h=(0.1,0.1). First Jh=(1×0.1+0×0.1,0×0.1+2×0.1)=(0.1,0.2), so the forecast is (2,0)+(0.1,0.2)=(2.1,0.2). The true point is (2.1cos0.1,2.1sin0.1)≈(2.0895,0.2097). The forecast is off by about 0.01 in each coordinate, the size of the step squared, just as the one-dimensional forecast was.
The Jacobian's determinant has a meaning too. In module 3, detA was the factor by which a matrix scales areas. The Jacobian is the matrix the map looks like near a point, so detJ is the factor by which the map scales small areas there. For polar coordinates, compute it one move at a time:
The last step uses cos2θ+sin2θ=1. So a small rectangle of parameter values, with sides Δr and Δθ (small changes in radius and angle), lands as a patch of area about rΔrΔθ. The same change of angle sweeps out more area far from the center.
The widget shows this for the polar map. The first panel (on the left, or on top on a phone) is the input plane, r across and θ up, with a blue point p and a shaded square of side s around it. The second panel is the output plane: the curved image of the input grid, the square's true image as a white patch, and in amber the parallelogram that J predicts, with its two sides drawn as a blue and a coral arrow. A magnifier in the corner enlarges both shapes. The readouts give p, J(p), detJ, the gap between white and amber as a share of the side, and the true area ratio (white area divided by square area).
Compute a Jacobian and its determinant by hand for a different map.
On paperA Jacobian by hand
Let f(x,y)=(x2−y2,2xy).
Write the Jacobian J(x,y) in general.
Evaluate it at (1,2).
Compute detJ there. By what factor does f scale small areas near (1,2)?
To check your work, enter J at (1,2) as a 2×2 matrix: row 1 for the output x2−y2, row 2 for 2xy, columns in the order x, y.
Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.
J at (1, 2)
One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Before multiplying Jacobians, it pays to predict their shapes, because shapes catch most mistakes. A map f:Rn→Rm has an m×n Jacobian: one row per output, one column per input. Three cases cover everything in this course.
function
Jacobian shape
number to number, R→R
1×1: the ordinary derivative
vector to number, such as a loss L:Rn→R
1×n: the gradient, as a row
vector to vector, such as a layer Rn→Rm
m×n
A layer that maps a 768-dimensional vector to a 3072-dimensional one (the expansion inside each feed-forward block of the smallest GPT-2) has a 3072×768 Jacobian with 3072×768=2,359,296 entries, for a single token. Keep that number in mind for the last section.
One piece of bookkeeping. The Jacobian of a scalar function is a row, 1×n. This course, like almost all code, writes the gradient ∇L as a column with the same shape as the input. They hold the same numbers, arranged differently: ∇L=J⊤. When a formula below needs the column version, you will see a transpose.
A network is one function applied after another, so we need the Jacobian of a composition. Compose two maps: let g:Rn→Rk and f:Rk→Rm, and apply g first. The composition f∘g goes from Rn to Rm, and its Jacobian is a product:
Jf∘g(x)=Jf(g(x))Jg(x).
The shapes fit: (m×k)(k×n) gives m×n, and the shared dimension k is the size of the vector passed from g to f. This is the one-dimensional chain rule with slopes replaced by matrices, and the reason is the same. Near x, g acts like the matrix Jg. Near g(x), f acts like Jf. Applying one linear map after another is multiplying their matrices.
Here is the rule at work on a concrete chain before we prove it. The diagram follows one point through two functions: g turns (x,y) into three numbers (u,v,w)=(x+y,xy,x−y), and f turns those three into one number, f(u,v,w)=uv+w2. The figure has three parts: the chain of vectors, with their shapes written underneath; the Jacobians Jf (1×3) and Jg (3×2) with their product; and, in the last step, a small table that checks the product by nudging. On a wide screen the chain sits to the left of the Jacobians, on a phone above them. Lime marks what each step computes or uses; step through with Next.
Why does the product give the right answer? The slow version below is the one-dimensional proof from the first lesson with every slope replaced by a matrix.
Go slower: The chain rule for Jacobians, step by step
Step 1, the inner map. For a small step h, g(x+h)≈g(x)+Jg(x)h. Write u=g(x) and δ=Jg(x)h, the nudge that arrives at f: a vector with k entries.
Step 2, the outer map. For that small nudge, f(u+δ)≈f(u)+Jf(u)δ.
Step 3, substitute. Replace u by g(x) and δ by Jg(x)h:
f(g(x+h))≈f(g(x))+Jf(g(x))Jg(x)h.
Step 4, read off the matrix. The Jacobian of f∘g is whatever matrix multiplies h in its linear approximation. So Jf∘g(x)=Jf(g(x))Jg(x).
The same thing entry by entry. Row i of the first matrix times column j of the second is
∂xj∂fi=q=1∑k∂uq∂fi∂xj∂uq.
Read it as paths. Input xj influences output fi through each intermediate value uq. Each path contributes the product of its two local slopes, and the paths add. In the diagram, x reached f through u (contributing 2), through v (6) and through w (−2), for a total of 6. This sum over paths is also how backprop handles a value that feeds into several places: the gradients arriving along the different paths are added.
This is the same fact you met in module 2, where the row-times-column rule came from composing two linear maps. Differentiation turns every smooth map into a linear one near a point, so composing smooth maps becomes multiplying their Jacobians.
A second worked example, where both functions return vectors. Let g(x,y)=(x+y,xy), the diagram's g without its third output, and f(u,v)=(u2,uv). Find the Jacobian of f∘g at (x,y)=(1,2).
Where g sends the point:g(1,2)=(1+2,1×2)=(3,2), so f receives (u,v)=(3,2).
Inner Jacobian at (1,2):Jg=[1y1x]=[1211].
Outer Jacobian at (3,2):u2 has partials (2u,0) and uv has partials (v,u), so Jf=[2uv0u]=[6203].
Multiply, outer on the left, one row-times-column product per entry: 6×1+0×2=6, 6×1+0×1=6, 2×1+3×2=8 and 2×1+3×1=5:
[6203][1211]=[6865].
Check directly: f(g(x,y))=((x+y)2,(x+y)xy)=((x+y)2,x2y+xy2). At (1,2) the first output has partial derivatives 2(x+y)=6 and 2(x+y)=6; the second has 2xy+y2=4+4=8 and x2+2xy=1+4=5.
Use that alarm on a chain of your own: predict the shape of its Jacobian and the order of the product.
Quick checkPredict the shapes
A map g takes vectors with 4 entries to vectors with 3 entries, and f takes vectors with 3 entries to vectors with 2 entries. Which statement about the Jacobian of f∘g is right?
Choose one answer, then check.
A network with K layers is a composition of K functions, so its Jacobian is a product of K matrices, JK⋯J2J1. In a recurrent network (module 7) the chain has one factor per time step, and nearly the same matrix appears again and again. Repeated multiplication by one matrix is exactly what module 4 studied: the eigenvalues decide whether the product dies away or explodes. That is the vanishing and exploding gradient problem, and it is the reason LSTMs exist.
So far. Each function in a chain has a Jacobian, evaluated where that function receives its input. The chain's Jacobian is their product, outer on the left, and each entry of the product is a sum over paths.
Now run the rule yourself on a chain whose Jacobians are both 2×2.
Work it outThe chain rule as a product
Let g(x,y)=(xy,x−y) and f(u,v)=(u2+v,uv). Compute the Jacobian of f∘g at (x,y)=(3,1) as the product Jf(g(3,1))Jg(3,1). Enter the 2×2 result with row i for output i of f and columns in the order x, y.
Jacobian of f∘g at (3, 1)
One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Every network in this course is built from three kinds of steps: linear layers, elementwise nonlinearities and a softmax at the end. Knowing their three Jacobians is enough to differentiate the whole network with the chain rule.
A linear layer. In the column convention a linear layer computes z=Wx+b. Output i is zi=∑jWijxj+bi. Only the term Wijxj involves xj, so ∂xj∂zi=Wij, and
J=W.
For example, with W=[102−103], output 1 is x1+2x2+b1, which changes at rate 2 when x2 moves: entry (1,2) of W. The Jacobian of a linear map is the matrix itself, at every point: the best linear approximation of a linear map is the map. (The output also depends on W and b; the gradients with respect to those are derived in module 6.)
In code the convention flips. With a batch in rows, a layer is X @ W + b with W of shape (inputs, outputs), so one example is a row x⊤ and its output row is x⊤W. That is the column formula with the matrix W⊤, so in this convention the Jacobian (outputs down, inputs across) is W⊤. Shapes settle it every time: the Jacobian of a layer from 3 inputs to 2 outputs is 2×3, whichever way the code stores its weights.
An elementwise nonlinearity. Apply a function ϕ, such as tanh or σ, to each entry: ai=ϕ(zi). Output ai depends only on zi, so every partial derivative off the diagonal is zero:
the square matrix with the slopes ϕ′(zi) on its diagonal and zeros elsewhere. For tanh at z=(0,2), the slopes 1−tanh2 are 1 and 0.0707, so J=[1000.0707]. Multiplying a vector by a diagonal matrix is the same as multiplying entry by entry: diag(d)v=d⊙v, where ⊙ is the elementwise product. So code never builds this matrix. For tanh, the diagonal is 1−a2, squared entry by entry, from lesson 1.
The two combine by the chain rule: a whole layer a=tanh(Wx+b) has Jacobian diag(1−a2)W with respect to x, the nonlinearity's Jacobian on the left because it is applied last. Its row i is row i of W scaled by that unit's slope, so a saturated unit (slope near zero) contributes a row of near zeros.
Check what the diagonal structure means for a layer of 4,096 units.
Quick checkThe Jacobian of an elementwise function
A layer applies tanh to each entry of a vector z with 4,096 entries, giving a. Which statement about the Jacobian of this step is right?
Choose one answer, then check.
Softmax. Softmax turns a vector of K scores z (called logits) into probabilities:
pi=Sezi,S=k=1∑Kezk.
Every output depends on every input through the shared sum S, so this Jacobian is full. Its entries are
∂zj∂pi=pi(δij−pj),soJ=diag(p)−pp⊤,
where δij (the Kronecker delta) is 1 when i=j and 0 otherwise, and pp⊤ is the outer product: the K×K matrix whose entry (i,j) is pipj.
Go slower: Deriving the softmax Jacobian
Step 1, write pi as a product.pi=eziS−1, with S=∑kezk.
Step 2, differentiate the first factor.ezi depends on zj only when i=j, so ∂zj∂ezi=δijezi.
Step 3, differentiate the second factor. Only one term of S involves zj, so ∂zj∂S=ezj. By the chain rule (outer S−1 with slope −S−2), ∂zj∂S−1=−S−2ezj.
Step 5, recognize the probabilities.Sezi=pi and Sezj=pj, so ∂zj∂pi=δijpi−pipj=pi(δij−pj).
Step 6, write it as a matrix. The term δijpi is nonzero only on the diagonal, where it is pi: that is diag(p). The term pipj is entry (i,j) of pp⊤. So J=diag(p)−pp⊤.
A worked example with two classes, z=(0,ln3).
Exponentiate.ez=(e0,eln3)=(1,3).
Sum and divide.S=1+3=4, so p=(41,43)=(0.25,0.75).
The outer product. Its entries are 0.25×0.25=0.0625, 0.25×0.75=0.1875 (twice) and 0.75×0.75=0.5625.
Three properties hold for every softmax, and each has a reason:
It is symmetric, because pp⊤ is.
Every column sums to zero (and so does every row, by symmetry). The probabilities always add up to 1, so a nudge to any logit cannot change their total: the changes it causes cancel.
Adding the same number to every logit changes nothing. That nudge points along 1=(1,1,…,1), and J1 is the vector of row sums, which is zero. The algebra agrees exactly, not only to first order: ezi+c=ecezi, and the factor ec cancels between the top and the bottom of pi. This is why the standard trick of subtracting the largest logit before exponentiating, done to avoid overflow, does not change the answer.
With two classes softmax is a sigmoid in disguise. Divide the top and bottom of p2 by ez2:
Here z2−z1=ln3, and the entry 0.1875=163 is exactly σ′(ln3)=43×41. In module 6 this Jacobian meets the cross-entropy loss, and the product collapses to the simple gradient p−y.
The three in one line each. A linear layer's Jacobian is its matrix W. An elementwise function's is the diagonal matrix of its slopes. Softmax's is diag(p)−pp⊤, symmetric with rows that sum to zero.
Derive the two-class softmax Jacobian yourself, through the sigmoid.
On paperThe softmax Jacobian for two classes
With two classes, softmax gives p1=ez1+ez2ez1 and p2=1−p1.
Show that p1=σ(z1−z2).
Use σ′=σ(1−σ) and the chain rule to find all four entries of the Jacobian, and confirm that it equals diag(p)−pp⊤.
Evaluate it at z=(ln4,0).
To check your work, enter J at z=(ln4,0) as a 2×2 matrix of decimals, row i for pi and column j for zj.
Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.
J at z = (ln 4, 0)
One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
The chain rule says the gradient of the loss is a long product of Jacobians. This section shows the cheap way to compute that product, which is what makes training big networks affordable.
Put the pieces together. A network computes a1=f1(x), then a2=f2(a1), and so on up to aK, and finally a scalar loss L(aK). By the chain rule, the derivative of the loss with respect to the input is a product of Jacobians with the loss's 1×n row at the far left:
∂x∂L=1×n∂aK∂LJKJK−1⋯J1.
Matrix multiplication is associative (module 2), so we may group this product however we like, and the grouping decides the cost. Suppose every layer has width n, so every Jk is n×n.
Right to left: compute J2J1, then J3(J2J1), and so on. Each step multiplies two n×n matrices: n2 entries, each a dot product of length n, so about n3 multiply-adds.
Left to right: multiply the row ∂aK∂L by JK, which gives another row; multiply that by JK−1; and so on. Each step multiplies a row vector by a matrix: n entries, each a dot product of length n, so about n2 multiply-adds.
With n=1000 and K=10, left to right costs 10 steps of 106, about 107 multiply-adds. Right to left needs 9 matrix-matrix products of 109 each, about 9×109, nearly a thousand times more. Left to right starts at the loss and works back toward the input, the reverse of the order in which the forward pass ran, so it is called reverse mode. It wins whenever there is one output (the loss) and many inputs (the parameters). That is exactly the situation in training. (The opposite, forward mode, pushes a nudge of one input forward through J1, then J2, and so on; it wins when there are few inputs and many outputs.)
Why the transpose? The row times J from the left-to-right product, g⊤J, is the same set of numbers as the column J⊤g, just stood upright, and code keeps gradients as columns shaped like the values they belong to. The three Jacobians above give three VJP rules:
step
forward
backward (VJP)
cost
linear
z=Wx+b
∇xL=W⊤g
one matrix-vector product
elementwise
a=ϕ(z)
∇zL=ϕ′(z)⊙g
n multiplications
softmax
p=softmax(z)
∇zL=p⊙g−(p⋅g)p
a few passes over K numbers
The softmax row comes from expanding the product. This Jacobian is symmetric, so J⊤=J, and (diag(p)−pp⊤)g=diag(p)g−p(p⊤g)=p⊙g−(p⋅g)p. The full matrix would have K2 entries: for a vocabulary of 50,000 tokens that is 2.5 billion numbers, while the VJP needs a few passes over 50,000.
The table uses the column convention. In row-batch code, where a layer computes Z = X @ W + b, the linear rule reads dX = dZ @ W.T, with dZ holding the gradient with respect to each example's output as a row. It is the same product with every vector turned into a row and the rows stacked, one per example. Module 6 derives it with shapes as the guide.
Here is the difference in practice. The code builds a six-layer tanh network of width 300, computes the gradient of the loss with respect to the input both ways, and checks three entries against finite differences.
⌘+Enter runs · edit freelyPython sleeps until you run code
All three routes agree. The vector route does n=300 times less arithmetic: six matrix-vector products of about n2 multiply-adds each, against six matrix-matrix products of about n3. The measured times differ by less than that, because small operations carry a fixed overhead, but the gap grows with the width. Notice what route 2 needed from the forward pass: the activations ak of every layer, to form the slopes 1−ak2. That is why training stores activations, and why training uses much more memory than running a model.
In short. The gradient is a product of Jacobians, and multiplying it from the loss end keeps every step a vector times a matrix. Each layer turns the incoming gradient g into J⊤g with its own cheap rule, and no Jacobian is ever built.
The table's three rules, plus the weight gradients, are all of backprop for a multilayer network. Module 6 derives the weight gradients and builds layers with forward and backward methods; module 7 and module 8 run the same vector-Jacobian products backwards through time.
Build the tools: a numerical Jacobian, a numerical check of the chain rule, and a softmax VJP that never builds the matrix.
Code itJacobians, the chain rule and a softmax VJP
Build the Jacobian tools from the lesson.
num_jacobian(f, x, h=1e-5): the central-difference Jacobian of a vector function, with shape (m, n) for m outputs and n inputs.
softmax(z), stable for large logits.
softmax_jacobian(z): diag(p)−pp⊤.
softmax_vjp(z, g): the vector-Jacobian product J⊤g with O(K) work, never building J.
One test uses your num_jacobian to verify the chain rule Jf∘g=JfJg on a small two-layer function.
⌘+Enter runsPython sleeps until you run code
Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.
For more practice, work through a few generated Jacobians: small maps by hand, linear layers, elementwise functions, softmax and a two-step chain.
Practice setJacobian reps
3 correct in a row completes the set. A miss starts the count again.
Let f(x,y)=(−2xy+x2,xy−2x2). Find the Jacobian J at (x,y)=(1,2).
J
One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Finally, explain the core idea of this section to a colleague who proposes the expensive route.
In your own wordsWhy backprop passes a vector
A colleague proposes: to get the gradient of the loss with respect to a network's input, compute each layer's Jacobian, multiply them all together, then multiply by the gradient of the loss. Explain why no framework does this, and what backprop does instead.