ml.lab
Python sleeps until you run code
05 Derivatives, gradients and Jacobians

Lesson 3 of 6

Gradients

With many inputs, the derivative becomes a vector with one entry per input. It points in the direction the function climbs fastest, and every training step moves against it.

About 65 minutes
By the end you can
  • Compute partial derivatives and assemble them into a gradient.

  • Use the directional derivative ∇f⋅u\nabla f\cdot\mathbf{u} to explain why the gradient points uphill fastest and is perpendicular to contour lines.

  • Derive the gradients of w⋅x\mathbf{w}\cdot\mathbf{x}, ∣x∣2|\mathbf{x}|^2, x⊤Ax\mathbf{x}^\top\mathbf{A}\mathbf{x} and the mean squared error of linear regression.

  • Write a numerical gradient for any function and use it to check an analytic gradient.

Now the model has two weights, and the loss L(w1,w2)L(w_1,\allowbreak w_2) is a landscape: a height above every point of a plane. You are standing somewhere on it in thick fog. You cannot see the valley, but you can feel the ground under your feet. Take a small step east and notice how much you drop. Take a small step north and notice again.

Those two numbers turn out to be enough. From them alone you can work out the slope in every other direction, including which way is steepest downhill and how steep it is. The pair is called the gradient. A language model's loss is a landscape over billions of weights, and its gradient has billions of entries, but it is built and read exactly the way this lesson builds and reads the two-entry version.

The lesson goes in five steps: measure one input at a time (partial derivatives), combine them into a forecast, find which way is uphill, work out the gradients of the pieces that models are built from, and assemble those pieces into the gradient of linear regression's loss.

Partial derivatives: wiggle one input

With one input there was one slope. With two inputs there is a slope for each, and the simplest way to measure one of them is to keep the other input still. For a function of several inputs, the partial derivative with respect to one input is the ordinary derivative you get by holding every other input fixed. It is written with a curly ∂\partial to signal that the others are frozen: ∂f∂x\frac{\partial f}{\partial x} is read "the partial of ff with respect to xx".

Take f(x,y)=x2y+3yf(x,\allowbreak y) = x^2 y + 3y at the point (2,1)(2,\allowbreak 1), where f=22×1+3×1=4+3=7f = 2^2 \times 1 + 3 \times 1 = 4 + 3 = 7.

  • For ∂f∂x\frac{\partial f}{\partial x}, treat yy as a constant. Then x2yx^2 y is a constant times x2x^2, with derivative 2x⋅y=2xy2x \cdot y = 2xy, and 3y3y does not change with xx at all. So ∂f∂x=2xy\frac{\partial f}{\partial x} = 2xy, which is 2×2×1=42 \times 2 \times 1 = 4 at (2,1)(2,\allowbreak 1).
  • For ∂f∂y\frac{\partial f}{\partial y}, treat xx as a constant. Then x2yx^2 y is the constant x2x^2 times yy, with slope x2x^2, and 3y3y has slope 3. So ∂f∂y=x2+3\frac{\partial f}{\partial y} = x^2 + 3, which is 4+3=74 + 3 = 7 at (2,1)(2,\allowbreak 1).

You can check both by experiment. Nudge xx alone: f(2.01,1)=2.012+3=4.0401+3=7.0401f(2.01,\allowbreak 1) = 2.01^2 + 3 = 4.0401 + 3 = 7.0401, a change of 0.04010.0401 for a nudge of 0.010.01, a rate of 4.014.01. Nudge yy alone: f(2,1.01)=4×1.01+3×1.01=4.04+3.03=7.07f(2,\allowbreak 1.01) = 4 \times 1.01 + 3 \times 1.01 = 4.04 + 3.03 = 7.07, a change of 0.070.07, a rate of exactly 77, because with xx held fixed ff is a straight line in yy.

The gradient collects all the partial derivatives into one vector:

∇f=(∂f∂x1,∂f∂x2,…,∂f∂xn).\nabla f = \left(\frac{\partial f}{\partial x_1}, \frac{\partial f}{\partial x_2}, \ldots, \frac{\partial f}{\partial x_n}\right).

The symbol ∇\nabla is read "grad" (or "nabla"). Here ∇f(2,1)=(4,7)\nabla f(2,\allowbreak 1) = (4,\allowbreak 7). When a function has several groups of inputs, a subscript says which ones we differentiate with respect to: ∇wL\nabla_{\mathbf{w}} L is the gradient of the loss with respect to the weights only, with the data held fixed.

The gradient predicts small changes

In one dimension the derivative gave a forecast, f(x+h)≈f(x)+f′(x) hf(x + h) \approx f(x) + f'(x)\,h. The gradient gives the same kind of forecast when every input moves at once. Take a small step h=(h1,…,hn)\mathbf{h} = (h_1,\allowbreak \ldots,\allowbreak h_n). Each input's nudge contributes its partial derivative times its size, and the contributions add up:

f(x+h)≈f(x)+∂f∂x1h1+⋯+∂f∂xnhn=f(x)+∇f(x)⋅h.f(\mathbf{x} + \mathbf{h}) \approx f(\mathbf{x}) + \frac{\partial f}{\partial x_1} h_1 + \cdots + \frac{\partial f}{\partial x_n} h_n = f(\mathbf{x}) + \nabla f(\mathbf{x})\cdot\mathbf{h}.

The last form is a dot product, from module 1: the gradient dotted with the step.

With our example, step from (2,1)(2,\allowbreak 1) to (2.1,0.9)(2.1,\allowbreak 0.9), so h=(0.1,−0.1)\mathbf{h} = (0.1,\allowbreak -0.1):

f(2.1,0.9)≈7+4×0.1+7×(−0.1)=7+0.4−0.7=6.7.f(2.1, 0.9) \approx 7 + 4 \times 0.1 + 7 \times (-0.1) = 7 + 0.4 - 0.7 = 6.7.

The true value is 2.12×0.9+3×0.9=3.969+2.7=6.6692.1^2 \times 0.9 + 3 \times 0.9 = 3.969 + 2.7 = 6.669, so the forecast is off by 0.0310.031.

Go slower: Why the separate contributions add

Take two inputs. Get from (x,y)(x,\allowbreak y) to (x+h1,y+h2)(x + h_1,\allowbreak y + h_2) in two moves, first changing yy and then changing xx. Adding and subtracting the halfway value f(x,y+h2)f(x,\allowbreak y + h_2) splits the total change into two pieces:

f(x+h1,y+h2)−f(x,y)=f(x+h1,y+h2)−f(x,y+h2)⏟only x changes+f(x,y+h2)−f(x,y)⏟only y changes.f(x + h_1, y + h_2) - f(x, y) = \underbrace{f(x + h_1, y + h_2) - f(x, y + h_2)}_{\text{only } x \text{ changes}} + \underbrace{f(x, y + h_2) - f(x, y)}_{\text{only } y \text{ changes}}.

The second piece changes only yy, so by the one-dimensional approximation it is about ∂f∂y(x,y) h2\frac{\partial f}{\partial y}(x,\allowbreak y)\,h_2.

The first piece changes only xx, at height y+h2y + h_2, so it is about ∂f∂x(x,y+h2) h1\frac{\partial f}{\partial x}(x,\allowbreak y + h_2)\,h_1.

When the partial derivatives change smoothly from point to point (true for every function in this course apart from kinks like ReLU's corner), ∂f∂x(x,y+h2)\frac{\partial f}{\partial x}(x,\allowbreak y + h_2) is close to ∂f∂x(x,y)\frac{\partial f}{\partial x}(x,\allowbreak y) for small h2h_2. So the total change is about ∂f∂xh1+∂f∂yh2\frac{\partial f}{\partial x}h_1 + \frac{\partial f}{\partial y}h_2.

With nn inputs, make nn moves, one coordinate at a time, and the same argument gives one term per coordinate.

So far. A partial derivative measures one input's effect with the others frozen. The gradient stacks them into a vector, and the gradient dotted with a small step forecasts how much the function changes.

Try both on one function: find its gradient at a point, then use it to forecast a nearby value.

On paperA gradient and a prediction

Let f(x,y)=x2y−3xy+y2f(x,\allowbreak y) = x^2 y - 3xy + y^2.

  1. Find ∂f∂x\frac{\partial f}{\partial x} and ∂f∂y\frac{\partial f}{\partial y}, and evaluate the gradient at (2,−1)(2,\allowbreak -1).
  2. Use the linear approximation to estimate f(2.1,−0.9)f(2.1,\allowbreak -0.9). You will need f(2,−1)f(2,\allowbreak -1).

To check your work, enter ∂f∂x\frac{\partial f}{\partial x} and ∂f∂y\frac{\partial f}{\partial y} at (2,−1)(2,\allowbreak -1) and then the estimate, in that order, as decimals.

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

∂f/∂x and ∂f/∂y at (2, -1), then the estimate

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Which way is uphill

Training wants the direction in which the loss falls fastest. The gradient answers that, and the reason takes one line of algebra plus the geometry of the dot product.

Stand at x\mathbf{x} and choose a direction, given as a unit vector u\mathbf{u} (a vector of length 1). Walk a small distance tt along it, so the step is h=t u\mathbf{h} = t\,\mathbf{u}. The forecast from the last section says

f(x+t u)≈f(x)+∇f(x)⋅(t u)=f(x)+t ∇f(x)⋅u.f(\mathbf{x} + t\,\mathbf{u}) \approx f(\mathbf{x}) + \nabla f(\mathbf{x})\cdot(t\,\mathbf{u}) = f(\mathbf{x}) + t\,\nabla f(\mathbf{x})\cdot\mathbf{u}.

So the function changes by about ∇f⋅u\nabla f\cdot\mathbf{u} per unit of distance walked. That rate is the slope in the direction u\mathbf{u}, called the directional derivative:

Duf=∇f⋅u.D_{\mathbf{u}} f = \nabla f\cdot\mathbf{u}.

The partial derivatives are the special cases u=(1,0)\mathbf{u} = (1,\allowbreak 0) and u=(0,1)\mathbf{u} = (0,\allowbreak 1). Now use the geometric reading of the dot product from module 1: ∇f⋅u=∣∇f∣ ∣u∣cos⁡θ=∣∇f∣cos⁡θ\nabla f\cdot\mathbf{u} = |\nabla f|\,|\mathbf{u}|\cos\theta = |\nabla f|\cos\theta, where θ\theta is the angle between the gradient and your direction, and ∣u∣=1|\mathbf{u}| = 1. Three facts follow.

  1. Uphill fastest along the gradient. cos⁡θ\cos\theta is largest, 11, at θ=0\theta = 0. The steepest slope available is ∣∇f∣|\nabla f|, the length of the gradient.
  2. Downhill fastest against it. At θ=180°\theta = 180° the slope is −∣∇f∣-|\nabla f|. This is the direction gradient descent steps in.
  3. Level along the contours. At θ=90°\theta = 90° the slope is zero: to first order, the function does not change. A contour line (a curve of constant height, as on a hiking map) runs in exactly those directions. So the gradient is perpendicular to the contour lines.

A smooth hill drawn above its own contour map. On the flat map below, a short arrow starts on one contour line and points across it, at a right angle, toward the higher contours in the middle; nearer the steep side, where the contour lines crowd together, a second arrow is longer.The gradient is an arrow on the map, not on the hillside: it crosses each contour at a right angle and grows where the lines crowd together.

A worked example. Let f(x,y)=x2+2y2f(x,\allowbreak y) = x^2 + 2y^2 at (3,1)(3,\allowbreak 1), where f=9+2=11f = 9 + 2 = 11. The partial derivatives are 2x2x and 4y4y, so the gradient is (2×3,4×1)=(6,4)(2 \times 3,\allowbreak 4 \times 1) = (6,\allowbreak 4), with length 62+42=36+16=52≈7.21\sqrt{6^2 + 4^2} = \sqrt{36 + 16} = \sqrt{52} \approx 7.21. Here is the slope ∇f⋅u\nabla f\cdot\mathbf{u} in five directions:

direction u\mathbf{u}slope ∇f⋅u\nabla f\cdot\mathbf{u}
east, (1,0)(1,\allowbreak 0)6×1+4×0=66 \times 1 + 4 \times 0 = 6
north, (0,1)(0,\allowbreak 1)6×0+4×1=46 \times 0 + 4 \times 1 = 4
diagonal, (1,1)/2(1,\allowbreak 1)/\sqrt{2}(6+4)/2=10/2≈7.07(6 + 4)/\sqrt{2} = 10/\sqrt{2} \approx 7.07
along the gradient, (6,4)/52(6,\allowbreak 4)/\sqrt{52}(36+16)/52=52≈7.21(36 + 16)/\sqrt{52} = \sqrt{52} \approx 7.21
perpendicular, (−4,6)/52(-4,\allowbreak 6)/\sqrt{52}(−24+24)/52=0(-24 + 24)/\sqrt{52} = 0

The last row's direction comes from swapping the gradient's two entries and flipping the sign of one: (6,4)(6,\allowbreak 4) becomes (−4,6)(-4,\allowbreak 6), and the dot product 6×(−4)+4×66 \times (-4) + 4 \times 6 is then zero by construction. No direction beats 7.217.21. The last direction, (−4,6)(-4,\allowbreak 6), is the way the contour x2+2y2=11x^2 + 2y^2 = 11 runs through (3,1)(3,\allowbreak 1): walk that way and, to first order, you stay at height 11.

The widget shows the same facts on a different surface, L=12x2+5y2L = \tfrac{1}{2}x^2 + 5y^2: a long valley whose minimum is at the origin. It is drawn from above as a contour map. The pink lines join points of equal loss, the shading darkens toward low loss, and the lime cross marks the minimum. The small amber arrows show the gradient's direction on a grid of points. The white probe is the point the readouts describe: its own amber arrow is ∇L\nabla L there, and the dashed white line through it follows the contour.

Here is the cosine law measured directly. The code walks around the point (3,1)(3,\allowbreak 1) of f=x2+2y2f = x^2 + 2y^2, one direction every 5 degrees, estimates the slope in each direction with a central difference from derivatives by experiment, and plots it next to the prediction ∇f⋅u=∣∇f∣cos⁡θ\nabla f\cdot\mathbf{u} = |\nabla f|\cos\theta.

⌘+Enter runs · edit freelyPython sleeps until you run code

The two curves lie on top of each other: a cosine wave with amplitude ∣∇f∣≈7.21|\nabla f| \approx 7.21, peaking in the gradient's direction (about 33.7°33.7°; the nearest direction tried is 35°35°).

So far. The slope in any unit direction u\mathbf{u} is ∇f⋅u=∣∇f∣cos⁡θ\nabla f\cdot\mathbf{u} = |\nabla f|\cos\theta. So the gradient points up the steepest slope, its length is that slope, and it crosses every contour at a right angle. Gradient descent walks the opposite way.

Check the three facts on a gradient of your own, including the classic slip of mixing up a direction with a step.

Quick checkSteepest way down

At a point, the gradient of a loss is ∇L=(3,−4)\nabla L = (3,\allowbreak -4). In which unit direction does the loss decrease fastest, and at what rate (change in loss per unit distance)?

Choose one answer, then check.

Now put the whole argument of this section into your own words, starting from the forecast and ending at the contour lines.

In your own wordsWhy the gradient points uphill

Explain, without assuming the result, why the gradient points in the direction of steepest ascent, why its length is the steepest slope, and why it is perpendicular to contour lines. Build the argument from the directional derivative.

A few sentences first: 0 of 60 characters.

Saved in this browser as you type.

Gradients of the pieces you will meet

Most gradients in machine learning are assembled from a few pieces, so it pays to know those pieces by heart. A reliable way to find a gradient you do not know has three moves: write the function as a sum over indices, take the partial derivative with respect to a single entry, say wkw_k, and then recognize the resulting list of numbers as a vector expression.

A dot product with a fixed vector. f(w)=w⋅x=w1x1+⋯+wnxnf(\mathbf{w}) = \mathbf{w}\cdot\mathbf{x} = w_1 x_1 + \cdots + w_n x_n. Only the term wkxkw_k x_k involves wkw_k, so ∂f∂wk=xk\frac{\partial f}{\partial w_k} = x_k. With two entries: ∂f∂w1=x1\frac{\partial f}{\partial w_1} = x_1 and ∂f∂w2=x2\frac{\partial f}{\partial w_2} = x_2, and stacking them gives back x\mathbf{x}. In general,

∇w(w⋅x)=x.\nabla_{\mathbf{w}}(\mathbf{w}\cdot\mathbf{x}) = \mathbf{x}.

This is the gradient of a neuron's weighted sum with respect to its weights: the input itself. In module 6 it becomes the rule that each weight's gradient is the input it saw times the error signal arriving from above.

A squared length. f(x)=∣x∣2=x12+⋯+xn2f(\mathbf{x}) = |\mathbf{x}|^2 = x_1^2 + \cdots + x_n^2. Only the term xk2x_k^2 involves xkx_k, so ∂f∂xk=2xk\frac{\partial f}{\partial x_k} = 2x_k, and

∇∣x∣2=2x.\nabla|\mathbf{x}|^2 = 2\mathbf{x}.

This is the one-dimensional rule x2↦2xx^2 \mapsto 2x, entry by entry. The gradient points straight away from the origin, which fits the circular contours of this function.

A quadratic form. For a square matrix A\mathbf{A}, let f(x)=x⊤Axf(\mathbf{x}) = \mathbf{x}^\top\mathbf{A}\mathbf{x}. (The transpose x⊤\mathbf{x}^\top turns the column x\mathbf{x} into a row, so the result is a single number: the dot product of x\mathbf{x} with Ax\mathbf{A}\mathbf{x}.) Written as a double sum, f=∑i∑jxiAijxjf = \sum_i\sum_j x_i A_{ij} x_j, and its gradient is

∇(x⊤Ax)=(A+A⊤) x,which is 2Ax when A is symmetric.\nabla(\mathbf{x}^\top\mathbf{A}\mathbf{x}) = (\mathbf{A} + \mathbf{A}^\top)\,\mathbf{x}, \qquad \text{which is } 2\mathbf{A}\mathbf{x} \text{ when } \mathbf{A} \text{ is symmetric}.

The symmetric case is the matrix version of ax2↦2axa x^2 \mapsto 2ax.

Go slower: The gradient of a quadratic form, index by index

Step 1, differentiate one term. Each term xiAijxjx_i A_{ij} x_j contains the product xixjx_i x_j. By the product rule, its partial derivative with respect to xkx_k is

Aij(∂xi∂xk xj+xi ∂xj∂xk),A_{ij}\left(\frac{\partial x_i}{\partial x_k}\,x_j + x_i\,\frac{\partial x_j}{\partial x_k}\right),

where ∂xi∂xk\frac{\partial x_i}{\partial x_k} is 11 when i=ki = k and 00 otherwise.

Step 2, sum over all terms. In the first part only the terms with i=ki = k survive; in the second only those with j=kj = k:

∂f∂xk=∑jAkj xj+∑ixi Aik.\frac{\partial f}{\partial x_k} = \sum_j A_{kj}\,x_j + \sum_i x_i\,A_{ik}.

Step 3, recognize the sums. The first sum is entry kk of Ax\mathbf{A}\mathbf{x} (row kk of A\mathbf{A} dotted with x\mathbf{x}). The second is ∑i(A⊤)ki xi\sum_i (\mathbf{A}^\top)_{ki}\,x_i, because Aik=(A⊤)kiA_{ik} = (\mathbf{A}^\top)_{ki}, so it is entry kk of A⊤x\mathbf{A}^\top\mathbf{x}.

Step 4, stack the entries. ∇f=Ax+A⊤x=(A+A⊤) x\nabla f = \mathbf{A}\mathbf{x} + \mathbf{A}^\top\mathbf{x} = (\mathbf{A} + \mathbf{A}^\top)\,\mathbf{x}. If A=A⊤\mathbf{A} = \mathbf{A}^\top, this is 2Ax2\mathbf{A}\mathbf{x}.

A worked example with the symmetric matrix A=[2113]\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 1 & 3 \end{bmatrix} and x=(1,2)\mathbf{x} = (1,\allowbreak 2).

  1. Multiply. Ax=(2×1+1×2, 1×1+3×2)=(4,7)\mathbf{A}\mathbf{x} = (2 \times 1 + 1 \times 2,\allowbreak \ 1 \times 1 + 3 \times 2) = (4,\allowbreak 7).
  2. The value. f=x⋅Ax=1×4+2×7=18f = \mathbf{x}\cdot\mathbf{A}\mathbf{x} = 1 \times 4 + 2 \times 7 = 18.
  3. The gradient. A\mathbf{A} is symmetric, so ∇f=2Ax=(8,14)\nabla f = 2\mathbf{A}\mathbf{x} = (8,\allowbreak 14).

Check by multiplying out: f=2x12+2x1x2+3x22f = 2x_1^2 + 2x_1x_2 + 3x_2^2, so ∂f∂x1=4x1+2x2=4+4=8\frac{\partial f}{\partial x_1} = 4x_1 + 2x_2 = 4 + 4 = 8 and ∂f∂x2=2x1+6x2=2+12=14\frac{\partial f}{\partial x_2} = 2x_1 + 6x_2 = 2 + 12 = 14.

Quadratic forms matter because every smooth loss looks like one near its minimum. That fact runs the whole analysis of gradient descent two lessons from now.

The next exercise uses a matrix that is not symmetric, where the shortcut 2Ax2\mathbf{A}\mathbf{x} gives the wrong answer.

Work it outA quadratic form that is not symmetric

Let A=[1203]\mathbf{A} = \begin{bmatrix} 1 & 2 \\ 0 & 3 \end{bmatrix} and f(x)=x⊤Axf(\mathbf{x}) = \mathbf{x}^\top\mathbf{A}\mathbf{x}. Find ∇f\nabla f at x=(1,1)\mathbf{x} = (1,\allowbreak 1). Enter it as a column: ∂f∂x1\frac{\partial f}{\partial x_1} on top, ∂f∂x2\frac{\partial f}{\partial x_2} below.

∇f(1, 1)

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

The gradient of linear regression

Linear regression is the first model you will train in this course, and its gradient has the shape every layer's weight gradient will have: the inputs, transposed, times the errors. It is built from the pieces above plus the chain rule.

Linear regression predicts a number from dd features with a weight vector w\mathbf{w}. Stack NN examples as the rows of a matrix X\mathbf{X} with shape N×dN \times d (the row-batch convention from module 2), and their targets in a vector y\mathbf{y} of length NN. The predictions are Xw\mathbf{X}\mathbf{w}, one per example. The residuals (prediction minus target) are r=Xw−y\mathbf{r} = \mathbf{X}\mathbf{w} - \mathbf{y}, and the loss is the mean squared error

L(w)=1N∑n=1Nrn2=1N ∣Xw−y∣2.L(\mathbf{w}) = \frac{1}{N}\sum_{n=1}^{N} r_n^2 = \frac{1}{N}\,|\mathbf{X}\mathbf{w} - \mathbf{y}|^2 .

Its gradient is

∇wL=2N X⊤(Xw−y).\nabla_{\mathbf{w}} L = \frac{2}{N}\,\mathbf{X}^\top(\mathbf{X}\mathbf{w} - \mathbf{y}).
Go slower: Deriving the regression gradient

Step 1, one residual. Residual nn is rn=∑kXnkwk−ynr_n = \sum_k X_{nk} w_k - y_n. Only the term XnkwkX_{nk} w_k involves wkw_k, so ∂rn∂wk=Xnk\frac{\partial r_n}{\partial w_k} = X_{nk}.

Step 2, one squared residual (chain rule). The outer function is squaring, with slope 2rn2r_n; the inner function is rnr_n. So ∂(rn2)∂wk=2rn∂rn∂wk=2rnXnk\frac{\partial (r_n^2)}{\partial w_k} = 2 r_n \frac{\partial r_n}{\partial w_k} = 2 r_n X_{nk}.

Step 3, average over examples (sum rule).

∂L∂wk=1N∑n=1N2rnXnk=2N∑n=1N(X⊤)kn rn,\frac{\partial L}{\partial w_k} = \frac{1}{N}\sum_{n=1}^{N} 2 r_n X_{nk} = \frac{2}{N}\sum_{n=1}^{N} (\mathbf{X}^\top)_{kn}\, r_n,

where the last form only renames XnkX_{nk} as (X⊤)kn(\mathbf{X}^\top)_{kn}.

Step 4, recognize the matrix product. The last sum is row kk of X⊤\mathbf{X}^\top dotted with r\mathbf{r}: entry kk of X⊤r\mathbf{X}^\top\mathbf{r}. So ∇wL=2NX⊤r=2NX⊤(Xw−y)\nabla_{\mathbf{w}} L = \frac{2}{N}\mathbf{X}^\top\mathbf{r} = \frac{2}{N}\mathbf{X}^\top(\mathbf{X}\mathbf{w} - \mathbf{y}).

Shape check. X⊤\mathbf{X}^\top is d×Nd \times N and r\mathbf{r} has length NN, so the result has length dd: one entry per weight, as a gradient must.

A worked example with three points. Let

X=[111213],y=[122],w=[01].\mathbf{X} = \begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix}, \qquad \mathbf{y} = \begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix}, \qquad \mathbf{w} = \begin{bmatrix} 0 \\ 1 \end{bmatrix}.

The column of ones is a standard trick: its weight w1w_1 acts as the bias (the intercept), so the model is y^=w1+w2x\hat{y} = w_1 + w_2 x with feature values x=1,2,3x = 1,\allowbreak 2,\allowbreak 3.

  1. Predictions. Xw=(0+1, 0+2, 0+3)=(1,2,3)\mathbf{X}\mathbf{w} = (0 + 1,\allowbreak \ 0 + 2,\allowbreak \ 0 + 3) = (1,\allowbreak 2,\allowbreak 3).

  2. Residuals. r=(1−1, 2−2, 3−2)=(0,0,1)\mathbf{r} = (1 - 1,\allowbreak \ 2 - 2,\allowbreak \ 3 - 2) = (0,\allowbreak 0,\allowbreak 1).

  3. Loss. L=13(02+02+12)=13L = \frac{1}{3}(0^2 + 0^2 + 1^2) = \frac{1}{3}.

  4. X⊤r\mathbf{X}^\top\mathbf{r}. Row 1 of X⊤\mathbf{X}^\top is (1,1,1)(1,\allowbreak 1,\allowbreak 1) and row 2 is (1,2,3)(1,\allowbreak 2,\allowbreak 3), so X⊤r=(0+0+1, 0+0+3)=(1,3)\mathbf{X}^\top\mathbf{r} = (0 + 0 + 1,\allowbreak \ 0 + 0 + 3) = (1,\allowbreak 3).

  5. Gradient. Multiply by 2N=23\frac{2}{N} = \frac{2}{3}:

    ∇wL=23(1,3)=(23, 2).\nabla_{\mathbf{w}} L = \frac{2}{3}(1, 3) = \left(\tfrac{2}{3},\ 2\right).

Read it: only the third example is wrong, and it is over-predicted, so both partial derivatives are positive and the gradient says "decrease both weights". The slope's entry is three times the bias's because that example's feature value is 33.

Now derive the same gradient for a line fit of your own, as sums over the points, and evaluate it.

On paperOne gradient for a line fit

Fit y^=wx+b\hat{y} = wx + b to three points (x,y)(x,\allowbreak y): (0,1)(0,\allowbreak 1), (1,3)(1,\allowbreak 3) and (2,4)(2,\allowbreak 4), with loss L=13∑n(y^n−yn)2L = \frac{1}{3}\sum_n (\hat{y}_n - y_n)^2.

  1. Write X\mathbf{X} (one row per point, with a column of ones for the bias) and derive ∂L∂w\frac{\partial L}{\partial w} and ∂L∂b\frac{\partial L}{\partial b} as sums over the points.
  2. Evaluate the loss and both partial derivatives at w=1w = 1, b=1b = 1.
  3. Which way should each parameter move?

To check your work, enter LL, ∂L∂w\frac{\partial L}{\partial w} and ∂L∂b\frac{\partial L}{\partial b} at w=1w = 1, b=1b = 1, in that order, as fractions (such as 5/6) or decimals to 3 places.

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

L, ∂L/∂w and ∂L/∂b at w = 1, b = 1

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Deriving gradients by hand is error-prone, so every gradient you derive in this course gets checked against a numerical one: the central differences from the previous lesson, applied one coordinate at a time. Nudge one entry up by hh, then down by hh, divide the difference in outputs by 2h2h, put the entry back, and move to the next one. Build that checker now; you will use the idea in every later module.

Code itA gradient checker

Write a gradient checker you can reuse, and two analytic gradients to check with it.

  • num_grad(f, x, h=1e-5): the central-difference gradient of a scalar function f at an array x of any shape. It must not change x, and it must work when x is a list of integers.
  • grad_quadratic(A, x): the exact gradient of x⊤Ax\mathbf{x}^\top\mathbf{A}\mathbf{x} for a square A\mathbf{A} that need not be symmetric.
  • mse_loss_and_grad(X, y, w): the mean squared error of the predictions X @ w, and its gradient 2NX⊤(Xw−y)\frac{2}{N}\mathbf{X}^\top(\mathbf{X}\mathbf{w} - \mathbf{y}).

The tests compare your analytic gradients with numerical ones, which is how every backward pass in this course gets checked.

⌘+Enter runsPython sleeps until you run code
Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.

For more practice, work through a few generated gradients: partial derivatives, directional derivatives, quadratic forms and a small regression loss.

Practice setGradient reps
3 correct in a row completes the set. A miss starts the count again.

Let f(x,y)=−x2+y2+2x−2yf(x,\allowbreak y) = -x^2 + y^2 + 2x - 2y. Find the directional derivative Duf=∇f⋅uD_{\mathbf{u}}f = \nabla f\cdot\mathbf{u} at (x,y)=(1,0)(x,\allowbreak y) = (1,\allowbreak 0) in the direction of the unit vector u=(0,−1)\mathbf{u} = (0,\allowbreak -1). Give the exact value.

D_u f

Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.

Next: Jacobians and the chain rule