ml.lab
Python sleeps until you run code
05 Derivatives, gradients and Jacobians

Lesson 1 of 6

Derivatives as zoom

Zoom in far enough on a smooth curve and it becomes a straight line. The slope of that line is the derivative, and the approximation it gives is the idea every training algorithm runs on.

About 65 minutes
By the end you can
  • Explain the derivative as the slope of the best local straight-line approximation, and use that approximation to estimate values.

  • Apply the sum, power, product and chain rules, and say why each one holds.

  • Derive σ′(x)=σ(x)(1−σ(x))\sigma'(x) = \sigma(x)(1 - \sigma(x)) and state the derivatives of exe^x, ln⁡x\ln x, tanh⁡x\tanh x and ReLU.

  • Explain why saturated sigmoid and tanh units pass almost no gradient back.

You are training a model with a single weight ww. The loss is L(w)=(w−3)2L(w) = (w - 3)^2, but pretend you cannot see that formula: all you can do is set the weight, run the model and read the loss. At w=1w = 1 the loss is 44. Nudge the weight to w=1.01w = 1.01 and the loss becomes 3.96013.9601. The loss changed by 3.9601−4=−0.03993.9601 - 4 = -0.0399 while the weight changed by 0.010.01, so it moved at a rate of −0.0399/0.01=−3.99-0.0399 / 0.01 = -3.99: about −4-4 units of loss per unit of weight.

That rate is the derivative, and it answers the one question training asks over and over: if I move this weight a little, what happens to the loss? A negative rate says "increase the weight and the loss goes down". A language model asks that question for billions of weights at once, on every training step. Modules 1 to 4 built the linear algebra of what a network computes; this module adds the calculus of how to change it, and builds the tools to answer that question for vectors and matrices. This lesson starts with one input and one output, because every later idea is this one wearing more dimensions.

The lesson builds the idea in four steps: the picture (zoom in until a curve is a line), what that line predicts, the rules that compute derivatives without any zooming, and the derivatives of the functions networks are made of. The next lesson measures derivatives by experiment, so you can check your formulas.

Zoom in until the curve is a line

A curve is hard to reason about, because its steepness keeps changing. A straight line is easy: it changes at one fixed rate everywhere. The first idea of calculus is that a smooth curve, looked at closely enough around one point, is a straight line.

A large magnifying glass held over one point of a smooth curve. Outside the lens the curve bends visibly; inside the lens the magnified piece of the same curve is a straight line.Up close, a smooth curve cannot be told apart from a straight line.

The widget below lets you test that claim on sin⁡x\sin x. The white curve is f(x)=sin⁡xf(x) = \sin x, the blue dot is the point x0=1x_0 = 1 on it, and the amber line is the straight line through that point with the curve's steepness there. The readouts give x0x_0, the height f(x0)f(x_0), the slope f′(x0)f'(x_0) of the amber line, and the largest gap between the curve and the line anywhere in view.

The line the curve turns into is the tangent line, and its slope is the derivative of ff at xx, written f′(x)f'(x) and read "f prime of x". For sin⁡\sin at x=1x = 1 it is 0.54030.5403, the number in the widget.

To pin the slope down with numbers, take a second point a small step hh to the right of the first. The straight line through (x,f(x))(x,\allowbreak f(x)) and (x+h,f(x+h))(x + h,\allowbreak f(x + h)) is called a secant. Its slope is rise over run: the change in output, f(x+h)−f(x)f(x + h) - f(x), divided by the change in input, hh. Now let the second point slide toward the first:

f′(x)=lim⁡h→0f(x+h)−f(x)h.f'(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h}.

Read lim⁡h→0\lim_{h \to 0} as "the value this fraction settles on as hh gets closer and closer to zero". We cannot set h=0h = 0 directly, because the fraction would become 0/00/0; the limit asks where the fraction is heading instead.

A worked example: f(x)=x2f(x) = x^2 at x=3x = 3, where f(3)=9f(3) = 9. Work out the secant's slope one move at a time.

  1. Expand the square: (3+h)2=9+6h+h2(3 + h)^2 = 9 + 6h + h^2.
  2. Subtract f(3)=9f(3) = 9 to get the rise: 6h+h26h + h^2.
  3. Divide by the run hh to get the slope: 6h+h2h=6+h\frac{6h + h^2}{h} = 6 + h.

With h=0.1h = 0.1 the secant's slope is 6.16.1; with h=0.01h = 0.01 it is 6.016.01. As hh shrinks the slope settles on 66, so f′(3)=6f'(3) = 6: near x=3x = 3, the output changes about 6 times as fast as the input.

The tangent line predicts nearby values

A slope is useful because it predicts. If you know a function's value at one point and its slope there, you can estimate its value a small step away without computing the function again. Training does exactly this: it uses the slope at the current weights to predict which small change lowers the loss.

The prediction comes straight out of the definition, in three moves.

  1. For a small but nonzero hh, the fraction is close to the number it settles on:

    f(x+h)−f(x)h≈f′(x).\frac{f(x + h) - f(x)}{h} \approx f'(x).
  2. Multiply both sides by hh:

    f(x+h)−f(x)≈f′(x) h.f(x + h) - f(x) \approx f'(x)\,h.
  3. Add f(x)f(x) to both sides:

    f(x+h)≈f(x)+f′(x) h.f(x + h) \approx f(x) + f'(x)\,h.

In words: start at the height f(x)f(x) and climb f′(x)f'(x) for every unit of step. That is the tangent line, used as a forecast.

How good is the forecast? For x2x^2 at 33 the line predicts 9+6h9 + 6h, and the truth is 9+6h+h29 + 6h + h^2, so the error is exactly h2h^2.

step hhtrue f(3+h)f(3 + h)line 9+6h9 + 6herror
0.10.19.619.619.69.60.010.01
0.010.019.06019.06019.069.060.00010.0001

Shrink the step 10 times and the error shrinks 100 times. That is the precise sense in which the tangent is the best line through the point. Any other line through (3,9)(3,\allowbreak 9), with some slope mm, predicts 9+mh9 + mh. Its error is the truth minus the prediction:

(9+6h+h2)−(9+mh)=(6−m)h+h2.(9 + 6h + h^2) - (9 + mh) = (6 - m)h + h^2.

For a small hh the term (6−m)h(6 - m)h is far bigger than h2h^2, so this error shrinks only as fast as hh: halve the step and the error only halves. The slope m=6m = 6 is the one choice that removes that term, leaving an error of h2h^2, which shrinks faster than the step itself: the error divided by the step, h2/h=hh^2 / h = h, goes to zero.

That property is the definition worth keeping. Written out, ff is differentiable at xx when some number f′(x)f'(x) makes

f(x+h)=f(x)+f′(x) h+r(h),with r(h)h→0 as h→0.f(x + h) = f(x) + f'(x)\,h + r(h), \qquad \text{with } \frac{r(h)}{h} \to 0 \text{ as } h \to 0.

Here r(h)r(h) is the leftover error. This version generalizes without change: in the gradients lesson f′(x)f'(x) becomes a vector, and in Jacobians and the chain rule a matrix.

The same pattern holds for curves that are not polynomials. The code below measures the error of the tangent line for sin⁡\sin at x=1x = 1 at three step sizes, and divides each error by h2h^2 to test whether it shrinks like h2h^2:

⌘+Enter runs · edit freelyPython sleeps until you run code

The error falls a hundredfold per row, and the ratio settles near −0.42-0.42; the next lesson explains exactly where that constant comes from.

A derivative has units: output per unit of input. That makes it a sensitivity you can use for quick estimates. Take f(x)=xf(x) = \sqrt{x}, whose derivative is f′(x)=12xf'(x) = \frac{1}{2\sqrt{x}} (the next section shows why), and estimate 4.1\sqrt{4.1} from the nearby point x=4x = 4, where the square root is easy.

  1. The value there: f(4)=4=2f(4) = \sqrt{4} = 2.
  2. The slope there: f′(4)=12×2=14f'(4) = \frac{1}{2 \times 2} = \frac{1}{4}.
  3. The step: h=4.1−4=0.1h = 4.1 - 4 = 0.1.
  4. The forecast: f(4.1)≈2+14×0.1=2+0.025=2.025f(4.1) \approx 2 + \frac{1}{4} \times 0.1 = 2 + 0.025 = 2.025.

The true value is 2.024852.02485 to five decimal places, so the forecast is off by about 0.000150.00015.

So far. Zoom in on a smooth curve and it becomes a straight line, its tangent. The tangent's slope is the derivative. The tangent predicts nearby values, f(x+h)≈f(x)+f′(x) hf(x + h) \approx f(x) + f'(x)\,h, with an error that shrinks faster than the step.

Now make a forecast of your own. The next exercise uses the tangent at x=2x = 2 to estimate a cube just below it.

Work it outEstimate with the tangent line

The derivative of f(x)=x3f(x) = x^3 is f′(x)=3x2f'(x) = 3x^2 (the power rule, in the next section). Use the tangent line at x=2x = 2 to estimate 1.931.9^3. Enter the estimate, not the exact value, as a decimal.

estimate of 1.9³

Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.

The rules, each with a reason

Computing a limit for every function would be slow. A handful of rules build every derivative you need from a few known ones. Each rule has a one-line reason, and the reasons matter more than the rules: they are the same reasons in many dimensions, in the next two lessons.

Sums and constant multiples. (af+bg)′=af′+bg′(a f + b g)' = a f' + b g' for constants aa and bb. Near xx, ff and gg each act like lines, so af+bga f + b g does too, and slopes of lines scale and add. Differentiation is linear, just as the dot product was in module 1.

Powers. (xn)′=nxn−1(x^n)' = n x^{n-1}. Take n=3n = 3 and multiply out the nudged value:

(x+h)3=x3+3x2h+3xh2+h3.(x + h)^3 = x^3 + 3x^2 h + 3x h^2 + h^3.

The first term is the old value and the second is 3x23x^2 times the step, so 3x23x^2 is the slope. The last two terms contain h2h^2 or h3h^3 and vanish faster than hh, so they do not change the slope. For any whole number nn the same happens. (x+h)n(x + h)^n is nn brackets (x+h)(x + h) multiplied together, and multiplying out means picking xx or hh from each bracket in every possible way and adding up the products. Picking xx every time gives xnx^n. Picking hh from exactly one bracket and xx from the other n−1n - 1 gives xn−1hx^{n-1}h, and there are nn brackets to pick that hh from, so together these give nxn−1hn x^{n-1} h. Every other way picks hh at least twice, giving terms with h2h^2 or higher powers. So the slope of xnx^n is nxn−1n x^{n-1}. The rule holds for any real exponent (the logarithm below shows why): x=x1/2\sqrt{x} = x^{1/2} has derivative 12x−1/2=12x\tfrac{1}{2} x^{-1/2} = \frac{1}{2\sqrt{x}}, and 1/x=x−11/x = x^{-1} has derivative −x−2-x^{-2}.

Products. (fg)′=f′g+fg′(f g)' = f' g + f g'. Picture f(x) g(x)f(x)\,g(x) as the area of a rectangle with sides ff and gg. Nudge xx by hh: the sides grow to about f+f′hf + f'h and g+g′hg + g'h. Multiply out the new area:

(f+f′h)(g+g′h)=fg+f′g h+fg′ h+f′g′ h2.(f + f'h)(g + g'h) = fg + f'g\,h + fg'\,h + f'g'\,h^2.

The first term is the old area. The next two are thin strips along two sides of the rectangle: f′g hf'g\,h is a strip f′hf'h wide running along the side of length gg, and fg′ hfg'\,h is a strip g′hg'h tall running along the side of length ff. The last is a tiny corner, f′hf'h by g′hg'h, that vanishes faster than hh. So the area grows by about (f′g+fg′) h(f'g + fg')\,h, and its slope is f′g+fg′f'g + fg'.

A rectangle, wider than it is tall, growing a little. A thin strip runs the full height of its right edge, another runs the full width of its top edge, and a very small rectangle fills the top-right corner where the two strips meet.A product grows by two thin strips, plus a corner too small to matter.

Composition (the chain rule). Write f∘gf \circ g for "apply gg, then ff", the function x↦f(g(x))x \mapsto f(g(x)). Then (f∘g)′(x)=f′(g(x)) g′(x)(f \circ g)'(x) = f'(g(x))\,g'(x).

The chain rule is the one that matters most, and the zoom picture explains it. Near xx, the inner function gg acts like multiplication by its slope: a small nudge hh at xx becomes a nudge of about g′(x) hg'(x)\,h at g(x)g(x). Near g(x)g(x), the outer function ff acts like multiplication by its own slope there, f′(g(x))f'(g(x)). Two stretches in a row multiply: if gg stretches small nudges by 3 and ff stretches them by 8, a nudge that passes through both is stretched by 24.

Go slower: The chain rule from two linear approximations

Let u=g(x)u = g(x) be the point where ff receives its input.

Step 1, the inner function. For a small step hh, g(x+h)≈g(x)+g′(x) h=u+kg(x + h) \approx g(x) + g'(x)\,h = u + k, where k=g′(x) hk = g'(x)\,h is the nudge that arrives at ff.

Step 2, the outer function. For the small step kk, f(u+k)≈f(u)+f′(u) kf(u + k) \approx f(u) + f'(u)\,k.

Step 3, substitute. Replace uu by g(x)g(x) and kk by g′(x) hg'(x)\,h:

f(g(x+h))≈f(g(x))+f′(g(x)) g′(x) h.f(g(x + h)) \approx f(g(x)) + f'(g(x))\,g'(x)\,h.

Step 4, read off the slope. This has the form "value plus slope times hh", so the slope of f∘gf \circ g at xx is f′(g(x)) g′(x)f'(g(x))\,g'(x).

Two errors were dropped along the way: the inner approximation's and the outer one's. Each shrinks faster than its own step, and the step that reaches ff is at most a fixed multiple of hh, so the combined error also shrinks faster than hh. That is what turns the picture into a proof.

Notice where f′f' is evaluated: at g(x)g(x), where ff actually receives its input, not at xx. Getting this wrong is the most common chain-rule slip.

A worked example: y=(3x+1)2y = (3x + 1)^2 at x=1x = 1.

  1. Split it. The inner function is u=3x+1u = 3x + 1 and the outer function is y=u2y = u^2.
  2. Find where the outer function receives its input. At x=1x = 1, u=3(1)+1=4u = 3(1) + 1 = 4.
  3. Inner slope. dudx=3\frac{du}{dx} = 3.
  4. Outer slope, at u=4u = 4. dydu=2u=2×4=8\frac{dy}{du} = 2u = 2 \times 4 = 8.
  5. Multiply. dydx=8×3=24\frac{dy}{dx} = 8 \times 3 = 24.

Check by expanding first: (3x+1)2=9x2+6x+1(3x + 1)^2 = 9x^2 + 6x + 1, whose derivative is 18x+618x + 6, and 18(1)+6=2418(1) + 6 = 24.

The notation dydx\frac{dy}{dx} ("the derivative of yy with respect to xx") makes the rule look like fractions cancelling: dydx=dydu dudx\frac{dy}{dx} = \frac{dy}{du}\,\frac{du}{dx}. That is a good memory aid, not a proof.

This rule is the seed of backpropagation. A neural network is a long composition of functions, so its derivative is a long product of local slopes. In the Jacobians lesson the slopes become Jacobian matrices and the product becomes matrix multiplication, the same composition-as-multiplication you met in module 2. In module 6 that product is backprop.

The four rules in one line each. Slopes of sums add. xnx^n has slope nxn−1n x^{n-1}. A product's slope is two strips, f′g+fg′f'g + fg'. A composition's slope is the product of the slopes, with the outer slope taken where the outer function receives its input.

Practice the chain rule on two composites. The second has three layers, which is the shape of every network: find each layer's slope where that layer receives its input, then multiply.

On paperThe chain rule on two composites

Differentiate each function with the chain rule. Name the inner and outer functions, find each slope where its function receives its input, then evaluate at the given point.

  1. f(x)=(x2+1)3f(x) = (x^2 + 1)^3 at x=1x = 1.
  2. g(x)=ln⁡(1+e2x)g(x) = \ln(1 + e^{2x}) at x=0x = 0. This one has three layers: v=2xv = 2x, then w=1+evw = 1 + e^{v}, then ln⁡w\ln w.

To check your work, enter f′(1)f'(1) and then g′(0)g'(0), both whole numbers.

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

f'(1) and g'(0)

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

The derivatives a network needs

A network is assembled from a few functions: exponentials and logarithms inside its losses, and squashing functions between its layers. Here are their derivatives, each built from the rules above.

The exponential. exe^x is its own derivative. Here is why, one move at a time.

  1. The exponent rule splits the nudged value: ex+h=ex⋅ehe^{x + h} = e^x \cdot e^h.
  2. The number e≈2.71828e \approx 2.71828 is exactly the base whose graph crosses the vertical axis with slope 1. So the tangent line of ehe^h at h=0h = 0 starts at height e0=1e^0 = 1 and climbs with slope 1, and for small hh, eh≈1+1⋅h=1+he^h \approx 1 + 1 \cdot h = 1 + h. Check: e0.001=1.0010005…e^{0.001} = 1.0010005\ldots, which is 1+0.0011 + 0.001 plus a leftover of about 0.00000050.0000005.
  3. Substitute: ex+h≈ex(1+h)=ex+ex he^{x + h} \approx e^x (1 + h) = e^x + e^x\,h.
  4. This has the form "value plus slope times hh", so the slope is exe^x.

The chain rule with inner function −x-x (slope −1-1) gives (e−x)′=e−x⋅(−1)=−e−x(e^{-x})' = e^{-x} \cdot (-1) = -e^{-x}, which we need in a moment.

The logarithm. The natural logarithm ln⁡x\ln x undoes the exponential: eln⁡x=xe^{\ln x} = x for x>0x > 0. Differentiate both sides of that equation.

  1. The left side is a composition: outer function eue^u (slope eue^u), inner function u=ln⁡xu = \ln x. The chain rule gives eln⁡x⋅(ln⁡x)′e^{\ln x} \cdot (\ln x)'. The right side, xx, has slope 11.
  2. Replace eln⁡xe^{\ln x} by xx: the equation reads x⋅(ln⁡x)′=1x \cdot (\ln x)' = 1.
  3. Divide by xx: (ln⁡x)′=1/x(\ln x)' = 1/x.

Logarithms appear in every loss you will train with, starting with cross-entropy in module 6.

The logarithm also settles the power rule for any real exponent nn. For x>0x > 0, write xn=enln⁡xx^n = e^{n \ln x}. The outer function eue^u has slope eue^u and the inner function u=nln⁡xu = n \ln x has slope n/xn/x, so the chain rule gives (xn)′=enln⁡x⋅nx=xn⋅nx=nxn−1(x^n)' = e^{n \ln x} \cdot \frac{n}{x} = x^n \cdot \frac{n}{x} = n x^{n-1}. With n=12n = \tfrac{1}{2} this is the square root's slope, 12x\frac{1}{2\sqrt{x}}, used for 4.1\sqrt{4.1} above.

The sigmoid. σ(x)=11+e−x\sigma(x) = \frac{1}{1 + e^{-x}} squashes any number into the interval from 0 to 1. (The letter σ\sigma has already named a standard deviation in module 0 and the singular values σ1,σ2,…\sigma_1,\allowbreak \sigma_2,\allowbreak \ldots in module 4; used as a function without a subscript, as in σ(x)\sigma(x), it is always the sigmoid.) It is the output of logistic regression and the gate in every LSTM. Its derivative has a compact form:

σ′(x)=σ(x) (1−σ(x)).\sigma'(x) = \sigma(x)\,\bigl(1 - \sigma(x)\bigr).
Go slower: Deriving the sigmoid's derivative

Step 1, rewrite as a power. σ(x)=(1+e−x)−1\sigma(x) = (1 + e^{-x})^{-1}. The outer function is v↦v−1v \mapsto v^{-1} and the inner function is v=1+e−xv = 1 + e^{-x}.

Step 2, the two slopes. The outer slope is −v−2-v^{-2} (power rule). The inner slope is (1+e−x)′=0+(−e−x)=−e−x(1 + e^{-x})' = 0 + (-e^{-x}) = -e^{-x}, using the slope of e−xe^{-x} from above.

Step 3, multiply (chain rule). Evaluate the outer slope at v=1+e−xv = 1 + e^{-x}:

σ′(x)=−(1+e−x)−2⋅(−e−x).\sigma'(x) = -(1 + e^{-x})^{-2} \cdot (-e^{-x}).

Step 4, tidy up. The two minus signs cancel, and a power of −2-2 means one over the square:

σ′(x)=e−x(1+e−x)2.\sigma'(x) = \frac{e^{-x}}{(1 + e^{-x})^2}.

Step 5, split into two factors.

e−x(1+e−x)2=11+e−x⋅e−x1+e−x.\frac{e^{-x}}{(1 + e^{-x})^2} = \frac{1}{1 + e^{-x}} \cdot \frac{e^{-x}}{1 + e^{-x}}.

The first factor is σ(x)\sigma(x).

Step 6, recognize the second factor. Add and subtract 1 in its numerator, then split the fraction:

e−x1+e−x=(1+e−x)−11+e−x=1−11+e−x=1−σ(x).\frac{e^{-x}}{1 + e^{-x}} = \frac{(1 + e^{-x}) - 1}{1 + e^{-x}} = 1 - \frac{1}{1 + e^{-x}} = 1 - \sigma(x).

Step 7, combine. σ′(x)=σ(x) (1−σ(x))\sigma'(x) = \sigma(x)\,(1 - \sigma(x)).

Two consequences. First, once the forward pass has computed s=σ(x)s = \sigma(x), the derivative costs one multiplication, s(1−s)s(1 - s). Backprop reuses forward values like this all the time. Second, the slope is never larger than 0.250.25. The product s(1−s)s(1 - s) is zero at s=0s = 0 and at s=1s = 1 and is a downward parabola in between, so its peak sits halfway, at s=12s = \tfrac{1}{2}, where it equals 12×12=0.25\tfrac{1}{2} \times \tfrac{1}{2} = 0.25. The sigmoid outputs 12\tfrac{1}{2} at x=0x = 0, because σ(0)=11+e0=12\sigma(0) = \frac{1}{1 + e^{0}} = \frac{1}{2}. So σ′(x)≤σ′(0)=0.25\sigma'(x) \le \sigma'(0) = 0.25 everywhere.

The hyperbolic tangent. tanh⁡x=ex−e−xex+e−x\tanh x = \frac{e^x - e^{-x}}{e^x + e^{-x}} squashes into the interval from −1-1 to 11 and is centered at zero. It is the nonlinearity of plain recurrent networks and appears twice inside an LSTM. Its derivative is

tanh⁡′(x)=1−tanh⁡2x,\tanh'(x) = 1 - \tanh^2 x,

which is 11 at x=0x = 0. Again the forward value is all you need: if a=tanh⁡xa = \tanh x, the slope is 1−a21 - a^2.

Go slower: Why tanh' = 1 - tanh²

Step 1, tanh is a stretched sigmoid. Start from 2σ(2x)−12\sigma(2x) - 1, write the sigmoid out, and put everything over one denominator:

2σ(2x)−1=21+e−2x−1=2−(1+e−2x)1+e−2x=1−e−2x1+e−2x.2\sigma(2x) - 1 = \frac{2}{1 + e^{-2x}} - 1 = \frac{2 - (1 + e^{-2x})}{1 + e^{-2x}} = \frac{1 - e^{-2x}}{1 + e^{-2x}}.

Multiply top and bottom by exe^x to get ex−e−xex+e−x=tanh⁡x\frac{e^x - e^{-x}}{e^x + e^{-x}} = \tanh x.

Step 2, differentiate with the chain rule. In 2σ(2x)−12\sigma(2x) - 1 the constant −1-1 has slope 0 and the factor 2 stays in front. The piece σ(2x)\sigma(2x) is the sigmoid of the inner function 2x2x, whose slope is 2, so its slope is σ′(2x)⋅2\sigma'(2x) \cdot 2. Together:

tanh⁡′(x)=2⋅σ′(2x)⋅2=4 σ′(2x).\tanh'(x) = 2 \cdot \sigma'(2x) \cdot 2 = 4\,\sigma'(2x).

Step 3, use the sigmoid's rule. Write s=σ(2x)s = \sigma(2x). Then σ′(2x)=s(1−s)\sigma'(2x) = s(1 - s), so tanh⁡′(x)=4s(1−s)\tanh'(x) = 4s(1 - s).

Step 4, compare with 1−tanh⁡2x1 - \tanh^2 x. Since tanh⁡x=2s−1\tanh x = 2s - 1, expand the square and subtract:

1−(2s−1)2=1−(4s2−4s+1)=4s−4s2=4s(1−s).1 - (2s - 1)^2 = 1 - (4s^2 - 4s + 1) = 4s - 4s^2 = 4s(1 - s).

The two agree, so tanh⁡′(x)=1−tanh⁡2x\tanh'(x) = 1 - \tanh^2 x.

ReLU. ReLU(x)=max⁡(0,x)\mathrm{ReLU}(x) = \max(0,\allowbreak x) is the default activation of deep feed-forward networks. Its derivative is 00 for x<0x < 0 and 11 for x>0x > 0. At x=0x = 0 the graph has a corner and no single tangent line: the slope from the left is 0 and from the right is 1. Code must pick a value there, and frameworks pick one of the two (commonly 0). It rarely matters, because an input lands on exactly zero only rarely. What does matter: a unit whose input is negative passes back no gradient at all.

Now compare the three side by side. The widget answers one question: for which inputs does each unit pass a gradient back, and for which does it all but block it? On screen are the sigmoid (teal), tanh (violet) and ReLU (coral), each with its slope drawn as a dashed curve of the same color. A dashed blue line marks the input xx, moved by the handle at the bottom, by tapping the plot or with the xx slider. The table lists each function's value and slope at xx. The thick curve is the focused function, and hatched bands mark where its slope is smaller than 0.050.05.

Here are the same quantities at three inputs, to more digits:

xxσ(x)\sigma(x)σ′(x)\sigma'(x)tanh⁡x\tanh xtanh⁡′(x)\tanh'(x)
000.50.50.250.250011
220.8810.8810.1050.1050.9640.9640.07070.0707
550.99330.99330.00660.00660.999910.999910.000180.00018

Once the input is more than a few units from zero, both curves are nearly flat and their slopes are close to zero. The unit is saturated. Combine that with the chain rule. Apply the sigmoid ten times in a row, σ(σ(⋯σ(x)))\sigma(\sigma(\cdots\sigma(x))), and the chain rule gives a product of ten slopes, each at most 0.250.25. Even in the best case the product is at most 0.2510≈9.5×10−70.25^{10} \approx 9.5 \times 10^{-7}, the slope¹⁰ readout above. In a real network, weights multiply into the chain as well and can partly compensate, but the tendency is real: the signal telling early layers how to change fades as it passes back through saturated units. This is the vanishing-gradient problem in its smallest form. It returns in module 7, where a recurrent network multiplies such factors once per time step, and the LSTM's cell state in module 8 is the fix.

So far. exe^x is its own slope and ln⁡x\ln x has slope 1/x1/x. The sigmoid's slope is σ(1−σ)\sigma(1 - \sigma), at most 0.250.25; tanh's is 1−tanh⁡21 - \tanh^2, at most 11; ReLU's is 00 or 11. All three slopes come from the forward output alone, and the two squashing functions' slopes fade toward zero for large inputs.

Derive the sigmoid's derivative yourself now, without looking back, and then evaluate it at a point where the numbers come out as simple fractions.

On paperDerive the sigmoid's derivative

Without looking back, derive σ′(x)=σ(x) (1−σ(x))\sigma'(x) = \sigma(x)\,(1 - \sigma(x)) for σ(x)=11+e−x\sigma(x) = \frac{1}{1 + e^{-x}}. Then use it:

  1. Check that σ(ln⁡3)=34\sigma(\ln 3) = \frac{3}{4}, then compute σ′(ln⁡3)\sigma'(\ln 3).
  2. Check that tanh⁡(ln⁡3)=0.8\tanh(\ln 3) = 0.8, then compute tanh⁡′(ln⁡3)\tanh'(\ln 3) from tanh⁡′=1−tanh⁡2\tanh' = 1 - \tanh^2.

To check your work, enter σ′(ln⁡3)\sigma'(\ln 3) and then tanh⁡′(ln⁡3)\tanh'(\ln 3), as fractions (such as 5/8) or decimals.

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

σ'(ln 3) and tanh'(ln 3)

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Next, use those slopes to compare how much gradient four different units pass back.

Quick checkWhich unit passes back the least?

Four units each receive an upstream gradient of 11 from the layer above. By the chain rule, each passes back 11 times its own slope at its input. Which one passes back the least?

Choose one answer, then check.

Then put the chain rule and the sigmoid's slope together to explain a classic training failure.

In your own wordsWhy deep sigmoid stacks learn slowly

A colleague builds a 10-layer network with a sigmoid after every layer. During training the last layers learn well, but the first layers barely change. Explain why, using the chain rule and the sigmoid's derivative, and suggest what to change.

A few sentences first: 0 of 60 characters.

Saved in this browser as you type.

For more practice with the rules and these derivatives, work through a few generated problems. Each comes with a worked solution.

Practice setDerivative reps
3 correct in a row completes the set. A miss starts the count again.

Let f(x)=(−x)4f(x) = (-x)^{4}. Find f′(1)f'(1).

f′(1)

Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.

You now have derivatives by formula. The next lesson measures them by experiment instead, which is how every formula in this course gets checked.

Next: Derivatives by experiment