Derivatives by experiment

In the last lesson you derived $\sigma'(x) = \sigma(x)\,(1 - \sigma(x))$ in seven lines of algebra. Suppose one of those lines had dropped a minus sign. The result would still look like a reasonable formula, and code built on it would still run. In module 6 you will write backward passes with dozens of such formulas, and a network trained with a wrong gradient often still learns a little, so a falling loss proves nothing.

You need a check that cannot share your mistakes. The definition of the derivative gives one: forget the formula, pick a small step, and measure the slope directly. This lesson makes that measurement precise in three steps: two ways to estimate a slope, where each one's error comes from, and why a step that is too small makes things worse.

#Two ways to estimate a slope

The definition says the derivative is the value the secant's slope settles on as the step shrinks. So pick one small $h$ and compute the fraction. This is a finite difference, and comparing it with your formula is called gradient checking. There are two common versions:

The forward difference is the slope of the secant from $x$ to $x + h$. The central difference is the slope of the secant from $x - h$ to $x + h$, which straddles the point; its run is $2h$.

Try both on $f(x) = x^3$ at $x = 2$, where the true derivative is $3x^2 = 3 \times 4 = 12$, with $h = 0.1$. You need three values: $f(2.1) = 2.1^3 = 9.261$, $f(2) = 8$ and $f(1.9) = 1.9^3 = 6.859$.

• Forward: $\frac{9.261 - 8}{0.1} = \frac{1.261}{0.1} = 12.61$, an error of $12.61 - 12 = 0.61$.
• Central: $\frac{9.261 - 6.859}{0.2} = \frac{2.402}{0.2} = 12.01$, an error of $12.01 - 12 = 0.01$.

The widget runs the same comparison for every step size. It answers: how does each estimate's error change as $h$ shrinks? On screen are the white curve $x^3$, the blue point at $x_0 = 2$ with its amber tangent (slope 12), a coral dot at $x_0 + h$ with the coral forward secant through it, and a violet dot at $x_0 - h$ with the dashed violet line of the central difference. The readouts give both estimates, with their errors underneath. Under the $h$ slider, a chart plots each estimate's error against $h$, both on log scales: a label of $-4$ on either axis means $10^{-4}$.

#Where the error comes from

Both estimates use two evaluations of $f$, yet at $h = 0.1$ the central one is about sixty times more accurate ($0.61 / 0.01 = 61$). Why? The answer comes from extending the tangent line with more terms. Taylor's formula says that for a smooth function

where $f''$ is the derivative of $f'$ (the second derivative, which measures curvature: how fast the slope itself changes) and $f'''$ is the third. The first two terms are the tangent line; each further term adds a correction that is one power of $h$ smaller. The factors $\tfrac{1}{2}$ and $\tfrac{1}{6}$ are what make the right side bend like $f$: differentiate $\tfrac{1}{2} f''(x)\,h^2$ twice with respect to $h$ and you get back $f''(x)$, and differentiate $\tfrac{1}{6} f'''(x)\,h^3$ three times and you get back $f'''(x)$.

Check it on $x^3$ at $x = 2$, one derivative at a time. The value is $f(2) = 8$. The slope is $f'(x) = 3x^2$, so $f'(2) = 12$. Differentiating $3x^2$ once more gives $f''(x) = 6x$, so $f''(2) = 12$. Differentiating $6x$ gives $f'''(x) = 6$. The coefficients are then $12$, $\tfrac{1}{2} \times 12 = 6$ and $\tfrac{1}{6} \times 6 = 1$, so the formula gives $8 + 12h + 6h^2 + h^3$, which is exactly $(2 + h)^3$ multiplied out. (The fourth derivative of $x^3$ is 0, so for this function the formula stops there.)

Taylor's formula also explains a number from the last lesson. There, the tangent line's error for $\sin$ at $x = 1$, divided by $h^2$, settled near $-0.42$. The tangent line keeps the first two terms of the formula, so its error starts with the third, $\tfrac{1}{2} f''(1)\,h^2$. For $\sin$, the slope is $\cos$ and the slope of $\cos$ is $-\sin$, so $f'' = -\sin$. That term is then $-\tfrac{1}{2}\sin(1)\,h^2 \approx -\tfrac{1}{2} \times 0.8415\,h^2 \approx -0.42\,h^2$.

So halving $h$ halves the forward error and quarters the central error. On the widget's log-log chart that shows as the coral curve falling one unit and the violet curve two units for every tenfold drop in $h$.

So far. Taylor's formula says the forward difference's error is about $\tfrac{1}{2}f''(x)\,h$ and the central difference's about $\tfrac{1}{6}f'''(x)\,h^2$. The central one wins because it straddles the point, so the curvature terms cancel.

Try one central difference by hand, then compare its error with the formula from the slow box.

#Why a tiny step fails

Both errors shrink as $h$ shrinks, which suggests making $h$ as small as possible. That is wrong, and the reason is the way computers store numbers.

A standard 64-bit float keeps about 16 significant decimal digits. The gap between $1$ and the next number a float can represent is machine epsilon, $\varepsilon \approx 2.2 \times 10^{-16}$. When $h$ is tiny, $f(x + h)$ and $f(x)$ agree in most of their leading digits. Subtracting them cancels those digits and leaves mostly rounding noise, of size roughly $\varepsilon\,|f(x)|$. Dividing by $h$ then magnifies that noise by $1/h$. The total error is a truncation part that shrinks as $h$ shrinks plus a roundoff part that grows like $1/h$, and the best $h$ is roughly where the two are equal. Take a function whose values and derivatives are of moderate size, around 1, so the constants can be dropped:

• Forward: truncation about $h$, roundoff about $\varepsilon/h$. They are equal when $h = \varepsilon / h$. Multiply both sides by $h$: $h^2 = \varepsilon$, so the best step is near $\sqrt{\varepsilon} \approx 1.5 \times 10^{-8}$, and the error there is about $h = \sqrt{\varepsilon} \approx 1.5 \times 10^{-8}$.
• Central: truncation about $h^2$, roundoff about $\varepsilon/h$. They are equal when $h^2 = \varepsilon / h$. Multiply both sides by $h$: $h^3 = \varepsilon$, so the best step is near $\varepsilon^{1/3} \approx 6 \times 10^{-6}$, and the error there is about $h^2 = \varepsilon^{2/3} \approx 4 \times 10^{-11}$, hundreds of times smaller than the forward difference's best.

The code below tests those predictions on $e^x$ at $x = 1$, whose exact derivative is $e \approx 2.718$. It prints both errors for $h = 10^{-1}$ down to $10^{-14}$ and plots them against $k$, where $h = 10^{-k}$:

Both curves fall, reach a bottom and climb again. The forward error falls tenfold with each row and bottoms out at about $7 \times 10^{-9}$ at $h = 10^{-8}$. The central error falls a hundredfold with each row, reaches about $6 \times 10^{-11}$ at $h = 10^{-5}$, wobbles at that level down to $h = 10^{-7}$ (roundoff is partly luck), and from $h = 10^{-8}$ on it climbs. On the left of each bottom (large $h$), truncation dominates; on the right (small $h$), roundoff does. This is why the course's numerical_gradient helper uses a central difference with $h = 10^{-5}$, and why a gradient check compares relative error against a small threshold rather than expecting exact agreement.

In short. A central difference has an error proportional to $h^2$ and a forward difference an error proportional to $h$, while rounding adds an error that grows like $1/h$. So a gradient check uses a central difference with $h$ near $10^{-5}$ and compares against a tolerance, never expecting exact agreement.

Now build the tools yourself. The last function finds the best step size by experiment, and the chart it draws should reproduce the bottoms you just saw.