Why gradients survive

The previous lesson ended with a four-step figure. Going forward, the LSTM's cell held a stored 1 almost perfectly, $0.9991,\allowbreak 0.9982,\allowbreak 0.9973,\allowbreak 0.9964$, while a plain RNN reading the same input faded, $0.7616,\allowbreak 0.5436,\allowbreak 0.4094,\allowbreak 0.3163$.

Holding a value is only half the story, though. A network has to learn to hold it, and learning runs backwards: the loss is computed at the end, at step 4, and the gradient that tells step 1's weights to store the value has to travel back through steps 4, 3 and 2 to get there. So this lesson asks one question: when a gradient travels backwards, how much of it survives each step? We answer it with numbers, first for the plain RNN and then along the LSTM's cell state, then for many steps at once, and last with an honest list of what the answer does not promise.

#One step back through a plain RNN

Start with the plain RNN, because it is the thing to beat. Take the scalar RNN from the figure, with recurrent weight $w = 0.8$:

$$a_t = w\,h_{t-1} + x_t, \qquad h_t = \tanh(a_t).$$

It reads $x = 1,\allowbreak 0,\allowbreak 0,\allowbreak 0$ from $h_0 = 0$. Going forward, one step at a time:

Now go one step back. How does $h_t$ change when $h_{t-1}$ changes? The chain rule takes two moves. First, $h_t = \tanh(a_t)$, and the slope of tanh is $1 - \tanh^2$, so $\partial h_t / \partial a_t = 1 - h_t^2$. Second, $a_t = w\,h_{t-1} + x_t$, so $\partial a_t / \partial h_{t-1} = w$. Multiply the two:

$$\frac{\partial h_t}{\partial h_{t-1}} = (1 - h_t^2)\,w.$$

A gradient going backwards from $h_t$ to $h_{t-1}$ is multiplied by this factor. From $h_4$ back to $h_1$ there are three such steps:

• from $h_4$ to $h_3$: $0.8\,(1 - 0.1000) = 0.8 \times 0.9000 = 0.7200$
• from $h_3$ to $h_2$: $0.8\,(1 - 0.1676) = 0.8 \times 0.8324 = 0.6659$
• from $h_2$ to $h_1$: $0.8\,(1 - 0.2955) = 0.8 \times 0.7045 = 0.5636$

Together: $0.7200 \times 0.6659 = 0.4794$, then $0.4794 \times 0.5636 = 0.2702$. After only three steps, 73% of the gradient is gone.

The same holds with vectors. For $\mathbf{h}_t = \tanh(\mathbf{W}_{hh}\mathbf{h}_{t-1} + \mathbf{W}_{xh}\mathbf{x}_t + \mathbf{b})$ the Jacobian of one step is $\partial \mathbf{h}_t / \partial \mathbf{h}_{t-1} = \mathrm{diag}(1 - \mathbf{h}_t^2)\,\mathbf{W}_{hh}$. Here $\mathrm{diag}(\mathbf{v})$ is the diagonal matrix with the entries of $\mathbf{v}$ on its diagonal, and $1 - \mathbf{h}_t^2$ (entry by entry) holds the tanh slopes. Going backwards, as in backpropagation through time, the gradient is multiplied by the transpose of that Jacobian at every step:

$$\frac{\partial L}{\partial \mathbf{h}_{t-1}} = \mathbf{W}_{hh}^\top\left(\frac{\partial L}{\partial \mathbf{h}_t} \odot (1 - \mathbf{h}_t^2)\right).$$

Two facts turn the transpose into this form. The transpose of a product reverses the order, and a diagonal matrix is its own transpose, so $\big(\mathrm{diag}(1 - \mathbf{h}_t^2)\,\mathbf{W}_{hh}\big)^\top = \mathbf{W}_{hh}^\top\,\mathrm{diag}(1 - \mathbf{h}_t^2)$. And multiplying a vector by a diagonal matrix scales each entry by the matching diagonal entry, which is the $\odot$ inside the brackets.

After $k$ steps the gradient has been multiplied by $k$ such matrices. Each one changes its length by at most the largest singular value of $\mathbf{W}_{hh}$ times the largest tanh slope (module 4), so it tends to shrink or grow geometrically, like the powers of a matrix in powers and stability and the product of Jacobians in module 7. And the network cannot freely tune that factor to 1, for two reasons. $\mathbf{W}_{hh}$ is also the matrix that does all of its computing. And the tanh slope $1 - h^2$ equals 1 only at $h = 0$: a unit that holds a large value, which is what storing something means, sits where the tanh is flatter and passes back less of its gradient. You can see it in the list above: the smallest of the three factors, $0.5636$, belongs to step 2, where the state $h_2 = 0.5436$ was the largest of the three states involved.

#One step back along the cell state

Now the same question for the LSTM. Its memory update is

$$c_t = f_t\,c_{t-1} + i_t\,g_t.$$

Ask how $c_t$ moves when only $c_{t-1}$ moves. The gates $f_t$, $i_t$ and the candidate $g_t$ are computed from $x_t$ and $h_{t-1}$, not from $c_{t-1}$, so along this direct route they are constants. The second term, $i_t g_t$, does not contain $c_{t-1}$ at all, so its slope is 0. The first term is the constant $f_t$ times $c_{t-1}$, so its slope is $f_t$. Adding the two slopes:

$$\frac{\partial c_t}{\partial c_{t-1}} = f_t + 0 = f_t, \qquad \frac{\partial L}{\partial c_{t-1}} = f_t\,\frac{\partial L}{\partial c_t}.$$

One step back along the cell state multiplies the gradient by the forget gate, and by nothing else: no weight matrix, no tanh slope. In the four-step figure the forget gate is $0.9991$ at every step, so from $c_4$ back to $c_1$:

• from $c_4$ to $c_3$: $\times\, 0.9991$
• from $c_4$ to $c_2$: $0.9991 \times 0.9991 = 0.9982$
• from $c_4$ to $c_1$: $0.9982 \times 0.9991 = 0.9973$

The RNN kept $0.2702$ over the same three steps; the cell path keeps $0.9973$. The figure from the previous lesson, opened at its last step, puts the two side by side.

With $n$ units the same argument runs entry by entry. Unit $j$'s memory $c_{t,j}$ depends only on its own $c_{t-1,j}$, with slope $f_{t,j}$, so along this direct path

$$\frac{\partial \mathbf{c}_t}{\partial \mathbf{c}_{t-1}} = \mathrm{diag}(\mathbf{f}_t), \qquad \frac{\partial L}{\partial \mathbf{c}_{t-1}} = \mathbf{f}_t \odot \frac{\partial L}{\partial \mathbf{c}_t}.$$

After $k$ steps back along the cell path, the gradient on unit $j$ has been multiplied by $f_{t,j}\,f_{t-1,j}\cdots f_{t-k+1,j}$: a product of ordinary numbers, one per step, each between 0 and 1. There is no matrix to raise to a power and no mixing between units.

Go slower: The cell-path Jacobian, entry by entry

Step 1: one entry. Write the cell update for unit $j$: $c_{t,j} = f_{t,j}\,c_{t-1,j} + i_{t,j}\,g_{t,j}$. The gates are computed from $\mathbf{x}_t$ and $\mathbf{h}_{t-1}$, not from $\mathbf{c}_{t-1}$. So with $\mathbf{x}_t$ and $\mathbf{h}_{t-1}$ held fixed, $f_{t,j}$, $i_{t,j}$ and $g_{t,j}$ are constants. Differentiate with respect to entry $k$ of $\mathbf{c}_{t-1}$: the term $i_{t,j} g_{t,j}$ gives 0, and the term $f_{t,j} c_{t-1,j}$ gives $f_{t,j}$ when $k = j$ and 0 when $k \ne j$: $\frac{\partial c_{t,j}}{\partial c_{t-1,k}} = \begin{cases} f_{t,j} & \text{if } k = j, \\ 0 & \text{if } k \ne j. \end{cases}$

Step 2: the matrix. Collect these partial derivatives into the Jacobian (row $j$, column $k$). Only the diagonal entries are nonzero, and they are $f_{t,1},\allowbreak \ldots,\allowbreak f_{t,n}$. That matrix is $\mathrm{diag}(\mathbf{f}_t)$.

Step 3: many steps. By the chain rule as matrix multiplication (module 5), the direct path over $k$ steps has the product of the one-step Jacobians: $\mathrm{diag}(\mathbf{f}_t)\,\mathrm{diag}(\mathbf{f}_{t-1})\cdots\mathrm{diag}(\mathbf{f}_{t-k+1}) = \mathrm{diag}(\mathbf{f}_t \odot \mathbf{f}_{t-1} \odot \cdots \odot \mathbf{f}_{t-k+1}).$ The equals sign holds because multiplying diagonal matrices multiplies their diagonals entry by entry: entry $(j,\allowbreak j)$ of the product is $f_{t,j} f_{t-1,j} \cdots f_{t-k+1,j}$, and every off-diagonal entry stays 0.

Step 4: backwards. Backprop multiplies the gradient by the transpose of the Jacobian. A diagonal matrix is its own transpose, so $\partial L/\partial \mathbf{c}_{t-1} = \mathrm{diag}(\mathbf{f}_t)\,\partial L/\partial \mathbf{c}_t$. Multiplying a vector by a diagonal matrix scales each entry by the matching diagonal entry, which is $\mathbf{f}_t \odot \partial L/\partial \mathbf{c}_t$.

Step 5: the RNN for comparison. For $\mathbf{h}_t = \tanh(\mathbf{a}_t)$ with $\mathbf{a}_t = \mathbf{W}_{hh}\mathbf{h}_{t-1} + \mathbf{W}_{xh}\mathbf{x}_t + \mathbf{b}$, the chain rule gives $\partial \mathbf{h}_t/\partial \mathbf{a}_t = \mathrm{diag}(1 - \mathbf{h}_t^2)$ (tanh acts entry by entry) and $\partial \mathbf{a}_t/\partial \mathbf{h}_{t-1} = \mathbf{W}_{hh}$, so $\partial \mathbf{h}_t/\partial \mathbf{h}_{t-1} = \mathrm{diag}(1 - \mathbf{h}_t^2)\,\mathbf{W}_{hh}$: a diagonal matrix of tanh slopes times a full matrix. Its transpose is $\mathbf{W}_{hh}^\top\mathrm{diag}(1 - \mathbf{h}_t^2)$. Since $|\mathbf{A}\mathbf{v}| \le \sigma_1(\mathbf{A})\,|\mathbf{v}|$, where $\sigma_1$ is the largest singular value of module 4 (not the sigmoid), each backward step changes the gradient's length by at most $\sigma_1(\mathbf{W}_{hh}) \cdot \max_j (1 - h_{t,j}^2)$.

In short: one step back through a plain RNN multiplies the gradient by a full matrix and a set of tanh slopes; one step back along the cell state multiplies each unit's gradient by that unit's forget gate.

#Many steps: a product of numbers near 1

Over one step the difference between $0.9991$ and $0.72$ looks modest. Over long gaps it decides everything, because the factors multiply. Say a unit's forget gate sits at $0.97$. Then $0.97^{10} \approx 0.7374$, and fifty steps are five blocks of ten, so $0.97^{50} = (0.97^{10})^5 \approx 0.7374^5 \approx 0.22$ of the gradient survives. An RNN whose per-step factor is $0.8$ has $0.8^{10} \approx 0.1074$ and keeps $0.8^{50} \approx 0.1074^5 \approx 0.000014$. One is a signal the network can learn from; the other is lost among everything else.

The chart below answers the same question for up to 100 steps. The horizontal axis is $k$, the number of steps back. The vertical axis is how much of the gradient is left, on a log scale: the line marked 1 means unchanged, and each labelled grid line is 100,000 times the one below it. On a log scale, shrinking by the same factor at every step draws a straight line. The lime line is a plain RNN with 8 tanh units whose recurrent matrix has every singular value equal to $\rho$ (starting at 1), with the tanh slopes included; the dashed white line is the bound $\rho^k$. The amber line is the LSTM's cell path with every forget gate equal to $\sigma(b_f)$, starting at $b_f = 3$. The table beside it gives each line's factor per step and what is left after 10, 50 and 100 steps.

This is the link from module 4 that the whole part has been building toward. Repeated multiplication by a matrix shrinks or grows along its eigenvectors according to its eigenvalues. A diagonal matrix's eigenvalues are its diagonal entries. Along the cell path the per-step matrix is $\mathrm{diag}(\mathbf{f}_t)$, so its eigenvalues are the forget-gate values themselves, set directly by gates that the network learns. When a unit needs to remember, the network can push its forget gate toward 1 with a large bias, and that unit's product of forget values stays near 1. It cannot explode either, because no forget value exceeds 1.

Because the factors multiply, a few steps with a lower forget value can cost more than many steps close to 1. The exercise below puts numbers on that: follow a gradient by hand through a forget gate that is not the same at every step.

#What the cell path does not promise

It would be easy to overclaim here, so this section states exactly what was shown. The cell state reaches the next step along two routes, not one.

1. The direct route, $\mathbf{c}_{t-1} \to \mathbf{c}_t$, through the forget gate. Its Jacobian is $\mathrm{diag}(\mathbf{f}_t)$.
2. The route through the output. $\mathbf{c}_{t-1}$ also produces $\mathbf{h}_{t-1} = \mathbf{o}_{t-1} \odot \tanh(\mathbf{c}_{t-1})$, and $\mathbf{h}_{t-1}$ feeds the forget gate, the input gate and the candidate at step $t$ through their $\mathbf{U}$ matrices, so it changes $\mathbf{c}_t$ too. (It feeds the output gate as well, but the output gate does not touch $\mathbf{c}_t$.)

The multivariable chain rule adds the two routes. The second route is itself a chain of two moves: from $\mathbf{c}_{t-1}$ to $\mathbf{h}_{t-1}$, whose Jacobian is $\mathrm{diag}\big(\mathbf{o}_{t-1} \odot (1 - \tanh^2(\mathbf{c}_{t-1}))\big)$ because both operations act entry by entry, then from $\mathbf{h}_{t-1}$ to $\mathbf{c}_t$, which we call $\partial \mathbf{c}_t / \partial \mathbf{h}_{t-1}$. So the total derivative is

$$\frac{d\mathbf{c}_t}{d\mathbf{c}_{t-1}} = \mathrm{diag}(\mathbf{f}_t) + \frac{\partial \mathbf{c}_t}{\partial \mathbf{h}_{t-1}}\,\mathrm{diag}\big(\mathbf{o}_{t-1} \odot (1 - \tanh^2(\mathbf{c}_{t-1}))\big).$$

Go slower: What $\partial \mathbf{c}_t / \partial \mathbf{h}_{t-1}$ is made of

Start from $\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \mathbf{g}_t$ and hold $\mathbf{c}_{t-1}$ fixed. Three of the four factors depend on $\mathbf{h}_{t-1}$: $\mathbf{f}_t$, $\mathbf{i}_t$ and $\mathbf{g}_t$.

The product rule, entry by entry. Changing $\mathbf{f}_t$ changes $\mathbf{c}_t$ at the rate $\mathbf{c}_{t-1}$; changing $\mathbf{i}_t$, at the rate $\mathbf{g}_t$; changing $\mathbf{g}_t$, at the rate $\mathbf{i}_t$. So $\partial \mathbf{c}_t/\partial \mathbf{h}_{t-1} = \mathrm{diag}(\mathbf{c}_{t-1})\,\frac{\partial \mathbf{f}_t}{\partial \mathbf{h}_{t-1}} + \mathrm{diag}(\mathbf{g}_t)\,\frac{\partial \mathbf{i}_t}{\partial \mathbf{h}_{t-1}} + \mathrm{diag}(\mathbf{i}_t)\,\frac{\partial \mathbf{g}_t}{\partial \mathbf{h}_{t-1}}$.

Each gate's Jacobian. $\mathbf{f}_t = \sigma(\mathbf{W}_f\mathbf{x}_t + \mathbf{U}_f\mathbf{h}_{t-1} + \mathbf{b}_f)$, so by the chain rule $\partial \mathbf{f}_t/\partial \mathbf{h}_{t-1} = \mathrm{diag}(\mathbf{f}_t \odot (1 - \mathbf{f}_t))\,\mathbf{U}_f$, using the sigmoid's slope $s(1 - s)$. In the same way $\partial \mathbf{i}_t/\partial \mathbf{h}_{t-1} = \mathrm{diag}(\mathbf{i}_t \odot (1 - \mathbf{i}_t))\,\mathbf{U}_i$ and $\partial \mathbf{g}_t/\partial \mathbf{h}_{t-1} = \mathrm{diag}(1 - \mathbf{g}_t^2)\,\mathbf{U}_g$.

Put together. Each of the three terms is a diagonal matrix times a full $\mathbf{U}$ matrix, the same shape as the RNN's $\mathrm{diag}(1 - \mathbf{h}_t^2)\,\mathbf{W}_{hh}$. In the four-step figure every $\mathbf{U}$ is zero, so this route is exactly zero there and $0.9973$ is the whole story. In a trained LSTM the $\mathbf{U}$ matrices are not zero.

That second term is a full matrix built from the $\mathbf{U}$ matrices, and it behaves much like an RNN's Jacobian: it can shrink or grow a gradient geometrically. The full state of an LSTM is the pair $(\mathbf{c}_t,\allowbreak \mathbf{h}_t)$, and its one-step Jacobian is a $2n \times 2n$ block matrix with paths through both. So the precise claim is this: among the many paths back through time, there is one whose per-step factor is a diagonal matrix of gate values that the network can hold near 1. When the forget gates are near 1, that path by itself carries a usable learning signal across long gaps, even if every other path has faded. And when the forget gates are not near 1, as with a bias of 0 in the chart above, the cell path fades like the others.

The original LSTM had no forget gate at all: its cell update was $\mathbf{c}_t = \mathbf{c}_{t-1} + \mathbf{i}_t \odot \mathbf{g}_t$, the accumulator of the previous lesson with an input gate, so the cell-path Jacobian was exactly the identity. That was the constant error carousel. The forget gate gave up that exactness in exchange for the ability to reset, and it hands the network a dial for how long each unit remembers.

To recap: the direct cell path multiplies a gradient by one forget value per unit per step, so it survives exactly as long as those values stay near 1. Other paths exist and behave like an RNN's. Test the distinction on a few tempting explanations.

Last, put the whole argument in your own words. It is the argument the rest of this module relies on.

The next lesson puts the gates to work: you choose all twelve weights of a one-unit cell yourself and make it latch, hold, count and toggle. It ends by returning to this lesson's product of forget values, at the moment when a network starts training.