ml.lab
Python sleeps until you run code
08 LSTMs

Lesson 3 of 6

Designing an LSTM by hand

With large weights every gate becomes an on-off switch, so you can program an LSTM cell like a small circuit. Build a latch, a register, a counter and a toggle by choosing twelve numbers, then count an LSTM's parameters and see why its forget gate should start open.

About 75 minutes
By the end you can
  • Choose weights that make a sigmoid gate open or closed in exactly the situations you want.

  • Design one-unit LSTMs that latch, hold a value, count and toggle, and verify them on input sequences.

  • Count the parameters of an LSTM layer, 4n(n+m)+4n4n(n + m) + 4n, and compare with a plain RNN.

  • Explain why the forget-gate bias starts positive, and predict how much gradient survives kk steps at initialization.

Before training an LSTM, program one. A cell with a single unit has twelve numbers: for each of its four gates, a weight ww on the input xtx_t, a weight uu on the previous output ht−1h_{t-1} and a bias bb. Choose them well and the cell becomes a small circuit with memory: a latch that remembers whether it has ever seen a 1, a register that holds a number, a counter, a toggle switch.

Doing this by hand is the fastest way to learn what each weight can and cannot do. It also makes one fact concrete: gates are programmable logic. Training, which the next two lessons set up, is a search for programs like the ones you are about to write, except that nobody tells the network which program it needs.

Throughout this lesson the cell has one unit, so every quantity is a number:

ft=σ(wfxt+ufht−1+bf),it=σ(wixt+uiht−1+bi),gt=tanh⁡(wgxt+ught−1+bg),ot=σ(woxt+uoht−1+bo),ct=ft ct−1+it gt,ht=ottanh⁡(ct),\begin{aligned} f_t &= \sigma(w_f x_t + u_f h_{t-1} + b_f), & i_t &= \sigma(w_i x_t + u_i h_{t-1} + b_i), \\ g_t &= \tanh(w_g x_t + u_g h_{t-1} + b_g), & o_t &= \sigma(w_o x_t + u_o h_{t-1} + b_o), \\ c_t &= f_t\,c_{t-1} + i_t\,g_t, & h_t &= o_t \tanh(c_t), \end{aligned}

starting from c0=h0=0c_0 = h_0 = 0. These are the equations of the first lesson with m=n=1m = n = 1: ff is the forget gate, ii the input gate, gg the candidate and oo the output gate.

Saturated gates are switches

To design by hand we want gates that are fully open or fully shut, not half open. Far from zero the sigmoid flattens out:

zz−10-10−5-5−2-20022551010
σ(z)\sigma(z)0.000050.00670.11920.50.88080.99330.99995

Check the ends: σ(10)=1/(1+e−10)=1/(1+0.0000454)≈0.99995\sigma(10) = 1/(1 + e^{-10}) = 1/(1 + 0.0000454) \approx 0.99995, and σ(−10)=1/(1+e10)=1/22,027≈0.00005\sigma(-10) = 1/(1 + e^{10}) = 1/22{,}027 \approx 0.00005. So a pre-activation of +10+10 means "open" and −10-10 means "closed", each with an error of about 0.000050.00005. The candidate works the same way with tanh: tanh⁡(10)\tanh(10) is within 10−810^{-8} of 1, and tanh⁡(0)=0\tanh(0) = 0 exactly. (The four-step figure in the first lesson used ±7\pm 7, where the error is σ(−7)≈0.0009\sigma(-7) \approx 0.0009; ±10\pm 10 is a sharper switch.)

That turns weight design into a recipe.

  1. List the situations the cell can be in: the possible inputs, and, if it matters, whether something is already stored.
  2. Decide what each gate should do in each situation: open (1) or closed (0). Decide what the candidate should write: −1-1, 00 or 11.
  3. Choose ww, uu and bb so that the gate's pre-activation is at least +10+10 in the situations where it must be open and at most −10-10 where it must be closed.

Step 3 is drawing a threshold. Say the inputs are 0 and 1 and a gate must open only when xt=1x_t = 1. Try w=20w = 20 and b=−10b = -10:

xtx_tpre-activation 20xt−1020x_t - 10gate
020(0)−10=−1020(0) - 10 = -10σ(−10)≈0.00005\sigma(-10) \approx 0.00005, closed
120(1)−10=+1020(1) - 10 = +10σ(10)≈0.99995\sigma(10) \approx 0.99995, open

For a gate that must open unless xt=−1x_t = -1, among inputs −1-1, 00 and 11, take w=20w = 20 and b=10b = 10: the pre-activations are 20(−1)+10=−1020(-1) + 10 = -10, 20(0)+10=+1020(0) + 10 = +10 and 20(1)+10=+3020(1) + 10 = +30. This is exactly what a single neuron does: it draws a line through its inputs and says which side you are on. A gate is a neuron whose output multiplies something.

One restriction shapes every design: a gate sees only xtx_t and ht−1h_{t-1}. It cannot read the cell state. If a gate needs to know what is stored, it has to read it through ht−1=ot−1tanh⁡(ct−1)h_{t-1} = o_{t-1}\tanh(c_{t-1}), which lies between −1-1 and 11, using its uu weight.

A latch

The first task: the inputs are 0s and 1s, and the output hth_t must switch on (at least 0.5) at the first 1 and stay on for good. Before the first 1 it must stay off (within 0.1 of 0).

The widget is the cell from the first lesson in design mode. It answers: do your twelve weights solve the task? At the top of the controls are the task, its goal, a switch between three test sequences and a verdict that says how many tests pass and where the first failure is. The cell shows one step of the test shown, and the strip chart under it shows the whole test: the inputs as squares (filled for a 1), the output hh as a lime line, the memory cc as a faint amber line, and a band at each step where hh must be (lime where it passes, pink where it fails). Pick a gate with its button (ff, ii, gg, oo) and set its ww, uu and bb with the sliders, or type into the grid of all twelve weights. Every weight starts at 0.

Each setting has a reason. The memory lives in ctc_t, so the forget gate must keep it at every step. Something positive must be written when xt=1x_t = 1 and nothing when xt=0x_t = 0. You could make the input gate do that selection, but the candidate already does it: with bg=0b_g = 0, gt=tanh⁡(wgxt)g_t = \tanh(w_g x_t) is exactly tanh⁡(0)=0\tanh(0) = 0 whenever xt=0x_t = 0, and about 1 when xt=1x_t = 1. So the input gate can stay open. And the output gate has to show the memory. (The widget's weights run from −10-10 to 1010. That is enough for every task it checks, though not for the ±20\pm 20 weights some designs below use; those you can try in code.)

Now check the design on paper, step by step, and find out what the cell is really computing.

On paperDerive and verify a latch

Design a one-unit LSTM latch. The inputs are 0s and 1s; the output hth_t must be at least 0.5 from the first 1 onward and within 0.1 of 0 before it. Use these restrictions: the only weight that looks at xtx_t is wg=10w_g = 10; every uu is 0; each bias is 00, 1010 or −10-10.

  1. For each gate, decide whether it should be open, closed, or depend on the input, and choose the four biases.
  2. Verify your design on the inputs 0,1,0,0,10,\allowbreak 1,\allowbreak 0,\allowbreak 0,\allowbreak 1: compute ctc_t and hth_t for t=1,…,5t = 1,\allowbreak \ldots,\allowbreak 5 from c0=h0=0c_0 = h_0 = 0. Treat σ(10)\sigma(10) as 1, σ(−10)\sigma(-10) as 0 and tanh⁡(10)\tanh(10) as 1.
  3. What does c5c_5 tell you about what this cell really computes? Why does that not break the latch?

The check at the top takes c5c_5 and h5h_5, in that order, to 3 decimal places.

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

c_5 and h_5

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

The latch you just verified is secretly a counter: every 1 adds another 1 to ctc_t. That does not break the latch, because tanh⁡(c)\tanh(c) is at least tanh⁡(1)≈0.76\tanh(1) \approx 0.76 for every c≥1c \ge 1. But there is a second design that keeps ctc_t at 1 instead of counting, and it uses the forget gate differently. When xt=1x_t = 1, erase the cell and write a fresh 1: set, instead of add.

  • Forget: open when xt=0x_t = 0, closed when xt=1x_t = 1: wf=−20w_f = -20, bf=10b_f = 10.
  • Input: open only when xt=1x_t = 1: wi=20w_i = 20, bi=−10b_i = -10.
  • Candidate: always 1: bg=10b_g = 10.
  • Output: always open: bo=10b_o = 10.

Check both situations with the pre-activations:

xtx_tforget: −20xt+10-20x_t + 10input: 20xt−1020x_t - 10candidatect=ftct−1+itgtc_t = f_t c_{t-1} + i_t g_t
0+10+10, open−10-10, closedtanh⁡(10)≈1\tanh(10) \approx 11⋅ct−1+0⋅1=ct−11 \cdot c_{t-1} + 0 \cdot 1 = c_{t-1}
1−10-10, closed+10+10, opentanh⁡(10)≈1\tanh(10) \approx 10⋅ct−1+1⋅1=10 \cdot c_{t-1} + 1 \cdot 1 = 1

At a 1 the cell becomes 1, whatever was there; at a 0 it keeps its value. Two designs, one behavior at the output, different memories. A trained network faces the same freedom, which is one reason its weights are hard to read.

A latch that passes short tests can still fail on long ones. Here is one; find the fix.

Quick checkA leaky latch

You enter a latch with bf=3b_f = 3, bi=10b_i = 10, wg=10w_g = 10, bo=10b_o = 10 and every other weight 0. It passes a test in which a 1 is followed by 10 zeros, but on a longer test hth_t drops below 0.5 at the 13th zero after the 1. Which change fixes it?

Choose one answer, then check.

Hold, count and toggle

The latch stored one bit. The next three tasks store a number, count, and make a decision that depends on the cell's own state.

Hold. Now the first input is a number between −0.7-0.7 and 0.70.7, every later input is 0, and hth_t must stay within 0.1 of that first number at every step. The widget checks three tests, with first inputs 0.60.6, −0.4-0.4 and 0.30.3, and draws each band around that first input.

The structure is the latch's: keep everything (forget open), write whenever the candidate is nonzero (input open), show everything (output open). Since every later input is 0 and bg=0b_g = 0, the candidate is tanh⁡(0)=0\tanh(0) = 0 after the first step, so nothing more is written. On the first step the cell starts empty, so c1=f1⋅0+i1g1≈tanh⁡(wgx1)c_1 = f_1 \cdot 0 + i_1 g_1 \approx \tanh(w_g x_1), and from then on

ht≈tanh⁡(c1)≈tanh⁡(tanh⁡(wgx1)).h_t \approx \tanh(c_1) \approx \tanh(\tanh(w_g x_1)).

The value is squashed twice, once by the candidate's tanh and once at the output. Each tanh pulls a value toward zero (∣tanh⁡(a)∣<∣a∣|\tanh(a)| < |a| for every a≠0a \ne 0), so wgw_g must be above 1 to compensate. Check the widget's answer, wg=1.4w_g = 1.4, on test 1, where x1=0.6x_1 = 0.6: 1.4×0.6=0.841.4 \times 0.6 = 0.84, then tanh⁡(0.84)≈0.6858\tanh(0.84) \approx 0.6858, then tanh⁡(0.6858)≈0.5953\tanh(0.6858) \approx 0.5953. That is the 0.5950.595 the widget showed, off by 0.6−0.5953=0.00470.6 - 0.5953 = 0.0047. With wg=1w_g = 1 the same two squashes give tanh⁡(0.6)≈0.5370\tanh(0.6) \approx 0.5370 and tanh⁡(0.5370)≈0.4908\tanh(0.5370) \approx 0.4908, the failing 0.490.49 of the walkthrough.

How large may wgw_g be? Two inputs set the limits. A large first input gets squashed the most, so it needs wgw_g large enough: at x1=0.7x_1 = 0.7, wg=1.22w_g = 1.22 gives tanh⁡(0.854)≈0.6932\tanh(0.854) \approx 0.6932 and tanh⁡(0.6932)≈0.6000\tanh(0.6932) \approx 0.6000, just within 0.1. A small first input sits where tanh is nearly straight, so a large wgw_g overshoots it: at x1=0.3x_1 = 0.3, wg=1.5w_g = 1.5 gives tanh⁡(0.45)≈0.4219\tanh(0.45) \approx 0.4219 and tanh⁡(0.4219)≈0.3985\tanh(0.4219) \approx 0.3985, just within 0.1 above. Checking every first input in [−0.7,0.7][-0.7,\allowbreak 0.7] shows that any wgw_g from about 1.22 to 1.50 keeps the error at or under 0.1. In a real network nobody needs hth_t to equal a number exactly: the next layer's weights can undo a squashing. The lesson is that hth_t is a bounded, distorted view of ctc_t.

Count. The inputs are 0s and 1s again, and now the cell state ctc_t itself must equal the number of 1s so far, within 0.25. The widget's third test is twelve 1s in a row.

The latch already counts. Each 1 writes g=tanh⁡(wg)g = \tanh(w_g) with the input gate open, and the forget gate keeps the total. The only question is precision: after twelve 1s the cell holds about 12tanh⁡(wg)12\tanh(w_g), which must be at least 12−0.25=11.7512 - 0.25 = 11.75. Divide by 12: tanh⁡(wg)≥11.75/12≈0.9792\tanh(w_g) \ge 11.75/12 \approx 0.9792. Undo the tanh with its inverse, artanh⁡(y)=12ln⁡1+y1−y\operatorname{artanh}(y) = \tfrac{1}{2}\ln\tfrac{1 + y}{1 - y}: wg≥12ln⁡1.97920.0208≈12ln⁡95.2≈2.28w_g \ge \tfrac{1}{2}\ln\tfrac{1.9792}{0.0208} \approx \tfrac{1}{2}\ln 95.2 \approx 2.28. That is why 2.22.2 failed and 2.32.3 passed.

Here the difference between ctc_t and hth_t matters. The cell state is not squashed, so it can hold 12, or 30. The output ht=ottanh⁡(ct)h_t = o_t\tanh(c_t) cannot tell them apart: tanh⁡(12)\tanh(12) and tanh⁡(30)\tanh(30) both equal 1 to nine decimal places, and even tanh⁡(3)≈0.995\tanh(3) \approx 0.995 is almost there. A count is available inside the cell, and a later layer can only read a saturated version of it. Run the counting latch and compare the two: the first printed row is the true count, the second is ctc_t, and the third is hth_t. The cell state climbs to 8 in steps of 1. The output reads 0.7620.762 after the first 1 and 0.9640.964 after the second, then barely moves: 0.9950.995, 0.9990.999, and 1.01.0 (to three decimals) from the fifth 1 on.

⌘+Enter runs · edit freelyPython sleeps until you run code

Toggle. Last, a design the widget does not check: each 1 flips the cell between off and on, and a 0 leaves it alone. This is harder, because what a 1 should do depends on the current state. Run the recipe's first two steps. There are four situations:

xtx_tcell beforeht−1h_{t-1}mustmemory
0offabout 0stay offkeep, write nothing
0onabout 0.760.76stay onkeep, write nothing
1offabout 0turn onwrite a 1
1onabout 0.760.76turn offerase, write nothing

Now look down each gate's column of decisions. The forget gate can decide from xtx_t alone: keep at every 0, erase at every 1 (erasing an off cell costs nothing, since it holds 0). But the input gate must open in the third row and stay shut in the fourth, and those two rows have the same input, xt=1x_t = 1. The only thing that differs is ht−1h_{t-1}: about 0.760.76 when the cell is on (that is tanh⁡(1)\tanh(1)) and about 0 when it is off. So the input gate needs a nonzero uu weight: the cell has to read its own output to decide. That is the first design here in which the recurrence ht−1→h_{t-1} \to gates does real work.

Now build all of these in code, including the toggle, and test them on many random sequences at once. The recipe's third step, choosing the numbers, is yours.

Code itProgram four circuits

Build hand-designed LSTMs in code and test them on many random sequences.

  1. pack(p) turns the 12 named weights (the widget's names: wf, uf, bf, wi, ... , bo) into the stacked W of shape (2,4)(2,\allowbreak 4) and b of shape (4,)(4,\allowbreak ): row 0 of W holds the ww weights, row 1 the uu weights, and the columns are the gates in the order [i,f,g,o][i,\allowbreak f,\allowbreak g,\allowbreak o].
  2. run(p, xs) runs the one-unit cell over the inputs from h=c=0h = c = 0 with the reference lstm_step and returns the arrays of hth_t and ctc_t.
  3. latch(), counter() and toggle() each return 12 weights (use the weights(...) helper) for the tasks in their docstrings. The toggle needs a uu weight: the gates must read the cell's own output to know whether it is on.
⌘+Enter runsPython sleeps until you run code
Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.

Counting parameters

Hand design used 12 numbers. A real layer has many more, and the count tells you what an LSTM costs. Each gate has an n×mn \times m matrix for the input (nmnm numbers), an n×nn \times n matrix for the previous output (n2n^2 numbers) and nn biases: nm+n2+n=n(n+m)+nnm + n^2 + n = n(n + m) + n numbers. The four gates together have

4n(n+m)+4n=4n(n+m+1).4n(n + m) + 4n = 4n(n + m + 1).

In the stacked layout this is the (m+n)×4n(m + n) \times 4n matrix W\mathbf{W} plus the 4n4n entries of b\mathbf{b}: (m+n)⋅4n+4n(m + n) \cdot 4n + 4n, the same count. A plain RNN, ht=tanh⁡(Whhht−1+Wxhxt+b)\mathbf{h}_t = \tanh(\mathbf{W}_{hh}\mathbf{h}_{t-1} + \mathbf{W}_{xh}\mathbf{x}_t + \mathbf{b}), has n⋅n+n⋅m+n=n(n+m)+nn \cdot n + n \cdot m + n = n(n + m) + n: one gate's worth. An LSTM has four times the parameters of an RNN of the same sizes, and does about four times the multiply-adds per step, because every parameter in the matrix is used once per example per step.

For example, with hidden size n=128n = 128 and input size m=32m = 32, one gate's worth is 128⋅(128+32)+128=128⋅160+128=20,480+128=20,608128 \cdot (128 + 32) + 128 = 128 \cdot 160 + 128 = 20{,}480 + 128 = 20{,}608. So

LSTM: 4⋅20,608=82,432,RNN: 20,608.\text{LSTM: } 4 \cdot 20{,}608 = 82{,}432, \qquad \text{RNN: } 20{,}608.

The same LSTM count from the formula: 4⋅128⋅160+4⋅128=81,920+512=82,4324 \cdot 128 \cdot 160 + 4 \cdot 128 = 81{,}920 + 512 = 82{,}432. The number grows with n2n^2: doubling the hidden size roughly quadruples the parameters and the cost per step. Libraries differ in one detail. PyTorch keeps two bias vectors, one for the input part and one for the hidden part, so its count is 4n(n+m)+8n4n(n + m) + 8n. The two biases are always added together, so the extra copy adds parameters without adding capability.

Try the formula on a larger layer.

Work it outParameters of one layer

An LSTM layer has hidden size n=256n = 256 and reads inputs of size m=64m = 64, with one bias vector per gate. How many parameters does it have? Enter the whole number, without commas.

parameters

Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.

The practice set mixes plain layers, RNN comparisons, stacked layers and PyTorch's two biases.

Practice setParameter counting reps
3 correct in a row completes the set. A miss starts the count again.

An LSTM layer and a plain RNN layer both have hidden size n=3n = 3 and input size m=50m = 50 (one bias vector per gate). How many more parameters does the LSTM have?

difference

Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.

Start with the forget gate open

Everything above used hand-picked weights. A network that is about to be trained starts from small random ones instead, and then every pre-activation is close to its bias. In particular, each forget gate starts near σ(bf)\sigma(b_f). From the previous lesson, the gradient along the cell path is multiplied by the forget gate at every step, so at initialization kk steps of cell path keep about σ(bf)k\sigma(b_f)^k of a gradient. For bf=1b_f = 1, for example, σ(1)≈0.7311\sigma(1) \approx 0.7311 and 0.731110≈0.04360.7311^{10} \approx 0.0436:

bfb_fσ(bf)\sigma(b_f)after 10 stepsafter 20 stepsafter 40 steps
00.50.000989.5×10−79.5 \times 10^{-7}9.1×10−139.1 \times 10^{-13}
10.7310.0440.00193.6×10−63.6 \times 10^{-6}
20.8810.280.0790.0062
30.9530.620.380.14
50.9930.940.870.76

With a zero bias, a fresh LSTM has a memory that halves at every step. It can still learn short dependencies, but a dependency 40 steps long produces a gradient near 10−1210^{-12} along the cell path: the signal that would teach the forget gate to stay open never arrives. A positive bias fixes the starting point. Jozefowicz, Zaremba and Sutskever (2015), in a large empirical study of recurrent architectures, recommended initializing the forget-gate bias to 1, and that is the default in this course's init_lstm.

A useful way to read the bias is as a time scale. A forget value ff close to 1 keeps information for about 1/(1−f)1/(1 - f) steps, and for f=σ(bf)f = \sigma(b_f) that is 1+ebf1 + e^{b_f} steps: about 3.7 steps for bf=1b_f = 1, 8.4 for bf=2b_f = 2, 21 for bf=3b_f = 3 and 150 for bf=5b_f = 5. (At bf=0b_f = 0 the gate is 0.5, too far from 1 for this approximation; the memory halves at every step.) So choose the bias with the gaps of your task in mind. In the training lesson this decides an experiment: on a task with a 40-step gap and a fixed training budget, an LSTM with bf=5b_f = 5 learned the task in all 24 of our runs, while the same LSTM with bf=1b_f = 1 learned it in only 1 of 24.

Go slower: Where the time scale 1+ebf1 + e^{b_f} comes from

Step 1: write the power as an exponential. fk=ekln⁡ff^k = e^{k \ln f}, because eln⁡f=fe^{\ln f} = f.

Step 2: approximate the logarithm near 1. Write f=1−ϵf = 1 - \epsilon with ϵ\epsilon small. The tangent line of ln⁡\ln at 1 has slope 1, so ln⁡(1−ϵ)≈−ϵ\ln(1 - \epsilon) \approx -\epsilon. For example ln⁡0.97≈−0.0305\ln 0.97 \approx -0.0305, close to −0.03-0.03.

Step 3: substitute. fk≈e−kϵ=e−k(1−f)f^k \approx e^{-k\epsilon} = e^{-k(1 - f)}.

Step 4: find when it reaches 1/e1/e. Take the time scale to be the number of steps until 1/e≈0.371/e \approx 0.37 of the signal is left. e−k(1−f)=e−1e^{-k(1 - f)} = e^{-1} exactly when the exponents match, k(1−f)=1k(1 - f) = 1, that is k=1/(1−f)k = 1/(1 - f) steps.

Step 5: put in the sigmoid. 1−σ(b)=1−11+e−b=e−b1+e−b1 - \sigma(b) = 1 - \dfrac{1}{1 + e^{-b}} = \dfrac{e^{-b}}{1 + e^{-b}}. Multiply the top and bottom by ebe^{b}: 1eb+1\dfrac{1}{e^{b} + 1}.

Step 6: invert. k=11−σ(bf)=1+ebfk = \dfrac{1}{1 - \sigma(b_f)} = 1 + e^{b_f}. For bf=3b_f = 3: 1+e3=1+20.09≈211 + e^3 = 1 + 20.09 \approx 21 steps.

The chart below is the amber line of the previous lesson's chart on its own, over the 40 steps of the table: the cell path at initialization, with every forget gate equal to σ(bf)\sigma(b_f). It answers: how much gradient does a given starting bias let through? The horizontal axis is kk, the number of steps back. The vertical axis is the fraction of the gradient left, on a log scale: the line marked 1 means nothing was lost, and each labelled grid line is 100 times the one below it. The readouts give the factor per step, f=σ(bf)f = \sigma(b_f), and what is left after 10 and after 40 steps.

A large bias does not stop the network from forgetting. It only sets where training starts. The weights on xtx_t and ht−1h_{t-1} can still drive the forget pre-activation far below zero when erasing is useful: the "set" latch above uses wf=−20w_f = -20 against bf=10b_f = 10. What the bias changes is the default when nothing else is known, and at the start of training nothing else is known.

Now choose a bias for a concrete gap.

Quick checkPick a starting forget bias

A freshly initialized LSTM has small random weights, so each forget gate starts close to σ(bf)\sigma(b_f). Your task needs a training signal to travel back 40 steps along the cell path. Which starting bias gives it the best chance?

Choose one answer, then check.

And explain the choice to someone who has not seen the table.

In your own wordsWhy the forget gate starts open

Explain to a colleague why LSTM code initializes the forget-gate bias to a positive number, what goes wrong if it starts at 0, and why a large positive bias does not stop the network from learning to forget.

A few sentences first: 0 of 60 characters.

Saved in this browser as you type.

Next: The forward and backward pass