Lesson 3 of 6
Designing an LSTM by hand
With large weights every gate becomes an on-off switch, so you can program an LSTM cell like a small circuit. Build a latch, a register, a counter and a toggle by choosing twelve numbers, then count an LSTM's parameters and see why its forget gate should start open.
Choose weights that make a sigmoid gate open or closed in exactly the situations you want.
Design one-unit LSTMs that latch, hold a value, count and toggle, and verify them on input sequences.
Count the parameters of an LSTM layer, , and compare with a plain RNN.
Explain why the forget-gate bias starts positive, and predict how much gradient survives steps at initialization.

Before training an LSTM, program one. A cell with a single unit has twelve numbers: for each of its four gates, a weight on the input , a weight on the previous output and a bias . Choose them well and the cell becomes a small circuit with memory: a latch that remembers whether it has ever seen a 1, a register that holds a number, a counter, a toggle switch.
Doing this by hand is the fastest way to learn what each weight can and cannot do. It also makes one fact concrete: gates are programmable logic. Training, which the next two lessons set up, is a search for programs like the ones you are about to write, except that nobody tells the network which program it needs.
Throughout this lesson the cell has one unit, so every quantity is a number:
starting from . These are the equations of the first lesson with : is the forget gate, the input gate, the candidate and the output gate.
Saturated gates are switches
To design by hand we want gates that are fully open or fully shut, not half open. Far from zero the sigmoid flattens out:
| 0.00005 | 0.0067 | 0.1192 | 0.5 | 0.8808 | 0.9933 | 0.99995 |
Check the ends: , and . So a pre-activation of means "open" and means "closed", each with an error of about . The candidate works the same way with tanh: is within of 1, and exactly. (The four-step figure in the first lesson used , where the error is ; is a sharper switch.)
That turns weight design into a recipe.
- List the situations the cell can be in: the possible inputs, and, if it matters, whether something is already stored.
- Decide what each gate should do in each situation: open (1) or closed (0). Decide what the candidate should write: , or .
- Choose , and so that the gate's pre-activation is at least in the situations where it must be open and at most where it must be closed.
Step 3 is drawing a threshold. Say the inputs are 0 and 1 and a gate must open only when . Try and :
| pre-activation | gate | |
|---|---|---|
| 0 | , closed | |
| 1 | , open |
For a gate that must open unless , among inputs , and , take and : the pre-activations are , and . This is exactly what a single neuron does: it draws a line through its inputs and says which side you are on. A gate is a neuron whose output multiplies something.
One restriction shapes every design: a gate sees only and . It cannot read the cell state. If a gate needs to know what is stored, it has to read it through , which lies between and , using its weight.
A latch
The first task: the inputs are 0s and 1s, and the output must switch on (at least 0.5) at the first 1 and stay on for good. Before the first 1 it must stay off (within 0.1 of 0).
The widget is the cell from the first lesson in design mode. It answers: do your twelve weights solve the task? At the top of the controls are the task, its goal, a switch between three test sequences and a verdict that says how many tests pass and where the first failure is. The cell shows one step of the test shown, and the strip chart under it shows the whole test: the inputs as squares (filled for a 1), the output as a lime line, the memory as a faint amber line, and a band at each step where must be (lime where it passes, pink where it fails). Pick a gate with its button (, , , ) and set its , and with the sliders, or type into the grid of all twelve weights. Every weight starts at 0.
Each setting has a reason. The memory lives in , so the forget gate must keep it at every step. Something positive must be written when and nothing when . You could make the input gate do that selection, but the candidate already does it: with , is exactly whenever , and about 1 when . So the input gate can stay open. And the output gate has to show the memory. (The widget's weights run from to . That is enough for every task it checks, though not for the weights some designs below use; those you can try in code.)
Now check the design on paper, step by step, and find out what the cell is really computing.
Design a one-unit LSTM latch. The inputs are 0s and 1s; the output must be at least 0.5 from the first 1 onward and within 0.1 of 0 before it. Use these restrictions: the only weight that looks at is ; every is 0; each bias is , or .
- For each gate, decide whether it should be open, closed, or depend on the input, and choose the four biases.
- Verify your design on the inputs : compute and for from . Treat as 1, as 0 and as 1.
- What does tell you about what this cell really computes? Why does that not break the latch?
The check at the top takes and , in that order, to 3 decimal places.
Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
The latch you just verified is secretly a counter: every 1 adds another 1 to . That does not break the latch, because is at least for every . But there is a second design that keeps at 1 instead of counting, and it uses the forget gate differently. When , erase the cell and write a fresh 1: set, instead of add.
- Forget: open when , closed when : , .
- Input: open only when : , .
- Candidate: always 1: .
- Output: always open: .
Check both situations with the pre-activations:
| forget: | input: | candidate | ||
|---|---|---|---|---|
| 0 | , open | , closed | ||
| 1 | , closed | , open |
At a 1 the cell becomes 1, whatever was there; at a 0 it keeps its value. Two designs, one behavior at the output, different memories. A trained network faces the same freedom, which is one reason its weights are hard to read.
A latch that passes short tests can still fail on long ones. Here is one; find the fix.
You enter a latch with , , , and every other weight 0. It passes a test in which a 1 is followed by 10 zeros, but on a longer test drops below 0.5 at the 13th zero after the 1. Which change fixes it?
Choose one answer, then check.
Hold, count and toggle
The latch stored one bit. The next three tasks store a number, count, and make a decision that depends on the cell's own state.
Hold. Now the first input is a number between and , every later input is 0, and must stay within 0.1 of that first number at every step. The widget checks three tests, with first inputs , and , and draws each band around that first input.
The structure is the latch's: keep everything (forget open), write whenever the candidate is nonzero (input open), show everything (output open). Since every later input is 0 and , the candidate is after the first step, so nothing more is written. On the first step the cell starts empty, so , and from then on
The value is squashed twice, once by the candidate's tanh and once at the output. Each tanh pulls a value toward zero ( for every ), so must be above 1 to compensate. Check the widget's answer, , on test 1, where : , then , then . That is the the widget showed, off by . With the same two squashes give and , the failing of the walkthrough.
How large may be? Two inputs set the limits. A large first input gets squashed the most, so it needs large enough: at , gives and , just within 0.1. A small first input sits where tanh is nearly straight, so a large overshoots it: at , gives and , just within 0.1 above. Checking every first input in shows that any from about 1.22 to 1.50 keeps the error at or under 0.1. In a real network nobody needs to equal a number exactly: the next layer's weights can undo a squashing. The lesson is that is a bounded, distorted view of .
Count. The inputs are 0s and 1s again, and now the cell state itself must equal the number of 1s so far, within 0.25. The widget's third test is twelve 1s in a row.
The latch already counts. Each 1 writes with the input gate open, and the forget gate keeps the total. The only question is precision: after twelve 1s the cell holds about , which must be at least . Divide by 12: . Undo the tanh with its inverse, : . That is why failed and passed.
Here the difference between and matters. The cell state is not squashed, so it can hold 12, or 30. The output cannot tell them apart: and both equal 1 to nine decimal places, and even is almost there. A count is available inside the cell, and a later layer can only read a saturated version of it. Run the counting latch and compare the two: the first printed row is the true count, the second is , and the third is . The cell state climbs to 8 in steps of 1. The output reads after the first 1 and after the second, then barely moves: , , and (to three decimals) from the fifth 1 on.
Toggle. Last, a design the widget does not check: each 1 flips the cell between off and on, and a 0 leaves it alone. This is harder, because what a 1 should do depends on the current state. Run the recipe's first two steps. There are four situations:
| cell before | must | memory | ||
|---|---|---|---|---|
| 0 | off | about 0 | stay off | keep, write nothing |
| 0 | on | about | stay on | keep, write nothing |
| 1 | off | about 0 | turn on | write a 1 |
| 1 | on | about | turn off | erase, write nothing |
Now look down each gate's column of decisions. The forget gate can decide from alone: keep at every 0, erase at every 1 (erasing an off cell costs nothing, since it holds 0). But the input gate must open in the third row and stay shut in the fourth, and those two rows have the same input, . The only thing that differs is : about when the cell is on (that is ) and about 0 when it is off. So the input gate needs a nonzero weight: the cell has to read its own output to decide. That is the first design here in which the recurrence gates does real work.
Now build all of these in code, including the toggle, and test them on many random sequences at once. The recipe's third step, choosing the numbers, is yours.
Build hand-designed LSTMs in code and test them on many random sequences.
pack(p)turns the 12 named weights (the widget's names:wf,uf,bf,wi, ... ,bo) into the stackedWof shape andbof shape : row 0 ofWholds the weights, row 1 the weights, and the columns are the gates in the order .run(p, xs)runs the one-unit cell over the inputs from with the referencelstm_stepand returns the arrays of and .latch(),counter()andtoggle()each return 12 weights (use theweights(...)helper) for the tasks in their docstrings. The toggle needs a weight: the gates must read the cell's own output to know whether it is on.
raise NotImplementedError, then press Run tests. Each check says what it expects.Counting parameters
Hand design used 12 numbers. A real layer has many more, and the count tells you what an LSTM costs. Each gate has an matrix for the input ( numbers), an matrix for the previous output ( numbers) and biases: numbers. The four gates together have
In the stacked layout this is the matrix plus the entries of : , the same count. A plain RNN, , has : one gate's worth. An LSTM has four times the parameters of an RNN of the same sizes, and does about four times the multiply-adds per step, because every parameter in the matrix is used once per example per step.
For example, with hidden size and input size , one gate's worth is . So
The same LSTM count from the formula: . The number grows with : doubling the hidden size roughly quadruples the parameters and the cost per step. Libraries differ in one detail. PyTorch keeps two bias vectors, one for the input part and one for the hidden part, so its count is . The two biases are always added together, so the extra copy adds parameters without adding capability.
Try the formula on a larger layer.
An LSTM layer has hidden size and reads inputs of size , with one bias vector per gate. How many parameters does it have? Enter the whole number, without commas.
Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.
The practice set mixes plain layers, RNN comparisons, stacked layers and PyTorch's two biases.
An LSTM layer and a plain RNN layer both have hidden size and input size (one bias vector per gate). How many more parameters does the LSTM have?
Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.
Start with the forget gate open
Everything above used hand-picked weights. A network that is about to be trained starts from small random ones instead, and then every pre-activation is close to its bias. In particular, each forget gate starts near . From the previous lesson, the gradient along the cell path is multiplied by the forget gate at every step, so at initialization steps of cell path keep about of a gradient. For , for example, and :
| after 10 steps | after 20 steps | after 40 steps | ||
|---|---|---|---|---|
| 0 | 0.5 | 0.00098 | ||
| 1 | 0.731 | 0.044 | 0.0019 | |
| 2 | 0.881 | 0.28 | 0.079 | 0.0062 |
| 3 | 0.953 | 0.62 | 0.38 | 0.14 |
| 5 | 0.993 | 0.94 | 0.87 | 0.76 |
With a zero bias, a fresh LSTM has a memory that halves at every step. It can still learn short dependencies, but a dependency 40 steps long produces a gradient near along the cell path: the signal that would teach the forget gate to stay open never arrives. A positive bias fixes the starting point. Jozefowicz, Zaremba and Sutskever (2015), in a large empirical study of recurrent architectures, recommended initializing the forget-gate bias to 1, and that is the default in this course's init_lstm.
A useful way to read the bias is as a time scale. A forget value close to 1 keeps information for about steps, and for that is steps: about 3.7 steps for , 8.4 for , 21 for and 150 for . (At the gate is 0.5, too far from 1 for this approximation; the memory halves at every step.) So choose the bias with the gaps of your task in mind. In the training lesson this decides an experiment: on a task with a 40-step gap and a fixed training budget, an LSTM with learned the task in all 24 of our runs, while the same LSTM with learned it in only 1 of 24.
Go slower: Where the time scale comes from
Step 1: write the power as an exponential. , because .
Step 2: approximate the logarithm near 1. Write with small. The tangent line of at 1 has slope 1, so . For example , close to .
Step 3: substitute. .
Step 4: find when it reaches . Take the time scale to be the number of steps until of the signal is left. exactly when the exponents match, , that is steps.
Step 5: put in the sigmoid. . Multiply the top and bottom by : .
Step 6: invert. . For : steps.
The chart below is the amber line of the previous lesson's chart on its own, over the 40 steps of the table: the cell path at initialization, with every forget gate equal to . It answers: how much gradient does a given starting bias let through? The horizontal axis is , the number of steps back. The vertical axis is the fraction of the gradient left, on a log scale: the line marked 1 means nothing was lost, and each labelled grid line is 100 times the one below it. The readouts give the factor per step, , and what is left after 10 and after 40 steps.
A large bias does not stop the network from forgetting. It only sets where training starts. The weights on and can still drive the forget pre-activation far below zero when erasing is useful: the "set" latch above uses against . What the bias changes is the default when nothing else is known, and at the start of training nothing else is known.
Now choose a bias for a concrete gap.
A freshly initialized LSTM has small random weights, so each forget gate starts close to . Your task needs a training signal to travel back 40 steps along the cell path. Which starting bias gives it the best chance?
Choose one answer, then check.
And explain the choice to someone who has not seen the table.
Explain to a colleague why LSTM code initializes the forget-gate bias to a positive number, what goes wrong if it starts at 0, and why a large positive bias does not stop the network from learning to forget.
Saved in this browser as you type.