Module 8 · 6 lessons · about 345 minutes
LSTMs
The cell state and its gates, an LSTM designed by hand, the full forward and backward pass, training on text in your browser, and your own variants.
- Lessons
- 0/6
- Exercises
- 0/24
- Code
- 0/7
- Checkpoint
- Not taken
Already comfortable with this? Take the checkpoint cold. If you pass, move on and come back only for what you missed.

- 1The cell state and its gatesAn LSTM adds a second memory, the cell state, that is scaled and added to instead of being rewritten through a matrix at every step. Four small gates decide what to erase, what to write and what to show.65 min
- 2Why gradients surviveFollow a gradient backwards through an LSTM. Along the cell state, each step multiplies it by the forget gate and by nothing else, a number the network can hold near 1. A plain RNN multiplies it by a full matrix and a tanh slope at every step.35 min
- 3Designing an LSTM by handWith large weights every gate becomes an on-off switch, so you can program an LSTM cell like a small circuit. Build a latch, a register, a counter and a toggle by choosing twelve numbers, then count an LSTM's parameters and see why its forget gate should start open.75 min
- 4The forward and backward passRun an LSTM over a whole sequence while saving what the backward pass will need, derive the gradient of one step gate by gate, carry it back through time, and prove the whole thing correct with a numerical gradient check.55 min
- 5Training an LSTMTrain a character-level LSTM on Shakespeare in your browser, run the head-to-head experiment against a plain RNN on a memory task, and open the trained model to see what its cells track.55 min
- 6Design your ownThe LSTM is one point in a space of designs. Build the gated recurrent unit from scratch, see which other changes matter and which do not, and learn to test a design idea with an experiment you can trust.60 min
- Checkpoint12 questions. Pass with 80% to finish the module.
Practice sets for this module
Fresh problems every time. Use them until the mechanics feel automatic.
A scalar LSTM (hidden size 1) has these weights:
| gate | (times ) | (times ) | (bias) |
|---|---|---|---|
| forget | 1 | 0 | 1 |
| input | 1 | -1 | 0 |
| candidate | 2 | -1 | -1 |
| output | 0 | 1 | 1 |
With , and , find and to 3 decimal places. Enter them as a column, first.
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
An LSTM layer and a plain RNN layer both have hidden size and input size (one bias vector per gate). How many more parameters does the LSTM have?
Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.