Module 7 · 5 lessons · about 295 minutes
Recurrent networks
Sequences and hidden state, next-character language models, backpropagation through time, and why plain recurrent networks forget.
- Lessons
- 0/5
- Exercises
- 0/19
- Code
- 0/7
- Checkpoint
- Not taken
Already comfortable with this? Take the checkpoint cold. If you pass, move on and come back only for what you missed.

- 1Sequences and stateA recurrent network reads a sequence one element at a time and carries a hidden state from each step to the next, using the same weights at every step. Unrolled through time, it is a deep network whose layers all share one set of weights.70 min
- 2Predicting the next characterA character language model turns every hidden state into a probability for each possible next character, and it is trained to make the real text probable. The loss is the average cross-entropy, the same objective used to pretrain GPT-style models.70 min
- 3Generating textA language model writes by running its prediction in a loop. It computes the distribution over the next character, chooses one, appends it and reads it back in. How it chooses (greedy, sampling, a temperature) decides what the text looks like.40 min
- 4Backpropagation through timeTraining an RNN means backpropagating through the unrolled network. Each hidden state collects gradient from its own output and from the next step, and each shared weight collects one term per step. With clipping as a safety net, that is enough to train a character model on Shakespeare.55 min
- 5Vanishing and exploding gradientsThe gradient from k steps back is a product of k factors built from the same recurrent weights, so it shrinks or grows geometrically. Clipping tames the growth; nothing simple fixes the shrinking, and a plain RNN fails a memory test once the delay grows beyond a short range.60 min
- Checkpoint12 questions. Pass with 80% to finish the module.
Practice sets for this module
Fresh problems every time. Use them until the mechanics feel automatic.
A two-unit RNN computes with applied to each entry. Use , , , the previous state and the input . Find to 3 decimal places.
hₜ
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.