ml.lab
Python sleeps until you run code

Module 7 · 5 lessons · about 295 minutes

Recurrent networks

Sequences and hidden state, next-character language models, backpropagation through time, and why plain recurrent networks forget.

Lessons
0/5
Exercises
0/19
Code
0/7
Checkpoint
Not taken

Already comfortable with this? Take the checkpoint cold. If you pass, move on and come back only for what you missed.

  1. 1Sequences and stateA recurrent network reads a sequence one element at a time and carries a hidden state from each step to the next, using the same weights at every step. Unrolled through time, it is a deep network whose layers all share one set of weights.70 min
  2. 2Predicting the next characterA character language model turns every hidden state into a probability for each possible next character, and it is trained to make the real text probable. The loss is the average cross-entropy, the same objective used to pretrain GPT-style models.70 min
  3. 3Generating textA language model writes by running its prediction in a loop. It computes the distribution over the next character, chooses one, appends it and reads it back in. How it chooses (greedy, sampling, a temperature) decides what the text looks like.40 min
  4. 4Backpropagation through timeTraining an RNN means backpropagating through the unrolled network. Each hidden state collects gradient from its own output and from the next step, and each shared weight collects one term per step. With clipping as a safety net, that is enough to train a character model on Shakespeare.55 min
  5. 5Vanishing and exploding gradientsThe gradient from k steps back is a product of k factors built from the same recurrent weights, so it shrinks or grows geometrically. Clipping tames the growth; nothing simple fixes the shrinking, and a plain RNN fails a memory test once the delay grows beyond a short range.60 min
  6. Checkpoint12 questions. Pass with 80% to finish the module.

Practice sets for this module

Fresh problems every time. Use them until the mechanics feel automatic.

Practice setRNN step reps
3 correct in a row completes the set. A miss starts the count again.

A two-unit RNN computes ht=tanh⁡(Whhht−1+wxhxt+b)\mathbf{h}_t = \tanh(\mathbf{W}_{hh}\mathbf{h}_{t-1} + \mathbf{w}_{xh}x_t + \mathbf{b}) with tanh⁡\tanh applied to each entry. Use Whh=[−1−0.5−0.5−0.5]\mathbf{W}_{hh} = \begin{bmatrix}-1 & -0.5 \\ -0.5 & -0.5\end{bmatrix}, wxh=[−1−1]\mathbf{w}_{xh} = \begin{bmatrix}-1 \\ -1\end{bmatrix}, b=[00.5]\mathbf{b} = \begin{bmatrix}0 \\ 0.5\end{bmatrix}, the previous state ht−1=[0.5−1]\mathbf{h}_{t-1} = \begin{bmatrix}0.5 \\ -1\end{bmatrix} and the input xt=−1x_t = -1. Find ht\mathbf{h}_t to 3 decimal places.

hₜ

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Next moduleLSTMs