ml.lab
Python sleeps until you run code

Module 8 · 6 lessons · about 345 minutes

LSTMs

The cell state and its gates, an LSTM designed by hand, the full forward and backward pass, training on text in your browser, and your own variants.

Lessons
0/6
Exercises
0/24
Code
0/7
Checkpoint
Not taken

Already comfortable with this? Take the checkpoint cold. If you pass, move on and come back only for what you missed.

  1. 1The cell state and its gatesAn LSTM adds a second memory, the cell state, that is scaled and added to instead of being rewritten through a matrix at every step. Four small gates decide what to erase, what to write and what to show.65 min
  2. 2Why gradients surviveFollow a gradient backwards through an LSTM. Along the cell state, each step multiplies it by the forget gate and by nothing else, a number the network can hold near 1. A plain RNN multiplies it by a full matrix and a tanh slope at every step.35 min
  3. 3Designing an LSTM by handWith large weights every gate becomes an on-off switch, so you can program an LSTM cell like a small circuit. Build a latch, a register, a counter and a toggle by choosing twelve numbers, then count an LSTM's parameters and see why its forget gate should start open.75 min
  4. 4The forward and backward passRun an LSTM over a whole sequence while saving what the backward pass will need, derive the gradient of one step gate by gate, carry it back through time, and prove the whole thing correct with a numerical gradient check.55 min
  5. 5Training an LSTMTrain a character-level LSTM on Shakespeare in your browser, run the head-to-head experiment against a plain RNN on a memory task, and open the trained model to see what its cells track.55 min
  6. 6Design your ownThe LSTM is one point in a space of designs. Build the gated recurrent unit from scratch, see which other changes matter and which do not, and learn to test a design idea with an experiment you can trust.60 min
  7. Checkpoint12 questions. Pass with 80% to finish the module.

Practice sets for this module

Fresh problems every time. Use them until the mechanics feel automatic.

Practice setLSTM step reps
3 correct in a row completes the set. A miss starts the count again.

A scalar LSTM (hidden size 1) has these weights:

gateww (times xtx_t)uu (times ht−1h_{t-1})bb (bias)
forget ftf_t101
input iti_t1-10
candidate gtg_t2-1-1
output oto_t011
ft=σ(wfxt+ufht−1+bf),it=σ(wixt+uiht−1+bi),gt=tanh⁡(wgxt+ught−1+bg),ot=σ(woxt+uoht−1+bo),ct=ft ct−1+it gt,ht=ottanh⁡(ct).\begin{aligned} f_t &= \sigma(w_f x_t + u_f h_{t-1} + b_f), & i_t &= \sigma(w_i x_t + u_i h_{t-1} + b_i), \\ g_t &= \tanh(w_g x_t + u_g h_{t-1} + b_g), & o_t &= \sigma(w_o x_t + u_o h_{t-1} + b_o), \\ c_t &= f_t\,c_{t-1} + i_t\,g_t, & h_t &= o_t\tanh(c_t). \end{aligned}

With xt=0x_t = 0, ht−1=1h_{t-1} = 1 and ct−1=1c_{t-1} = 1, find ctc_t and hth_t to 3 decimal places. Enter them as a column, ctc_t first.

c_t, h_t

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Practice setParameter counting reps
3 correct in a row completes the set. A miss starts the count again.

An LSTM layer and a plain RNN layer both have hidden size n=3n = 3 and input size m=50m = 50 (one bias vector per gate). How many more parameters does the LSTM have?

difference

Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.

Next moduleFrom LSTMs to today's models