ml.lab
Python sleeps until you run code
08 LSTMs

Lesson 5 of 6

Training an LSTM

Train a character-level LSTM on Shakespeare in your browser, run the head-to-head experiment against a plain RNN on a memory task, and open the trained model to see what its cells track.

About 55 minutes
By the end you can
  • Write a training loop for a recurrent language model with Adam, gradient clipping and a held-out validation set.

  • Judge a loss curve against honest baselines, and convert losses to perplexities.

  • Run and interpret a controlled experiment comparing an RNN and an LSTM on delayed recall.

  • Inspect gate and cell activations, find units that track features of the text, and say what such findings do and do not show.

Everything is in place: a cell, a backward pass that passes its gradient check, and the optimizer from module 6. This lesson trains the model twice.

First on Shakespeare. In about a minute of browser time, a network with 25,596 parameters goes from uniform noise to text with speaker names, line breaks and many real words. Then on a delayed-recall task like the one that defeated the plain RNN in module 7, head to head with that RNN. That second experiment is the reason this module exists, and you will run it yourself, with its variability. At the end you open the trained Shakespeare model and look at what its cells are doing.

The training recipe

The model is CharLSTM from mlref/lstm.py. Its LSTM layer runs the reference version of the forward and backward pass you wrote in the previous lesson: the same functions, names and shapes. It has three parts:

  1. An embedding table E\mathbf{E} of shape V×DV \times D. Character number kk becomes row kk of E\mathbf{E}, a vector of DD numbers. As in module 7, a lookup is the same as multiplying a one-hot vector by E\mathbf{E}, and the rows are learned.
  2. One LSTM layer with hidden size nn, reading the embeddings.
  3. An output layer that maps each ht\mathbf{h}_t to VV scores (logits), one per possible next character.

With V=60V = 60 characters, D=16D = 16 and n=64n = 64, count each part. The embedding has 60⋅16=96060 \cdot 16 = 960 numbers. The LSTM reads inputs of size m=D=16m = D = 16, so it has 4n(n+m)+4n=4⋅64⋅(64+16)+4⋅64=20,480+256=20,7364n(n + m) + 4n = 4 \cdot 64 \cdot (64 + 16) + 4 \cdot 64 = 20{,}480 + 256 = 20{,}736 (lesson 3). The output layer is a 64×6064 \times 60 matrix plus 60 biases: 64⋅60+60=3,840+60=3,90064 \cdot 60 + 60 = 3{,}840 + 60 = 3{,}900. In all, 960+20,736+3,900=25,596960 + 20{,}736 + 3{,}900 = 25{,}596, and 20,736/25,596≈0.8120{,}736 / 25{,}596 \approx 0.81: four fifths of the model is the LSTM.

The loss is the one from module 7: at every position, the cross-entropy of the true next character under the model's softmax, averaged over all positions in the batch. One training step:

  1. Sample a batch of 16 random windows of 48 characters. The targets are the same windows shifted by one character.
  2. Forward: embed, run the LSTM from a zero state, compute the logits and the mean cross-entropy.
  3. Backward: the softmax gradient p−y\mathbf{p} - \mathbf{y} from module 6, then the output layer, then the LSTM backward pass through all 48 steps, then the embedding rows.
  4. Clip the gradient: if the combined norm of all gradients, as one long vector, exceeds 5, scale every gradient down so that it equals 5. For example, a combined norm of 8 multiplies every gradient by 5/8=0.6255/8 = 0.625; a norm of 3 leaves them alone. The direction of the step is unchanged, only its length is capped. The LSTM's cell path does not explode, but the paths through ht−1\mathbf{h}_{t-1} can, and one bad batch should not wreck the weights (module 7).
  5. Update with Adam at a learning rate of 0.008. Adam gives every parameter its own step size from the history of its gradients, which matters here: Q appears once in the whole text, so its embedding row receives a gradient in fewer than 1% of the batches, while the LSTM weights receive one every step. This is the conditioning problem of module 5, handled one coordinate at a time.

Every window starts from a zero state, so the model never sees more than 48 characters of context during training. That is truncated backpropagation through time: it bounds the longest dependency the model can be taught. When sampling, though, the state is carried forward indefinitely.

In short: embed, run the LSTM, score every next character, backpropagate through the window, clip, and take an Adam step. Before training, check the starting point. An untrained model spreads its probability roughly evenly over the VV characters, so each correct character gets about 1/V1/V and the loss, −ln⁡(1/V)=ln⁡V-\ln(1/V) = \ln V, should come out close to ln⁡60≈4.09\ln 60 \approx 4.09. The run below prints both, and a sample of untrained text:

⌘+Enter runs · edit freelyPython sleeps until you run code

Then know what the numbers mean. Three baselines, computed on this text:

ModelLoss (nats per character)Perplexity elosse^{\text{loss}}
uniform over 60 charactersln⁡60≈4.09\ln 60 \approx 4.0960
character frequencies only (unigram)3.3027
previous character only (bigram)2.3710.7

Perplexity is elosse^{\text{loss}}, the size of a uniform guess that would be equally uncertain: a uniform guess over kk characters has loss ln⁡k\ln k, so undoing the logarithm gives kk back. A loss of 2.37 gives e2.37≈10.7e^{2.37} \approx 10.7: as uncertain as choosing among about 11 equally likely characters. The unigram and bigram numbers are measured on the same text they were counted from, so they flatter those models a little (on held-out text, module 7 measured 3.32 and 2.38); they are still the bar that any use of context must clear.

Convert a trained model's loss the same way.

Work it outFrom loss to perplexity

A character model on the 60-character Shakespeare vocabulary starts at a loss of ln⁡60\ln 60 and trains down to a loss of 1.61.6 nats per character. What is its perplexity at the end? Give 2 decimal places.

perplexity

Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.

The baselines also tell you what a stuck loss curve means.

Quick checkRead the loss curve

You train a character LSTM on the Shakespeare text (V=60V = 60). The loss starts at 4.094.09, falls quickly to about 3.33.3, and then stays flat for hundreds of steps. Samples look like random letters and spaces in roughly the right proportions. What is the most likely situation?

Choose one answer, then check.

Now train it. The run holds back the last tenth of the text and never trains on it, so you can see whether the model learned the language or memorized the windows. It also prints a sample and ends with a look inside, which the last section of this lesson interprets.

Code itTrain a character LSTM on Shakespeare

Train a character-level LSTM language model in your browser, then look inside it. The model is the reference CharLSTM (embedding, one LSTM layer with your forward and backward pass, output layer); you write the training machinery.

  1. evaluate(model, ids, seq_len, batch_size, n_batches, rng): the mean loss over n_batches random windows, without changing the model.
  2. train(model, ids, steps, ...): each step, sample a batch of windows, compute the loss and gradients, clip the gradients in place with clip_grad_norm so their combined norm is at most clip, and take one Adam step. Plot the loss with show.line every log_every steps and return the list of per-step losses.
  3. best_cell(C, feature): the column of C whose correlation with feature is largest in size, and that correlation (with its sign). A column that never changes has correlation 0.

The run at the bottom trains for 1,200 steps (about 35 to 60 seconds), measures the loss on the last tenth of the text, which it never trains on, prints a sample, and searches the cell states for units that track the layout of the text.

This one trains a model in your browser, so a run can take a minute or more. The charts update as it goes, and Stop ends it early.

⌘+Enter runsPython sleeps until you run code
Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.

In our runs, 1,200 steps took about 35 seconds in the same Python runtime your browser uses. The training loss ended near 1.6 and the validation loss near 1.7: far below the bigram baseline, and close together, so the model is learning the play's language rather than memorizing it. The samples have the right shape before they have the right words: speaker names in capitals followed by a colon and a newline, lines of plausible length, common words, punctuation in roughly the right places.

Memory across a gap: RNN versus LSTM

Language modeling mixes short and long dependencies, so it is a poor test of memory alone. The delayed-recall task isolates it. This version differs from module 7's in two details: the distractors are drawn from the same four symbols as the answer, and the model is a classifier that answers once, at the end, instead of predicting every next token.

  • Step 1 shows a symbol from {0,1,2,3}\{0,\allowbreak 1,\allowbreak 2,\allowbreak 3\} (one-hot, 4 channels) and a marker bit that says "this is the first step".
  • Steps 2 to d+1d + 1 show random symbols from the same set. They carry no information about the answer.
  • The last step shows only a query bit. The model must output the symbol from step 1.

Each sequence has d+2d + 2 steps, where dd is the delay. Guessing gives an accuracy of 0.25. The classifier reads the last hidden state only, so the loss is computed at the last step, and its gradient has to travel back through all d+1d + 1 earlier steps to reach the one that matters.

Both models get the same treatment: hidden size 16, Adam at a learning rate of 0.01, batches of 64 freshly generated sequences, gradient clipping at 1, and 1,200 steps; accuracy is measured on 500 new sequences. Count them with 6 input channels (4 symbols, the marker and the query) and a 4-way output layer. The RNN has 16⋅16+16⋅6+16=36816 \cdot 16 + 16 \cdot 6 + 16 = 368 in its layer plus 16⋅4+4=6816 \cdot 4 + 4 = 68 in the output layer: 436. The LSTM has 4⋅16⋅(16+6)+4⋅16=1,408+64=1,4724 \cdot 16 \cdot (16 + 6) + 4 \cdot 16 = 1{,}408 + 64 = 1{,}472 plus the same 68: 1,540, because each LSTM gate is a full RNN-sized layer. Here is what happened in reference runs with 24 different seeds per setting. A run counts as a success when its test accuracy is at least 0.9.

Delay dd510204060
RNN24 of 2420 of 247 of 242 of 240 of 24
LSTM, bf=5b_f = 524 of 2424 of 2424 of 2424 of 2419 of 24

One more comparison, at d=40d = 40, answers the natural objection that the LSTM wins only because it has more parameters. Here is the same LSTM with only its starting forget bias changed:

LSTM at d=40d = 40bf=1b_f = 1bf=3b_f = 3bf=5b_f = 5
successes1 of 2423 of 2424 of 24

Same parameters, same data, same budget. What changes is how much gradient survives the cell path at the start of training: σ(1)40≈0.731140≈3.6×10−6\sigma(1)^{40} \approx 0.7311^{40} \approx 3.6 \times 10^{-6}, σ(3)40≈0.952640≈0.14\sigma(3)^{40} \approx 0.9526^{40} \approx 0.14 and σ(5)40≈0.993340≈0.76\sigma(5)^{40} \approx 0.9933^{40} \approx 0.76 (lesson 3). The LSTM's advantage is not that it has gates; it is that its gates can hold a path open, and at initialization only the bias holds it open.

Two more observations. First, failures are mostly all or nothing. A run either finds the solution and reaches 1.0, or stays near 0.25; a few runs stall in between, near 0.5 or 0.75, having learned to tell apart two or three of the four symbols. When the learning signal is weak, whether training ever picks it up depends on the luck of the initialization and the first batches. Second, the RNN's limit is about learning, not representing. The same 16-unit RNN solved delay 40 in 2 runs, so weights that do the task exist; training usually fails to find them. An RNN with 32 units, trained the same way, solved delay 40 in 7 of 24 runs in a separate test, and got partway in several more. The difficulty grows with the gap; there is no fixed wall.

To recap the experiment: at short delays both models learn; as the delay grows the RNN fails more and more often; the LSTM keeps learning as long as its starting forget bias holds the cell path open.

Now run it. Your numbers will differ from the table, because one run per setting is a single sample, which is why the experiment averages three RNN runs at every delay.

Code itRNN versus LSTM on delayed recall

Run the experiment that motivates this whole module. A model sees a symbol (0 to 3) at the first step, then delay steps of random symbols from the same set, then a query, and must report the first symbol. You write the task and the LSTM side; a plain RNN classifier is provided.

The setup defines N_SYMBOLS = 4, QUERY = 4, MARKER = 5, N_INPUTS = 6, and RNNClassifier(n_in, hidden, n_classes, rng) with a params dict, loss_and_grads(X, y) -> (loss, grads) and predict(X). It also imports init_lstm, lstm_forward, lstm_backward, Adam, clip_grad_norm and softmax_cross_entropy.

  1. make_recall_batch(delay, batch_size, rng): the inputs, one-hot, and the targets.
  2. LSTMClassifier: the same interface as RNNClassifier, using lstm_forward and lstm_backward, with the loss on the last step only.
  3. train_and_test(model, delay, steps, rng, ...): Adam with clipping on fresh batches, then the accuracy on new sequences.
  4. compare(delays, steps): for each delay, the mean accuracy of three RNNs and the accuracy of one LSTM with forget bias 5. The run takes about a minute.

This one trains a model in your browser, so a run can take a minute or more. The charts update as it goes, and Stop ends it early.

⌘+Enter runsPython sleeps until you run code
Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.

Then decide what the results actually support.

Quick checkWhat the experiment shows

In the delayed-recall experiment, the plain RNN reliably solves delay 5, is unreliable at delay 20, and is at chance at delay 60, while the LSTM with forget bias 5 solves delay 40 and usually delay 60. Which conclusion does this support best?

Choose one answer, then check.

Look inside

The Shakespeare model learned something. The last question is where that knowledge sits. A trained network is a set of numbers, but it can be observed while it reads. CharLSTM.gate_activations(ids) runs one sequence from a zero state and returns the four gates, the cell state and the hidden state at every step, each as an array with one row per character and one column per unit. The end of the training exercise drew the cell states over 80 characters of held-out text as a heatmap (units as rows, characters as columns). Most rows look like noise at first. Some have visible structure: bands that switch at every newline, or that stay on for the length of a speaker's name.

To find such units systematically, pick a feature of the text that you can compute, such as "this character is on a line that names a speaker" or "number of characters since the last newline", and find the unit whose cell state correlates with it most strongly. That is what best_cell did. In five reference runs of the training exercise, with seeds 0 to 4:

  • the best unit for speaker-name lines had a correlation between 0.66 and 0.78 in size;
  • the best unit for position in the line had a correlation between 0.62 and 0.83 in size;
  • the typical (median) unit had a correlation of only about 0.2 to 0.3 with the speaker feature, and under 0.2 with the position.

The sign of a correlation is arbitrary, since the output layer can read a unit either way, which is why only the size is reported. And a different run puts these features in different units: the speaker-line unit was number 44 in one run and number 31 in another.

These are real findings, and they match what Karpathy, Johnson and Fei-Fei reported in 2015 for character LSTMs trained on larger texts: a few cells that tracked line length, quotation marks or the nesting depth of code, alongside many cells with no simple interpretation. Treat them with care. A correlation on one passage shows that a unit moves with a feature; it does not show that the unit exists for that feature, or that the model depends on it. To test that, you would intervene: change the unit's value and watch the predictions.

Put that caution into words for a colleague who has just found a clean unit.

In your own wordsWhat one interpretable cell means

After training, you find a unit whose cell state has a correlation of −0.75-0.75 with "this line is a speaker's name". Your colleague concludes that the model has a speaker-name neuron and that the rest of its units are noise. Explain what the correlation does and does not show, and what you would expect from the other units.

A few sentences first: 0 of 60 characters.

Saved in this browser as you type.

Next: Design your own