Lesson 4 of 5
Backpropagation through time
Training an RNN means backpropagating through the unrolled network. Each hidden state collects gradient from its own output and from the next step, and each shared weight collects one term per step. With clipping as a safety net, that is enough to train a character model on Shakespeare.
Derive the backward pass of an RNN, including the gradient that reaches each hidden state and the weight gradients as sums over time.
Run backpropagation through time by hand for a scalar RNN and check the result with finite differences.
Implement and gradient-check backpropagation through time for a batch of sequences.
Clip gradients by their combined norm, and explain why clipping keeps the direction of the step.
Train a character RNN on Shakespeare in the browser and read its loss against the baselines.

The RNN from the first lesson, used as the language model of the second, starts at a loss of about 4.2 nats, no better than guessing uniformly (). It has to learn. Its loss depends on the recurrent weights through every step. When it ends a sentence too early, the evidence that the sentence still needed its verb may be 40 characters back. How does the blame for a mistake at step 50 reach the weights as they were used at step 10?
The answer is the one module 6 gave for any network: backpropagation. Unroll the RNN into the deep network of the first lesson and run the backward pass through it. This is called backpropagation through time (BPTT). Most of it is familiar, with two new bookkeeping rules. This lesson derives it, checks it by hand and in code, and uses it to train a character model on Shakespeare. The next lesson looks at what happens to the gradient on its long way back, which explains why plain RNNs forget.
Unroll, then backpropagate
We derive the gradients for one sequence, with column vectors. First, the forward pass it undoes. For it computes
and the loss with . Here is the pre-activation, is the id of the target character at step and is its one-hot vector. (Training uses the mean over positions instead of the sum; that divides every gradient by and changes nothing else.)
Two new rules make this different from a plain feed-forward network.
A state is used twice, so its gradient is a sum. feeds its own output and the next pre-activation . The rule from backpropagation for a value used in two places applies: add the gradients from both uses.
A weight is used at every step, so its gradient is a sum over time. The same multiplies , then , and so on. Its total gradient is the sum of the gradients from each use.
Name the two gradients we carry backward, both vectors with entries:
- , how much the loss changes per small change in each entry of the state ;
- , the same for the pre-activation, just before the tanh.
Then walk from down to . At each step, three lines, each one move:
where multiplies entry by entry and squares each entry. The order matters: needs , which the walk computed one step earlier, because it goes backwards. At the last step nothing comes from the future, so .
The weight gradients collect one term per step, each the same outer product as a module 6 layer:
The part of that arrives from step is the rule to remember:
Through the tanh, then back through the recurrent matrix. It is the vector-Jacobian product of Jacobians and the chain rule for one step of the network. Notice that the slope uses , the output of the step being undone, not .
Go slower: Every line, entry by entry
The tanh. Entry of depends only on entry of , and (derivatives as zoom). Since , that slope is . By the chain rule, , which for all at once is .
The state, used twice. Entry of appears in two places. In the output, , so . In the next step, , so . The chain rule adds up every path from to : A sum over the first index of a matrix is a product with its transpose: is entry of . So in vectors this is . We can use here because we walk backwards: it was computed one step earlier in the loop.
The shared weight. appears in for every , each time multiplied by . So , and adding the uses over time, which is entry of . The same argument with in place of gives , and with the constant 1 gives . The output layer is module 6's linear layer applied at every step, summed.
The starting state. is used only by , so .
A worked example
The derivation has many symbols. A scalar RNN over two steps shows every one of them as a number. Take , , , and inputs , , and suppose the loss depends only on the last state, with (for instance ). Write and .
Forward. Step 1: and . Step 2: and .
Backward, step 2. The gradient arrives at the last state: . Through the tanh, the slope at step 2 is , so .
Backward, step 1. Step 1 has no output of its own, so has only the part that arrives from step 2: . The slope at step 1 is , so .
Weights. Each weight collects one term per step. For , the term at step is times what multiplied there, . For it is times :
The second term of , , is the effect of through step 1, reaching by way of : that is the path through time. Without it the answer would be , 30% too small.
A finite-difference check confirms both numbers: nudge (or ) by up and down, rerun the forward pass each time, and divide the change in by . The code below does the whole example, the backward walk and the check.
The two columns agree to four decimals: and .
In code
With batches as rows and the code layout of the first lesson (Whh is ), one backward step is
h_prev = Hs[:, t - 1] if t > 0 else h0 # (B, n): the states h_(t-1) that step t read
dh = dHs[:, t] + dh_next # (B, n): from the output at t, plus from step t + 1
da = dh * (1 - Hs[:, t] ** 2) # through the tanh
dWhh += h_prev.T @ da # this step's share of the shared weight's gradient
dh_next = da @ Whh.T # on to step t - 1
where Hs[:, t] holds the states of the whole batch, dHs[:, t] is the gradient from the output at step (the term , computed for every step at once as dHs = dlogits @ Why.T), and dh_next starts at zero. (Code counts steps from 0, so Hs[:, t] is the state after reading X[:, t].) Each line is a column formula transposed. Remember that Whh stores , so itself is Whh.T:
dhis anddais , one row per sequence;- transposed is (transposing reverses the product), and with rows that is
da @ Whh.T; dWhhis the gradient ofWhh, the transposed matrix, so it is the transpose of , which is . With a batch of rows that ish_prev.T @ da, and the matrix product also adds up over the sequences in the batch.
The forward pass has to store every state: memory grows with the sequence length, which is one reason to keep training windows short.
Now work a two-step example by hand where both steps have a loss, so that has both of its parts: one from its own loss and one arriving back from step 2.
A scalar RNN has , , and , and reads and . There is a loss at each step, , with targets and , and .
- Run the forward pass: , and .
- Run the backward pass: and , then (it has two parts) and .
- Find and as sums over the two steps.
(The output here is the state itself, so plays the role of .)
Keep four decimals throughout. To check your work, enter two numbers as a column: , then .
Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
The same bookkeeping in a longer sequence: which losses reach a state in the middle?
An RNN reads 10 characters with a loss at every step. Which losses contribute to ? Select all that apply.
Choose every option that applies, then check.
Now write the backward pass for a batch and prove it right with a gradient check, the test from module 6 that compares every gradient with central differences.
Implement backpropagation through time for the RNN you wrote in lesson 1, in the code layout.
rnn_backward(params, cache, dlogits)takes the cache fromrnn_forwardand the gradient of the loss with respect to every logit, and returns the gradients of every parameter and of the starting state.loss_and_grads(params, X, Y, h0=None)puts the forward pass, the loss and your backward pass together.
Provided: init_rnn and rnn_forward from mlref.rnn (the same function you wrote, with the same cache) and softmax_cross_entropy from mlref.nn. Taking dlogits as an input keeps the backward pass independent of the loss: the tests use it with a loss at every step and with a loss at the last step only. They check every gradient against central differences (numerical_gradient), the gradient check of module 6, and then against the reference implementation.
raise NotImplementedError, then press Run tests. Each check says what it expects.Clipping the gradient
Before training, one safety net. Every step back multiplies the gradient by and by the tanh slopes. Over a window of 32 steps that is 32 multiplications by nearly the same matrix, and now and then the product becomes very large (the next lesson works out exactly when). A gradient step is the learning rate times the gradient, so one huge gradient makes one huge step. It can throw the weights somewhere useless and undo everything learned so far.
The standard remedy is clipping by norm. Treat all the parameters' gradients as one long vector , and if its length exceeds a threshold , shrink it to length :
Read the as two cases. If , then , the minimum is 1, and nothing changes. If , the factor is , and the new length is . Every entry is multiplied by the same positive number, so the direction is unchanged; only the step length is capped.
For example, gradients for one parameter array and for another form the vector . Its length is . With , the length is above the threshold, so every entry is multiplied by : , of length 5. Length and similarity introduced this rescaling; this is where it earns its keep. Pascanu, Mikolov and Bengio (2013) analyzed exploding and vanishing gradients in RNNs and proposed clipping the norm this way. It is now a routine part of training large models, transformers included.
Clip a gradient that comes in two arrays, as a real model's does.
A model has two parameter arrays whose gradients are and . You clip the combined gradient to norm 2. Enter the five clipped entries as a column in the same order, the first array's three entries first. They come out exact, so no rounding is needed.
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Training a character RNN
You now have every piece: batches of windows and their shifted targets (predicting the next character), the forward pass (sequences and state), backpropagation through time, clipping, and an optimizer. The loop is the one from training a network:
- Cut a batch of random windows from the training text.
- Forward pass, loss, and backward pass through time.
- Clip the gradient norm.
- Update the weights with Adam.
- Every few hundred steps, measure the loss on the held-out text and print a sample.
We use hidden size 64. With characters in and out, the parameter count from the first lesson is
Training runs 1,500 steps, each on 32 windows of 32 characters. That is character predictions, about 17 passes over the 90,018 training characters (). It takes about half a minute in the browser.
Train a character RNN on Shakespeare, using the reference forward and backward pass from mlref (the same computations you wrote).
clip_grads(grads, max_norm): clip a dict of gradients by their combined norm, in place, and return the norm before clipping.evaluate(params, ids, seq_len): the mean loss on held-out text cut into whole windows.train(...): the lesson's training loop, with Adam, clipping, a live loss chart and a sample every 250 steps.
Provided: load_corpus, Vocab and sample_batch from mlref.text; init_rnn, rnn_forward, rnn_loss_and_grads and rnn_sample from mlref.rnn; Adam and softmax_cross_entropy from mlref.nn. The tests run 1,500 steps with hidden size 64, about half a minute. Watch the chart and the samples while it trains.
This one trains a model in your browser, so a run can take a minute or more. The charts update as it goes, and Stop ends it early.
raise NotImplementedError, then press Run tests. Each check says what it expects.In the reference run the validation loss starts at 4.18 (a little above , because the random initial weights make the logits slightly unequal), is already 1.99 after 250 steps, and ends at about 1.78, well below the bigram's 2.38. As a perplexity, : the model is about as unsure as a uniform choice among 6 characters, against 10.8 for the bigram. The samples tell the same story: speaker names in capitals followed by a colon and a line break, lines of about the right length, real short words and plausible invented long ones.
Look at what is missing too. Little carries over from one line to the next. Part of the reason is the training window: the model never sees more than 32 characters of context while it trains. The other part is the gradient itself. The learning signal that would teach the model to keep something for 40 steps has to travel back 40 steps, multiplied at every one of them. The next lesson follows it back and isolates the effect with an experiment.