ml.lab
Python sleeps until you run code
07 Recurrent networks

Lesson 4 of 5

Backpropagation through time

Training an RNN means backpropagating through the unrolled network. Each hidden state collects gradient from its own output and from the next step, and each shared weight collects one term per step. With clipping as a safety net, that is enough to train a character model on Shakespeare.

About 55 minutes
By the end you can
  • Derive the backward pass of an RNN, including the gradient that reaches each hidden state and the weight gradients as sums over time.

  • Run backpropagation through time by hand for a scalar RNN and check the result with finite differences.

  • Implement and gradient-check backpropagation through time for a batch of sequences.

  • Clip gradients by their combined norm, and explain why clipping keeps the direction of the step.

  • Train a character RNN on Shakespeare in the browser and read its loss against the baselines.

The RNN from the first lesson, used as the language model of the second, starts at a loss of about 4.2 nats, no better than guessing uniformly (ln⁡60≈4.09\ln 60 \approx 4.09). It has to learn. Its loss depends on the recurrent weights through every step. When it ends a sentence too early, the evidence that the sentence still needed its verb may be 40 characters back. How does the blame for a mistake at step 50 reach the weights as they were used at step 10?

The answer is the one module 6 gave for any network: backpropagation. Unroll the RNN into the deep network of the first lesson and run the backward pass through it. This is called backpropagation through time (BPTT). Most of it is familiar, with two new bookkeeping rules. This lesson derives it, checks it by hand and in code, and uses it to train a character model on Shakespeare. The next lesson looks at what happens to the gradient on its long way back, which explains why plain RNNs forget.

Unroll, then backpropagate

We derive the gradients for one sequence, with column vectors. First, the forward pass it undoes. For t=1,…,Tt = 1,\allowbreak \ldots,\allowbreak T it computes

at=Whhht−1+Wxhxt+b,ht=tanh⁡(at),zt=Whyht+by,pt=softmax(zt),\mathbf{a}_t = \mathbf{W}_{hh}\mathbf{h}_{t-1} + \mathbf{W}_{xh}\mathbf{x}_t + \mathbf{b}, \quad \mathbf{h}_t = \tanh(\mathbf{a}_t), \quad \mathbf{z}_t = \mathbf{W}_{hy}\mathbf{h}_t + \mathbf{b}_y, \quad \mathbf{p}_t = \mathrm{softmax}(\mathbf{z}_t),

and the loss L=∑t=1TℓtL = \sum_{t=1}^{T} \ell_t with ℓt=−ln⁡pt(yt)\ell_t = -\ln p_t(y_t). Here at\mathbf{a}_t is the pre-activation, yty_t is the id of the target character at step tt and yt\mathbf{y}_t is its one-hot vector. (Training uses the mean over positions instead of the sum; that divides every gradient by TT and changes nothing else.)

Two new rules make this different from a plain feed-forward network.

A state is used twice, so its gradient is a sum. ht\mathbf{h}_t feeds its own output zt\mathbf{z}_t and the next pre-activation at+1\mathbf{a}_{t+1}. The rule from backpropagation for a value used in two places applies: add the gradients from both uses.

A weight is used at every step, so its gradient is a sum over time. The same Whh\mathbf{W}_{hh} multiplies h0\mathbf{h}_0, then h1\mathbf{h}_1, and so on. Its total gradient is the sum of the gradients from each use.

Name the two gradients we carry backward, both vectors with nn entries:

  • gt=∂L/∂ht\mathbf{g}_t = \partial L/\partial\mathbf{h}_t, how much the loss changes per small change in each entry of the state ht\mathbf{h}_t;
  • δt=∂L/∂at\boldsymbol{\delta}_t = \partial L/\partial\mathbf{a}_t, the same for the pre-activation, just before the tanh.

Then walk from t=Tt = T down to t=1t = 1. At each step, three lines, each one move:

∂L∂zt=pt−ytsoftmax with cross-entropy (module 6)gt=Why⊤(pt−yt)+Whh⊤δt+1own output, plus the next step (δT+1=0)δt=gt⊙(1−ht2)through the tanh, entry by entry\begin{aligned} \frac{\partial L}{\partial \mathbf{z}_t} &= \mathbf{p}_t - \mathbf{y}_t && \text{softmax with cross-entropy (module 6)} \\ \mathbf{g}_t &= \mathbf{W}_{hy}^\top(\mathbf{p}_t - \mathbf{y}_t) + \mathbf{W}_{hh}^\top\boldsymbol{\delta}_{t+1} && \text{own output, plus the next step } (\boldsymbol{\delta}_{T+1} = \mathbf{0}) \\ \boldsymbol{\delta}_t &= \mathbf{g}_t \odot (1 - \mathbf{h}_t^2) && \text{through the tanh, entry by entry} \end{aligned}

where ⊙\odot multiplies entry by entry and ht2\mathbf{h}_t^2 squares each entry. The order matters: gt\mathbf{g}_t needs δt+1\boldsymbol{\delta}_{t+1}, which the walk computed one step earlier, because it goes backwards. At the last step nothing comes from the future, so δT+1=0\boldsymbol{\delta}_{T+1} = \mathbf{0}.

The weight gradients collect one term per step, each the same outer product as a module 6 layer:

∂L∂Whh=∑t=1Tδtht−1⊤,∂L∂Wxh=∑t=1Tδtxt⊤,∂L∂b=∑t=1Tδt,∂L∂Why=∑t=1T(pt−yt)ht⊤,∂L∂by=∑t=1T(pt−yt).\frac{\partial L}{\partial \mathbf{W}_{hh}} = \sum_{t=1}^{T} \boldsymbol{\delta}_t\mathbf{h}_{t-1}^\top, \quad \frac{\partial L}{\partial \mathbf{W}_{xh}} = \sum_{t=1}^{T} \boldsymbol{\delta}_t\mathbf{x}_t^\top, \quad \frac{\partial L}{\partial \mathbf{b}} = \sum_{t=1}^{T} \boldsymbol{\delta}_t, \quad \frac{\partial L}{\partial \mathbf{W}_{hy}} = \sum_{t=1}^{T} (\mathbf{p}_t - \mathbf{y}_t)\mathbf{h}_t^\top, \quad \frac{\partial L}{\partial \mathbf{b}_y} = \sum_{t=1}^{T} (\mathbf{p}_t - \mathbf{y}_t).

The part of gt−1\mathbf{g}_{t-1} that arrives from step tt is the rule to remember:

∂L∂ht−1∣through step t=Whh⊤(∂L∂ht⊙(1−ht2)).\left.\frac{\partial L}{\partial \mathbf{h}_{t-1}}\right|_{\text{through step } t} = \mathbf{W}_{hh}^\top\left(\frac{\partial L}{\partial \mathbf{h}_t} \odot (1 - \mathbf{h}_t^2)\right).

Through the tanh, then back through the recurrent matrix. It is the vector-Jacobian product of Jacobians and the chain rule for one step of the network. Notice that the slope 1−ht21 - \mathbf{h}_t^2 uses ht\mathbf{h}_t, the output of the step being undone, not ht−1\mathbf{h}_{t-1}.

Go slower: Every line, entry by entry

The tanh. Entry ii of ht\mathbf{h}_t depends only on entry ii of at\mathbf{a}_t, and ddatanh⁡a=1−tanh⁡2a\frac{d}{da}\tanh a = 1 - \tanh^2 a (derivatives as zoom). Since tanh⁡at,i=ht,i\tanh a_{t,i} = h_{t,i}, that slope is 1−ht,i21 - h_{t,i}^2. By the chain rule, ∂L∂at,i=∂L∂ht,i(1−ht,i2)\frac{\partial L}{\partial a_{t,i}} = \frac{\partial L}{\partial h_{t,i}}(1 - h_{t,i}^2), which for all ii at once is δt=gt⊙(1−ht2)\boldsymbol{\delta}_t = \mathbf{g}_t \odot (1 - \mathbf{h}_t^2).

The state, used twice. Entry jj of ht−1\mathbf{h}_{t-1} appears in two places. In the output, zt−1,k=∑j(Why)kjht−1,j+(by)kz_{t-1,k} = \sum_j (W_{hy})_{kj}h_{t-1,j} + (b_y)_k, so ∂zt−1,k/∂ht−1,j=(Why)kj\partial z_{t-1,k}/\partial h_{t-1,j} = (W_{hy})_{kj}. In the next step, at,i=∑j(Whh)ijht−1,j+∑q(Wxh)iqxt,q+bia_{t,i} = \sum_j (W_{hh})_{ij}h_{t-1,j} + \sum_q (W_{xh})_{iq}x_{t,q} + b_i, so ∂at,i/∂ht−1,j=(Whh)ij\partial a_{t,i}/\partial h_{t-1,j} = (W_{hh})_{ij}. The chain rule adds up every path from ht−1,jh_{t-1,j} to LL: ∂L∂ht−1,j=∑k∂L∂zt−1,k(Why)kj+∑i∂L∂at,i(Whh)ij.\frac{\partial L}{\partial h_{t-1,j}} = \sum_k \frac{\partial L}{\partial z_{t-1,k}}(W_{hy})_{kj} + \sum_i \frac{\partial L}{\partial a_{t,i}}(W_{hh})_{ij}. A sum over the first index of a matrix is a product with its transpose: ∑i(Whh)ijvi\sum_i (W_{hh})_{ij}v_i is entry jj of Whh⊤v\mathbf{W}_{hh}^\top\mathbf{v}. So in vectors this is gt−1=Why⊤(pt−1−yt−1)+Whh⊤δt\mathbf{g}_{t-1} = \mathbf{W}_{hy}^\top(\mathbf{p}_{t-1} - \mathbf{y}_{t-1}) + \mathbf{W}_{hh}^\top\boldsymbol{\delta}_t. We can use δt\boldsymbol{\delta}_t here because we walk backwards: it was computed one step earlier in the loop.

The shared weight. (Whh)ij(W_{hh})_{ij} appears in at,ia_{t,i} for every tt, each time multiplied by ht−1,jh_{t-1,j}. So ∂at,i/∂(Whh)ij=ht−1,j\partial a_{t,i}/\partial (W_{hh})_{ij} = h_{t-1,j}, and adding the uses over time, ∂L∂(Whh)ij=∑t=1T∂L∂at,i ht−1,j=∑t=1Tδt,i ht−1,j,\frac{\partial L}{\partial (W_{hh})_{ij}} = \sum_{t=1}^{T}\frac{\partial L}{\partial a_{t,i}}\,h_{t-1,j} = \sum_{t=1}^{T}\delta_{t,i}\,h_{t-1,j}, which is entry (i,j)(i,\allowbreak j) of ∑tδtht−1⊤\sum_t \boldsymbol{\delta}_t\mathbf{h}_{t-1}^\top. The same argument with xt,qx_{t,q} in place of ht−1,jh_{t-1,j} gives ∂L/∂Wxh\partial L/\partial\mathbf{W}_{xh}, and with the constant 1 gives ∂L/∂b\partial L/\partial\mathbf{b}. The output layer is module 6's linear layer applied at every step, summed.

The starting state. h0\mathbf{h}_0 is used only by a1\mathbf{a}_1, so ∂L/∂h0=Whh⊤δ1\partial L/\partial\mathbf{h}_0 = \mathbf{W}_{hh}^\top\boldsymbol{\delta}_1.

A worked example

The derivation has many symbols. A scalar RNN over two steps shows every one of them as a number. Take whh=0.5w_{hh} = 0.5, wxh=1w_{xh} = 1, b=0b = 0, h0=0h_0 = 0 and inputs x1=1x_1 = 1, x2=0.5x_2 = 0.5, and suppose the loss depends only on the last state, with ∂L/∂h2=1\partial L/\partial h_2 = 1 (for instance L=h2L = h_2). Write w=whhw = w_{hh} and u=wxhu = w_{xh}.

Forward. Step 1: a1=0.5×0+1×1=1a_1 = 0.5 \times 0 + 1 \times 1 = 1 and h1=tanh⁡1=0.7616h_1 = \tanh 1 = 0.7616. Step 2: a2=0.5×0.7616+1×0.5=0.3808+0.5=0.8808a_2 = 0.5 \times 0.7616 + 1 \times 0.5 = 0.3808 + 0.5 = 0.8808 and h2=tanh⁡(0.8808)=0.7068h_2 = \tanh(0.8808) = 0.7068.

Backward, step 2. The gradient arrives at the last state: g2=∂L/∂h2=1g_2 = \partial L/\partial h_2 = 1. Through the tanh, the slope at step 2 is 1−h22=1−0.70682=1−0.4996=0.50041 - h_2^2 = 1 - 0.7068^2 = 1 - 0.4996 = 0.5004, so δ2=g2×0.5004=0.5004\delta_2 = g_2 \times 0.5004 = 0.5004.

Backward, step 1. Step 1 has no output of its own, so g1g_1 has only the part that arrives from step 2: g1=w δ2=0.5×0.5004=0.2502g_1 = w\,\delta_2 = 0.5 \times 0.5004 = 0.2502. The slope at step 1 is 1−h12=1−0.76162=1−0.5800=0.42001 - h_1^2 = 1 - 0.7616^2 = 1 - 0.5800 = 0.4200, so δ1=0.2502×0.4200=0.1051\delta_1 = 0.2502 \times 0.4200 = 0.1051.

Weights. Each weight collects one term per step. For ww, the term at step tt is δt\delta_t times what ww multiplied there, ht−1h_{t-1}. For uu it is δt\delta_t times xtx_t:

∂L∂w=δ2h1+δ1h0=0.5004×0.7616+0.1051×0=0.3811,∂L∂u=δ2x2+δ1x1=0.5004×0.5+0.1051×1=0.2502+0.1051=0.3553.\frac{\partial L}{\partial w} = \delta_2 h_1 + \delta_1 h_0 = 0.5004 \times 0.7616 + 0.1051 \times 0 = 0.3811, \qquad \frac{\partial L}{\partial u} = \delta_2 x_2 + \delta_1 x_1 = 0.5004 \times 0.5 + 0.1051 \times 1 = 0.2502 + 0.1051 = 0.3553 .

The second term of ∂L/∂u\partial L/\partial u, 0.10510.1051, is the effect of uu through step 1, reaching h2h_2 by way of h1h_1: that is the path through time. Without it the answer would be 0.25020.2502, 30% too small.

A finite-difference check confirms both numbers: nudge ww (or uu) by 10−610^{-6} up and down, rerun the forward pass each time, and divide the change in LL by 2×10−62 \times 10^{-6}. The code below does the whole example, the backward walk and the check.

⌘+Enter runs · edit freelyPython sleeps until you run code

The two columns agree to four decimals: 0.38110.3811 and 0.35530.3553.

In code

With batches as rows and the code layout of the first lesson (Whh is Whh⊤\mathbf{W}_{hh}^\top), one backward step is

h_prev = Hs[:, t - 1] if t > 0 else h0   # (B, n): the states h_(t-1) that step t read
dh = dHs[:, t] + dh_next                 # (B, n): from the output at t, plus from step t + 1
da = dh * (1 - Hs[:, t] ** 2)            # through the tanh
dWhh += h_prev.T @ da                    # this step's share of the shared weight's gradient
dh_next = da @ Whh.T                     # on to step t - 1

where Hs[:, t] holds the states ht\mathbf{h}_t of the whole batch, dHs[:, t] is the gradient from the output at step tt (the term Why⊤(pt−yt)\mathbf{W}_{hy}^\top(\mathbf{p}_t - \mathbf{y}_t), computed for every step at once as dHs = dlogits @ Why.T), and dh_next starts at zero. (Code counts steps from 0, so Hs[:, t] is the state after reading X[:, t].) Each line is a column formula transposed. Remember that Whh stores Whh⊤\mathbf{W}_{hh}^\top, so Whh\mathbf{W}_{hh} itself is Whh.T:

  • dh is gt\mathbf{g}_t and da is δt\boldsymbol{\delta}_t, one row per sequence;
  • Whh⊤δt\mathbf{W}_{hh}^\top\boldsymbol{\delta}_t transposed is δt⊤Whh\boldsymbol{\delta}_t^\top\mathbf{W}_{hh} (transposing reverses the product), and with rows that is da @ Whh.T;
  • dWhh is the gradient of Whh, the transposed matrix, so it is the transpose of δtht−1⊤\boldsymbol{\delta}_t\mathbf{h}_{t-1}^\top, which is ht−1δt⊤\mathbf{h}_{t-1}\boldsymbol{\delta}_t^\top. With a batch of rows that is h_prev.T @ da, and the matrix product also adds up over the sequences in the batch.

The forward pass has to store every state: memory grows with the sequence length, which is one reason to keep training windows short.

Now work a two-step example by hand where both steps have a loss, so that g1g_1 has both of its parts: one from its own loss and one arriving back from step 2.

On paperBackpropagation through two steps

A scalar RNN has whh=w=0.5w_{hh} = w = 0.5, wxh=u=1w_{xh} = u = 1, b=0b = 0 and h0=0h_0 = 0, and reads x1=1x_1 = 1 and x2=−1x_2 = -1. There is a loss at each step, ℓt=12(ht−yt)2\ell_t = \tfrac{1}{2}(h_t - y_t)^2, with targets y1=1y_1 = 1 and y2=0y_2 = 0, and L=ℓ1+ℓ2L = \ell_1 + \ell_2.

  1. Run the forward pass: h1h_1, a2a_2 and h2h_2.
  2. Run the backward pass: g2=∂L/∂h2g_2 = \partial L/\partial h_2 and δ2\delta_2, then g1g_1 (it has two parts) and δ1\delta_1.
  3. Find ∂L/∂w\partial L/\partial w and ∂L/∂u\partial L/\partial u as sums over the two steps.

(The output here is the state itself, so ∂ℓt/∂ht=ht−yt\partial \ell_t/\partial h_t = h_t - y_t plays the role of Why⊤(pt−yt)\mathbf{W}_{hy}^\top(\mathbf{p}_t - \mathbf{y}_t).)

Keep four decimals throughout. To check your work, enter two numbers as a column: ∂L/∂w\partial L/\partial w, then ∂L/∂u\partial L/\partial u.

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

∂L/∂w, then ∂L/∂u

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

The same bookkeeping in a longer sequence: which losses reach a state in the middle?

Quick checkWhere the gradient at h₄ comes from

An RNN reads 10 characters with a loss at every step. Which losses contribute to g4=∂L/∂h4\mathbf{g}_4 = \partial L/\partial\mathbf{h}_4? Select all that apply.

Choose every option that applies, then check.

Now write the backward pass for a batch and prove it right with a gradient check, the test from module 6 that compares every gradient with central differences.

Code itBackpropagation through time, gradient-checked

Implement backpropagation through time for the RNN you wrote in lesson 1, in the code layout.

  1. rnn_backward(params, cache, dlogits) takes the cache from rnn_forward and the gradient of the loss with respect to every logit, and returns the gradients of every parameter and of the starting state.
  2. loss_and_grads(params, X, Y, h0=None) puts the forward pass, the loss and your backward pass together.

Provided: init_rnn and rnn_forward from mlref.rnn (the same function you wrote, with the same cache) and softmax_cross_entropy from mlref.nn. Taking dlogits as an input keeps the backward pass independent of the loss: the tests use it with a loss at every step and with a loss at the last step only. They check every gradient against central differences (numerical_gradient), the gradient check of module 6, and then against the reference implementation.

⌘+Enter runsPython sleeps until you run code
Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.

Clipping the gradient

Before training, one safety net. Every step back multiplies the gradient by Whh⊤\mathbf{W}_{hh}^\top and by the tanh slopes. Over a window of 32 steps that is 32 multiplications by nearly the same matrix, and now and then the product becomes very large (the next lesson works out exactly when). A gradient step is the learning rate times the gradient, so one huge gradient makes one huge step. It can throw the weights somewhere useless and undo everything learned so far.

The standard remedy is clipping by norm. Treat all the parameters' gradients as one long vector g\mathbf{g}, and if its length exceeds a threshold cc, shrink it to length cc:

g←g⋅min⁡(1,c∣g∣).\mathbf{g} \leftarrow \mathbf{g}\cdot\min\left(1, \frac{c}{|\mathbf{g}|}\right).

Read the min⁡\min as two cases. If ∣g∣≤c|\mathbf{g}| \le c, then c/∣g∣≥1c/|\mathbf{g}| \ge 1, the minimum is 1, and nothing changes. If ∣g∣>c|\mathbf{g}| > c, the factor is c/∣g∣<1c/|\mathbf{g}| < 1, and the new length is ∣g∣×c/∣g∣=c|\mathbf{g}| \times c/|\mathbf{g}| = c. Every entry is multiplied by the same positive number, so the direction is unchanged; only the step length is capped.

For example, gradients (3,4)(3,\allowbreak 4) for one parameter array and (12)(12) for another form the vector (3,4,12)(3,\allowbreak 4,\allowbreak 12). Its length is 32+42+122=9+16+144=169=13\sqrt{3^2 + 4^2 + 12^2} = \sqrt{9 + 16 + 144} = \sqrt{169} = 13. With c=5c = 5, the length is above the threshold, so every entry is multiplied by 5/135/13: (3,4,12)×5/13=(1.154,1.538,4.615)(3,\allowbreak 4,\allowbreak 12) \times 5/13 = (1.154,\allowbreak 1.538,\allowbreak 4.615), of length 5. Length and similarity introduced this rescaling; this is where it earns its keep. Pascanu, Mikolov and Bengio (2013) analyzed exploding and vanishing gradients in RNNs and proposed clipping the norm this way. It is now a routine part of training large models, transformers included.

Clip a gradient that comes in two arrays, as a real model's does.

Work it outClip by the global norm

A model has two parameter arrays whose gradients are (1,−2,2)(1,\allowbreak -2,\allowbreak 2) and (4,0)(4,\allowbreak 0). You clip the combined gradient to norm 2. Enter the five clipped entries as a column in the same order, the first array's three entries first. They come out exact, so no rounding is needed.

clipped gradient

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Training a character RNN

You now have every piece: batches of windows and their shifted targets (predicting the next character), the forward pass (sequences and state), backpropagation through time, clipping, and an optimizer. The loop is the one from training a network:

  1. Cut a batch of random windows from the training text.
  2. Forward pass, loss, and backward pass through time.
  3. Clip the gradient norm.
  4. Update the weights with Adam.
  5. Every few hundred steps, measure the loss on the held-out text and print a sample.

We use hidden size 64. With V=60V = 60 characters in and out, the parameter count from the first lesson is

64×(60+64+1)+60×(64+1)=64×125+60×65=8,000+3,900=11,900.64 \times (60 + 64 + 1) + 60 \times (64 + 1) = 64 \times 125 + 60 \times 65 = 8{,}000 + 3{,}900 = 11{,}900 .

Training runs 1,500 steps, each on 32 windows of 32 characters. That is 1,500×32×32=1,536,0001{,}500 \times 32 \times 32 = 1{,}536{,}000 character predictions, about 17 passes over the 90,018 training characters (1,536,000/90,018≈17.11{,}536{,}000 / 90{,}018 \approx 17.1). It takes about half a minute in the browser.

Code itTrain a character RNN on Shakespeare

Train a character RNN on Shakespeare, using the reference forward and backward pass from mlref (the same computations you wrote).

  1. clip_grads(grads, max_norm): clip a dict of gradients by their combined norm, in place, and return the norm before clipping.
  2. evaluate(params, ids, seq_len): the mean loss on held-out text cut into whole windows.
  3. train(...): the lesson's training loop, with Adam, clipping, a live loss chart and a sample every 250 steps.

Provided: load_corpus, Vocab and sample_batch from mlref.text; init_rnn, rnn_forward, rnn_loss_and_grads and rnn_sample from mlref.rnn; Adam and softmax_cross_entropy from mlref.nn. The tests run 1,500 steps with hidden size 64, about half a minute. Watch the chart and the samples while it trains.

This one trains a model in your browser, so a run can take a minute or more. The charts update as it goes, and Stop ends it early.

⌘+Enter runsPython sleeps until you run code
Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.

In the reference run the validation loss starts at 4.18 (a little above ln⁡60=4.09\ln 60 = 4.09, because the random initial weights make the logits slightly unequal), is already 1.99 after 250 steps, and ends at about 1.78, well below the bigram's 2.38. As a perplexity, e1.78≈5.9e^{1.78} \approx 5.9: the model is about as unsure as a uniform choice among 6 characters, against 10.8 for the bigram. The samples tell the same story: speaker names in capitals followed by a colon and a line break, lines of about the right length, real short words and plausible invented long ones.

Look at what is missing too. Little carries over from one line to the next. Part of the reason is the training window: the model never sees more than 32 characters of context while it trains. The other part is the gradient itself. The learning signal that would teach the model to keep something for 40 steps has to travel back 40 steps, multiplied at every one of them. The next lesson follows it back and isolates the effect with an experiment.