ml.lab
Python sleeps until you run code
08 LSTMs

Lesson 1 of 6

The cell state and its gates

An LSTM adds a second memory, the cell state, that is scaled and added to instead of being rewritten through a matrix at every step. Four small gates decide what to erase, what to write and what to show.

About 65 minutes
By the end you can
  • Explain why a memory that is added to can carry a signal across many steps, and name the three problems that the forget, input and output gates fix.

  • Write the LSTM equations with the shape of every vector and matrix, both in column form and in the stacked row form used in code.

  • Compute one LSTM step by hand.

  • Say what each of the four gates does, and predict how changing one of them changes the cell's behavior over several steps.

  • Implement a vectorized LSTM step with all four gates in one matrix product.

In module 7 a plain recurrent network failed a simple memory test. It sees one symbol, then a long stretch of unrelated tokens, then a cue, and it has to answer with the first symbol. With a short gap it learns the task. With a long gap it stays at chance, and training longer moves the limit out only slowly.

The reason was in the backward pass. The gradient that would teach the network to store the first symbol has to travel back to the first step, and on the way it is multiplied by one Jacobian per step. If each multiplication shrinks it by a factor of 0.8, then after 10 steps 0.810≈0.1070.8^{10} \approx 0.107 of it is left, after 20 steps 0.820≈0.01150.8^{20} \approx 0.0115, and after 40 steps 0.840≈0.000130.8^{40} \approx 0.00013. The network never finds out that the first symbol mattered.

This lesson builds the fix from one question: what would a memory have to look like for a signal to cross 40 steps intact? The answer is the cell state of a long short-term memory network, or LSTM. The name says what it is for: a short-term memory (the network's changing activations, as opposed to its weights) that can last a long time. Here we build the cell going forward: what it computes, what each part is for, and how to code it. The next lesson follows a gradient backwards through it.

A memory that is added to

To see what shape a good memory needs, start from what goes wrong with the plain RNN's. It rewrites its whole state at every step:

ht=tanh⁡(Whhht−1+Wxhxt+b).\mathbf{h}_t = \tanh(\mathbf{W}_{hh}\mathbf{h}_{t-1} + \mathbf{W}_{xh}\mathbf{x}_t + \mathbf{b}).

Here xt\mathbf{x}_t is the input at step tt, ht\mathbf{h}_t is the hidden state, Whh\mathbf{W}_{hh} and Wxh\mathbf{W}_{xh} are weight matrices and b\mathbf{b} is a bias vector. Even when the network wants to keep one number exactly as it is, that number goes through the matrix Whh\mathbf{W}_{hh} and the squashing function tanh⁡\tanh at every step. Keeping it intact for 40 steps would need Whh\mathbf{W}_{hh} to leave it alone and the tanh to stay in its straight middle part, while the same matrix is also doing all of the network's other work.

Now try the simplest memory that does keep things: an accumulator. Call the memory ct\mathbf{c}_t and let the network add something to it at every step:

ct=ct−1+ut,\mathbf{c}_t = \mathbf{c}_{t-1} + \mathbf{u}_t,

where ut\mathbf{u}_t is whatever the network computes to add at step tt. Follow the direct route from the old memory to the new one, with ut\mathbf{u}_t held fixed. Look at one entry first: ct,j=ct−1,j+ut,jc_{t,j} = c_{t-1,j} + u_{t,j}. If ct−1,jc_{t-1,j} goes up by a small amount, ct,jc_{t,j} goes up by exactly the same amount, so the slope is ∂ct,j/∂ct−1,j=1\partial c_{t,j}/\partial c_{t-1,j} = 1. Entry jj of ct\mathbf{c}_t does not depend on any other entry of ct−1\mathbf{c}_{t-1}, so all the other slopes are 0. Collect these slopes into the Jacobian of module 5 (row jj, column kk holds ∂ct,j/∂ct−1,k\partial c_{t,j}/\partial c_{t-1,k}) and you get the identity matrix: ∂ct/∂ct−1=I\partial \mathbf{c}_t / \partial \mathbf{c}_{t-1} = \mathbf{I}. Going backwards, whatever gradient arrives at ct\mathbf{c}_t is multiplied by I⊤=I\mathbf{I}^\top = \mathbf{I}, so it reaches ct−1\mathbf{c}_{t-1} unchanged, and after 40 such steps it reaches ct−40\mathbf{c}_{t-40} unchanged too. That is exactly the property we asked for. The first LSTM, published in 1997, used this idea for its memory, and its authors called the path the constant error carousel: an error signal can go round it for as long as needed without shrinking or growing.

A pure accumulator has three problems, and each one becomes a gate.

  1. It can never erase. Once something is added it stays forever, and the sum can grow without limit. Fix: multiply the old memory by a number between 0 and 1 before adding. That number is the forget gate.
  2. It adds at every step, including junk. Fix: multiply what is written by a number between 0 and 1, so the network can decide when to write. That is the input gate.
  3. Everything stored is always visible to the rest of the network, even when it is not relevant yet. Fix: multiply what is shown by a number between 0 and 1. That is the output gate.

A small example shows why the second gate matters. Say one unit should remember a +1+1 seen at step 1, and the next four steps would add noise of 0.40.4, −0.3-0.3, 0.50.5 and 0.20.2. The accumulator adds everything: 1+0.4−0.3+0.5+0.2=1.81 + 0.4 - 0.3 + 0.5 + 0.2 = 1.8, so the noise has corrupted the memory by 0.80.8. Now put an input gate of 1 on step 1 and 0.05 on the noisy steps. The noise adds up to 0.4−0.3+0.5+0.2=0.80.4 - 0.3 + 0.5 + 0.2 = 0.8 as before, but only 0.05×0.8=0.040.05 \times 0.8 = 0.04 of it gets in, and the memory ends at 1+0.04=1.041 + 0.04 = 1.04.

A glowing amber memory line runs left to right. First a coral valve on the line lets part of it drain away; after the valve, a blue gate lets violet material rise from below and join the line; after that, a teal window above the line shows a dimmer copy of it without taking anything off the lineThe cell state as a line that carries memory forward. First a valve decides how much stays, then a gate decides how much new content is added, and a window shows a squashed copy without removing anything.

To recap: the plain RNN pushes its memory through a matrix and a tanh at every step. An accumulator only adds to its memory, so a signal passes through it untouched, but it can neither erase, nor refuse to write, nor hide. The LSTM keeps the adding and puts a gate on each of those three jobs.

The full cell, with shapes

Before the equations, the question is what exactly gets written, kept and shown. One more piece completes the cell: the content that gets written, which the list above called "what is written", is computed by a fourth block, the candidate gt\mathbf{g}_t. It uses tanh, so it lies between −1-1 and 11, and the input gate scales it before it is added.

The widget below is one LSTM cell with a single unit, so every quantity on it is one number. It answers one question: what does each gate do to the memory, and what does it do to the output? The thick amber line along the top is the memory: ct−1c_{t-1} comes in on the left and ctc_t leaves on the right. The four blocks are the forget gate (coral), the input gate (blue), the candidate (violet) and the output gate (teal), each filled to its current value. The input xtx_t (white) and the previous output ht−1h_{t-1} (lime) run along the bottom and feed every block, and the output hth_t leaves on the right. Each gate adds up a weight times xtx_t, a weight times ht−1h_{t-1} and a bias bb, then squashes the sum, so raising a gate's bias opens it further; the sliders set these weights and biases, one gate at a time. The readouts give ctc_t, hth_t, the kept part (forget gate times ct−1c_{t-1}) and the written part (input gate times candidate). It starts at xt=1x_t = 1, ht−1=0.2h_{t-1} = 0.2 and ct−1=0.5c_{t-1} = 0.5, with the widget's own weights, and the formulas beside the figure show every value.

Now the equations. The LSTM keeps two vectors from step to step: the cell state ct\mathbf{c}_t (the memory) and the hidden state ht\mathbf{h}_t (the output, which is also fed back in). At step tt it reads the input xt\mathbf{x}_t and the previous output ht−1\mathbf{h}_{t-1}, and computes four vectors:

ft=σ(Wfxt+Ufht−1+bf)forget gateit=σ(Wixt+Uiht−1+bi)input gategt=tanh⁡(Wgxt+Ught−1+bg)candidateot=σ(Woxt+Uoht−1+bo)output gate\begin{aligned} \mathbf{f}_t &= \sigma(\mathbf{W}_f\mathbf{x}_t + \mathbf{U}_f\mathbf{h}_{t-1} + \mathbf{b}_f) && \text{forget gate} \\ \mathbf{i}_t &= \sigma(\mathbf{W}_i\mathbf{x}_t + \mathbf{U}_i\mathbf{h}_{t-1} + \mathbf{b}_i) && \text{input gate} \\ \mathbf{g}_t &= \tanh(\mathbf{W}_g\mathbf{x}_t + \mathbf{U}_g\mathbf{h}_{t-1} + \mathbf{b}_g) && \text{candidate} \\ \mathbf{o}_t &= \sigma(\mathbf{W}_o\mathbf{x}_t + \mathbf{U}_o\mathbf{h}_{t-1} + \mathbf{b}_o) && \text{output gate} \end{aligned}

Then it updates the memory and computes the output:

ct=ft⊙ct−1+it⊙gt,ht=ot⊙tanh⁡(ct).\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \mathbf{g}_t, \qquad \mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{c}_t).

The symbol σ\sigma is the sigmoid σ(z)=1/(1+e−z)\sigma(z) = 1/(1 + e^{-z}), applied to every entry, and ⊙\odot is the elementwise product: multiply matching entries. Many sources write the candidate as c~t\tilde{\mathbf{c}}_t instead of gt\mathbf{g}_t; it is the same vector. Each block has its own W\mathbf{W} on the input and its own U\mathbf{U} on the previous output, a common naming for LSTMs: they play the roles of the plain RNN's Wxh\mathbf{W}_{xh} and Whh\mathbf{W}_{hh}. (Module 4 happened to use these two letters the other way round; in this module U\mathbf{U} is always a weight on the previous output.)

The shapes, with input size mm and hidden size nn:

SymbolShapeWhat it is
xt\mathbf{x}_tmmthe input at step tt
ht−1,ht\mathbf{h}_{t-1},\allowbreak \mathbf{h}_tnnprevious and new output
ct−1,ct\mathbf{c}_{t-1},\allowbreak \mathbf{c}_tnnprevious and new cell state
Wf,Wi,Wg,Wo\mathbf{W}_f,\allowbreak \mathbf{W}_i,\allowbreak \mathbf{W}_g,\allowbreak \mathbf{W}_on×mn \times mweights on the input
Uf,Ui,Ug,Uo\mathbf{U}_f,\allowbreak \mathbf{U}_i,\allowbreak \mathbf{U}_g,\allowbreak \mathbf{U}_on×nn \times nweights on the previous output
bf,bi,bg,bo\mathbf{b}_f,\allowbreak \mathbf{b}_i,\allowbreak \mathbf{b}_g,\allowbreak \mathbf{b}_onnbiases
ft,it,gt,ot\mathbf{f}_t,\allowbreak \mathbf{i}_t,\allowbreak \mathbf{g}_t,\allowbreak \mathbf{o}_tnngates and candidate

Check one line: Wfxt\mathbf{W}_f\mathbf{x}_t is (n×m)(n \times m) times (m)(m), a vector of length nn; Ufht−1\mathbf{U}_f\mathbf{h}_{t-1} is (n×n)(n \times n) times (n)(n), also length nn; add bf\mathbf{b}_f (length nn) and squash, and ft\mathbf{f}_t has length nn. Every gate has one entry per unit. Unit jj has its own forget value ft,jf_{t,j}, its own input value it,ji_{t,j} and its own output value ot,jo_{t,j}, and the products act entry by entry. So an LSTM layer is nn registers side by side. They interact only through ht−1\mathbf{h}_{t-1}, which every gate reads in full.

Three design choices are worth a sentence each.

  • Gates use the sigmoid because they multiply: a gate value is a fraction, and the sigmoid's range from 0 to 1 is exactly the range of fractions.
  • The candidate uses tanh because it is content, and content needs a sign. Its range from −1-1 to 11 keeps each write bounded, and it is centered on zero.
  • The output applies tanh to the cell state because ct\mathbf{c}_t itself is not bounded. Repeated writes can push it well past 1 (the lesson after next uses this to count), so it is squashed back into (−1,1)(-1,\allowbreak 1) before it is shown.

In short: three sigmoid gates and one tanh candidate, all computed from xt\mathbf{x}_t and ht−1\mathbf{h}_{t-1}; the memory keeps a gated part of itself and adds a gated candidate; the output is a gated, squashed view of the memory.

One step by hand

Every later lesson in this module works with numbers like these, so it is worth doing one step completely. Take a single unit (m=n=1m = n = 1), so every weight is a number. Write ww for the weight on xtx_t, uu for the weight on ht−1h_{t-1} and bb for the bias:

ww (times xtx_t)uu (times ht−1h_{t-1})bb
forget120
input2−2-2−1-1
candidate100
output−1-12−1-1

The cell holds ct−1=1.5c_{t-1} = 1.5 from earlier, the previous output was ht−1=0.5h_{t-1} = 0.5, and the new input is xt=1x_t = 1.

The pre-activations. Each gate first computes wxt+uht−1+bw x_t + u h_{t-1} + b:

  • forget: 1(1)+2(0.5)+0=1+1+0=21(1) + 2(0.5) + 0 = 1 + 1 + 0 = 2
  • input: 2(1)−2(0.5)−1=2−1−1=02(1) - 2(0.5) - 1 = 2 - 1 - 1 = 0
  • candidate: 1(1)+0(0.5)+0=11(1) + 0(0.5) + 0 = 1
  • output: −1(1)+2(0.5)−1=−1+1−1=−1-1(1) + 2(0.5) - 1 = -1 + 1 - 1 = -1

The squashing. Sigmoid for the three gates, tanh for the candidate. With e≈2.7183e \approx 2.7183:

  • ft=σ(2)=11+e−2=11+0.1353=11.1353≈0.8808f_t = \sigma(2) = \dfrac{1}{1 + e^{-2}} = \dfrac{1}{1 + 0.1353} = \dfrac{1}{1.1353} \approx 0.8808
  • it=σ(0)=11+e0=12=0.5i_t = \sigma(0) = \dfrac{1}{1 + e^{0}} = \dfrac{1}{2} = 0.5
  • gt=tanh⁡(1)=e−e−1e+e−1=2.7183−0.36792.7183+0.3679=2.35043.0862≈0.7616g_t = \tanh(1) = \dfrac{e - e^{-1}}{e + e^{-1}} = \dfrac{2.7183 - 0.3679}{2.7183 + 0.3679} = \dfrac{2.3504}{3.0862} \approx 0.7616
  • ot=σ(−1)=11+e1=13.7183≈0.2689o_t = \sigma(-1) = \dfrac{1}{1 + e^{1}} = \dfrac{1}{3.7183} \approx 0.2689

The memory. Keep 88% of the old value and add half the candidate:

ct=ft ct−1+it gt=(0.8808)(1.5)+(0.5)(0.7616)=1.3212+0.3808=1.7020.c_t = f_t\,c_{t-1} + i_t\,g_t = (0.8808)(1.5) + (0.5)(0.7616) = 1.3212 + 0.3808 = 1.7020.

The output. Squash the memory (a calculator gives tanh⁡(1.7020)≈0.9357\tanh(1.7020) \approx 0.9357), then show about a quarter of it:

ht=ottanh⁡(ct)=(0.2689)(0.9357)≈0.2516.h_t = o_t \tanh(c_t) = (0.2689)(0.9357) \approx 0.2516.

The memory grew from 1.5 to 1.70, but the output is small, because the output gate is mostly closed. The cell is holding something it is not yet showing.

Now do one yourself with new weights and inputs. The steps are the same four: pre-activations, squash, memory, output.

On paperOne LSTM step by hand

A one-unit LSTM (m=n=1m = n = 1) has these weights, where ww multiplies xtx_t, uu multiplies ht−1h_{t-1} and bb is the bias:

wwuubb
forget110
input10−2-2
candidate−1-1−1-10.5
output0.5−1-10

The input is xt=2x_t = 2, the previous output is ht−1=−1h_{t-1} = -1 and the previous cell state is ct−1=0.8c_{t-1} = 0.8.

  1. Compute the four pre-activations, then ftf_t, iti_t, gtg_t and oto_t. Use a calculator for σ\sigma and tanh⁡\tanh and keep four decimals.
  2. Compute ctc_t and hth_t.
  3. Which is larger in size, the part of the old memory that was kept or the part that was written? What sign does the write have, and why?

The check at the top takes ctc_t and hth_t, in that order, to 3 decimal places.

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

c_t and h_t

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

The practice set below generates as many fresh steps as you want, each with a full worked solution.

Practice setLSTM step reps
3 correct in a row completes the set. A miss starts the count again.

A scalar LSTM (hidden size 1) has these weights:

gateww (times xtx_t)uu (times ht−1h_{t-1})bb (bias)
forget ftf_t101
input iti_t1-10
candidate gtg_t2-1-1
output oto_t011
ft=σ(wfxt+ufht−1+bf),it=σ(wixt+uiht−1+bi),gt=tanh⁡(wgxt+ught−1+bg),ot=σ(woxt+uoht−1+bo),ct=ft ct−1+it gt,ht=ottanh⁡(ct).\begin{aligned} f_t &= \sigma(w_f x_t + u_f h_{t-1} + b_f), & i_t &= \sigma(w_i x_t + u_i h_{t-1} + b_i), \\ g_t &= \tanh(w_g x_t + u_g h_{t-1} + b_g), & o_t &= \sigma(w_o x_t + u_o h_{t-1} + b_o), \\ c_t &= f_t\,c_{t-1} + i_t\,g_t, & h_t &= o_t\tanh(c_t). \end{aligned}

With xt=0x_t = 0, ht−1=1h_{t-1} = 1 and ct−1=1c_{t-1} = 1, find ctc_t and hth_t to 3 decimal places. Enter them as a column, ctc_t first.

c_t, h_t

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

What each gate is for

With the arithmetic in hand, here is the job of each piece in plain words.

  • Forget gate ft\mathbf{f}_t: keep or erase. Near 1, the old memory survives this step. Near 0, it is wiped.
  • Input gate it\mathbf{i}_t: write or ignore. Near 1, the candidate is added to the memory. Near 0, this step's content is ignored.
  • Candidate gt\mathbf{g}_t: what to write. A value between −1-1 and 11 for each unit, computed from the current input and the previous output.
  • Output gate ot\mathbf{o}_t: reveal or hide. Near 1, the squashed memory is passed on as ht\mathbf{h}_t to the next layer and to the next step's gates. Near 0, the memory is kept but not shown.

The forget and input gates together decide what happens to the memory. Put each at its extremes in ct=ftct−1+itgtc_t = f_t c_{t-1} + i_t g_t and you get four behaviors:

ftf_titi_tct=ftct−1+itgtc_t = f_t c_{t-1} + i_t g_tBehavior
101⋅ct−1+0⋅gt=ct−11 \cdot c_{t-1} + 0 \cdot g_t = c_{t-1}hold
010⋅ct−1+1⋅gt=gt0 \cdot c_{t-1} + 1 \cdot g_t = g_toverwrite
111⋅ct−1+1⋅gt=ct−1+gt1 \cdot c_{t-1} + 1 \cdot g_t = c_{t-1} + g_taccumulate
000⋅ct−1+0⋅gt=00 \cdot c_{t-1} + 0 \cdot g_t = 0reset

A concrete case: a character-level model reading text could use one unit to track whether it is inside a quotation. At an opening quotation mark, the input gate opens and the candidate is +1+1. Inside the quote, the forget gate stays near 1 and the input gate stays near 0: hold. At the closing mark, the forget gate drops to 0: reset. The output gate opens whenever the next character depends on being inside a quote. Nobody writes these rules. Training finds weights that produce them, when they help predict the next character, and in the training lesson you will look for units like this in a model you train.

Notice what the gates can see: the current input xt\mathbf{x}_t and the previous output ht−1\mathbf{h}_{t-1}. They do not read the cell state directly. A cell whose output gate has been closed for a while is invisible even to its own gates. (Variants with "peephole" connections let the gates read the cell state; the last lesson of this module describes them.)

Here is a situation to test the roles against.

Quick checkWhich gate fixes it

A trained LSTM reads long product reviews. One unit is supposed to hold the product's name, seen in the first sentence, until the last sentence. You find that this unit's cell value drifts as the review goes on, because later words keep adding to it. Which change fixes the drift most directly?

Choose one answer, then check.

Four steps in a row: write, keep, reveal

One step shows the arithmetic, but the behaviors in the table only appear across several steps. The figure below answers: what do the gates do, step after step, when a cell has to remember something and report it later? It unrolls one cell with a single unit over four time steps, one column per step. Along the top runs the memory cc (amber) with its value after each step. Below it are the four gates as bars that fill from 0 (empty) to 1 (full); the candidate fills from its middle line, to the right when it is positive. Then comes the output hh (lime). Under the columns are the two inputs of each step, because this cell reads two input lines (m=2m = 2): a value xx to remember and a cue rr that asks for it. The weights make every gate a switch, with pre-activations of +7+7 or −7-7:

it=σ(14xt−7),ft=σ(7),gt=tanh⁡(7xt),ot=σ(14rt−7).i_t = \sigma(14x_t - 7), \qquad f_t = \sigma(7), \qquad g_t = \tanh(7x_t), \qquad o_t = \sigma(14r_t - 7).

So the input gate is σ(14−7)=σ(7)\sigma(14 - 7) = \sigma(7) when xt=1x_t = 1 and σ(−7)\sigma(-7) when xt=0x_t = 0; the forget gate is always σ(7)\sigma(7); the output gate opens only when the cue is 1. Every weight on ht−1h_{t-1} is 0, and the cell starts from c0=h0=0c_0 = h_0 = 0.

Why does this cell need the cue at all? Because the gates read only xtx_t and ht−1h_{t-1}. At steps 2, 3 and 4 the value input is 0 and ht−1h_{t-1} is 0.00070.0007 each time, so without the cue those three steps would look exactly the same to every gate, and the output gate could not open at step 4 without also opening at steps 2 and 3. The restriction from the previous paragraph, gates cannot see the cell, is a real design constraint.

To recap the section: writing needs the input gate open and a nonzero candidate; keeping needs the forget gate near 1 and nothing written; revealing needs the output gate open. A value kept with the output gate shut is stored silently.

Four gates, one matrix

So far every gate had its own weights. Code computes all four at once, because the four gate equations have the same form: a matrix times xt\mathbf{x}_t, plus a matrix times ht−1\mathbf{h}_{t-1}, plus a bias. Stack the four pre-activations into one long vector of length 4n4n, and join the input and the previous output into one vector of length m+nm + n.

In code, as everywhere in this course after module 2, a batch of BB examples is stored as rows. So the step is written with row vectors:

z=[xt,ht−1] W+b,\mathbf{z} = [\mathbf{x}_t, \mathbf{h}_{t-1}]\,\mathbf{W} + \mathbf{b},

where [xt,ht−1][\mathbf{x}_t,\allowbreak \mathbf{h}_{t-1}] is the concatenation (shape B×(m+n)B \times (m + n) for a batch), W\mathbf{W} has shape (m+n)×4n(m + n) \times 4n, and b\mathbf{b} has length 4n4n and is added to every row. The result z\mathbf{z} has shape B×4nB \times 4n. Its columns come in four blocks of nn, in the order input, forget, candidate, output:

z=[ zi  ∣  zf  ∣  zg  ∣  zo ],i=σ(zi), f=σ(zf), g=tanh⁡(zg), o=σ(zo).\mathbf{z} = \big[\, \mathbf{z}_i \;\big|\; \mathbf{z}_f \;\big|\; \mathbf{z}_g \;\big|\; \mathbf{z}_o \,\big], \qquad \mathbf{i} = \sigma(\mathbf{z}_i),\ \mathbf{f} = \sigma(\mathbf{z}_f),\ \mathbf{g} = \tanh(\mathbf{z}_g),\ \mathbf{o} = \sigma(\mathbf{z}_o).

Within W\mathbf{W}, the first mm rows multiply xt\mathbf{x}_t and the last nn rows multiply ht−1\mathbf{h}_{t-1}. The forget block of columns therefore holds Wf⊤\mathbf{W}_f^\top on top of Uf⊤\mathbf{U}_f^\top: transposes, because code uses rows where the math above uses columns. The numbers are the same.

W=[Wi⊤Wf⊤Wg⊤Wo⊤Ui⊤Uf⊤Ug⊤Uo⊤]←m rows←n rows\mathbf{W} = \begin{bmatrix} \mathbf{W}_i^\top & \mathbf{W}_f^\top & \mathbf{W}_g^\top & \mathbf{W}_o^\top \\ \mathbf{U}_i^\top & \mathbf{U}_f^\top & \mathbf{U}_g^\top & \mathbf{U}_o^\top \end{bmatrix} \begin{matrix} \leftarrow m \text{ rows} \\ \leftarrow n \text{ rows} \end{matrix}

Here is the worked step from above in this layout. With m=n=1m = n = 1, W\mathbf{W} is 2×42 \times 4: row 1 holds the four ww weights and row 2 the four uu weights, with the columns in the order input, forget, candidate, output. Read the weight table column by column:

W=[211−1−2202],b=[−100−1].\mathbf{W} = \begin{bmatrix} 2 & 1 & 1 & -1 \\ -2 & 2 & 0 & 2 \end{bmatrix}, \qquad \mathbf{b} = \begin{bmatrix} -1 & 0 & 0 & -1 \end{bmatrix}.

The joined input is [xt,ht−1]=[1,0.5][x_t,\allowbreak h_{t-1}] = [1,\allowbreak 0.5], a 1×21 \times 2 row. Each entry of z\mathbf{z} is that row dotted with one column of W\mathbf{W}, plus that column's bias:

z=[ 1(2)+0.5(−2)−1,  1(1)+0.5(2)+0,  1(1)+0.5(0)+0,  1(−1)+0.5(2)−1 ]=[ 0, 2, 1, −1 ].\mathbf{z} = \big[\, 1(2) + 0.5(-2) - 1,\ \ 1(1) + 0.5(2) + 0,\ \ 1(1) + 0.5(0) + 0,\ \ 1(-1) + 0.5(2) - 1 \,\big] = [\,0,\ 2,\ 1,\ -1\,].

These are the four pre-activations from the hand calculation (input 0, forget 2, candidate 1, output −1-1), now from one product.

One product of shape (B,m+n)(B,\allowbreak m + n) times (m+n,4n)(m + n,\allowbreak 4n) replaces eight separate products, which is much faster on real hardware. The order [i,f,g,o][\mathbf{i},\allowbreak \mathbf{f},\allowbreak \mathbf{g},\allowbreak \mathbf{o}] is the one PyTorch uses and the one in this course's reference code, mlref/lstm.py. PyTorch stores the input and hidden parts as two separate matrices, each with its own bias vector, but the blocks come in the same order. Other libraries may order the blocks differently, so check before loading someone else's weights.

Before writing code, practice reading the blocks with two units, where each block has two entries.

Work it outRead the stacked gates

An LSTM layer with n=2n = 2 units computes the stacked pre-activation z=(0, 2, 3, −3, 0.5, −1, 1, 0)\mathbf{z} = (0,\allowbreak \ 2,\allowbreak \ 3,\allowbreak \ -3,\allowbreak \ 0.5,\allowbreak \ -1,\allowbreak \ 1,\allowbreak \ 0), with blocks in the order [i,f,g,o][\mathbf{i},\allowbreak \mathbf{f},\allowbreak \mathbf{g},\allowbreak \mathbf{o}]. The previous cell state is ct−1=(1,−2)\mathbf{c}_{t-1} = (1,\allowbreak -2). Find ct\mathbf{c}_t. Enter it as a column, unit 1 on top, to 3 decimal places.

c_t

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Now write the step. It is the core of everything else in this module: the next lessons follow its gradient, design weights for it, differentiate it and train it. The starter already contains the worked example above as a first check.

Code itOne LSTM step, four gates in one product

Write the LSTM step you will use for the rest of the module, in the stacked layout of mlref/lstm.py.

  1. init_lstm(input_size, hidden_size, rng, forget_bias=1.0) returns a dict with W of shape (m+n,4n)(m + n,\allowbreak 4n), drawn from a normal distribution with standard deviation 1/m+n1/\sqrt{m + n}, and b of shape (4n,)(4n,\allowbreak ), zero except for the forget block, which is forget_bias.
  2. lstm_step(x, h_prev, c_prev, W, b) takes a batch as rows (x of shape (B,m)(B,\allowbreak m), h_prev and c_prev of shape (B,n)(B,\allowbreak n)) and returns h, c and a dict of the four gate values. Use one matrix product for all four gates, with column blocks in the order [i,f,g,o][\mathbf{i},\allowbreak \mathbf{f},\allowbreak \mathbf{g},\allowbreak \mathbf{o}] and the rows of W for xt\mathbf{x}_t above the rows for ht−1\mathbf{h}_{t-1}.
⌘+Enter runsPython sleeps until you run code
Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.

Next, the payoff: why a memory built this way lets a gradient cross 40 steps when the plain RNN's cannot.

Next: Why gradients survive