ml.lab
Python sleeps until you run code
08 LSTMs

Lesson 6 of 6

Design your own

The LSTM is one point in a space of designs. Build the gated recurrent unit from scratch, see which other changes matter and which do not, and learn to test a design idea with an experiment you can trust.

About 60 minutes
By the end you can
  • Write the GRU equations, compare them with the LSTM's, and count the parameters of both.

  • Implement a GRU step forward and backward and verify it with a gradient check.

  • Describe peephole connections, stacked layers, hidden size, training length and dropout as design decisions with costs.

  • Design a fair experiment (one change, several seeds, held-out measurement), run it, and write up what it shows.

Every piece of the LSTM is a decision someone made. Four gates, not three or five. A separate cell state next to the hidden state. A tanh on the way out. Gates that look at the input and the previous output but not at the cell. Each decision could have gone another way, and over the years almost every alternative has been tried.

This lesson builds the most successful alternative, the gated recurrent unit or GRU, from scratch. Then it surveys the other knobs and what is known about them. It ends with the part that transfers to any model you will ever work on: how to find out whether a design change actually helps, with an experiment whose result you can trust.

The GRU: two gates, one state

The GRU, introduced by Cho and colleagues in 2014, removes the separate cell. It keeps a single state ht\mathbf{h}_t and updates it with two gates:

zt=σ(Wzxt+Uzht−1+bz)update gatert=σ(Wrxt+Urht−1+br)reset gateh~t=tanh⁡(Wcxt+Uc(rt⊙ht−1)+bc)candidateht=zt⊙ht−1+(1−zt)⊙h~t\begin{aligned} \mathbf{z}_t &= \sigma(\mathbf{W}_z\mathbf{x}_t + \mathbf{U}_z\mathbf{h}_{t-1} + \mathbf{b}_z) && \text{update gate} \\ \mathbf{r}_t &= \sigma(\mathbf{W}_r\mathbf{x}_t + \mathbf{U}_r\mathbf{h}_{t-1} + \mathbf{b}_r) && \text{reset gate} \\ \tilde{\mathbf{h}}_t &= \tanh\big(\mathbf{W}_c\mathbf{x}_t + \mathbf{U}_c(\mathbf{r}_t \odot \mathbf{h}_{t-1}) + \mathbf{b}_c\big) && \text{candidate} \\ \mathbf{h}_t &= \mathbf{z}_t \odot \mathbf{h}_{t-1} + (1 - \mathbf{z}_t) \odot \tilde{\mathbf{h}}_t \end{aligned}

The shapes follow the LSTM's: xt\mathbf{x}_t has mm entries, ht\mathbf{h}_t and every gate have nn, the W\mathbf{W} matrices are n×mn \times m and the U\mathbf{U} matrices n×nn \times n. The subscript cc marks the candidate's weights, and h~t\tilde{\mathbf{h}}_t (read "h tilde") is the candidate itself. Read the equations against the LSTM's.

  • The update gate plays the forget gate. Unit jj keeps the fraction zt,jz_{t,j} of its old value and fills the rest, 1−zt,j1 - z_{t,j}, with the candidate. In LSTM terms, the forget gate is zt\mathbf{z}_t and the input gate is forced to be 1−zt1 - \mathbf{z}_t: the gates are coupled, so keeping more always means writing less.
  • The reset gate decides what the candidate may see. With rt\mathbf{r}_t near 0, the candidate is computed as if the past were blank, from xt\mathbf{x}_t alone. The LSTM has nothing exactly like it.
  • There is no output gate and no separate cell. The whole state is always visible to the next layer and to the gates.

One warning about conventions. Sources differ on whether zt\mathbf{z}_t or 1−zt1 - \mathbf{z}_t multiplies the old state. This lesson follows PyTorch, where zt\mathbf{z}_t multiplies ht−1\mathbf{h}_{t-1}, so that zt\mathbf{z}_t lines up with the forget gate. The two conventions describe the same model with the sign of the gate's weights and bias flipped. PyTorch also applies the reset gate after the matrix product, rt⊙(Ucht−1+b′)\mathbf{r}_t \odot (\mathbf{U}_c\mathbf{h}_{t-1} + \mathbf{b}') with an extra bias vector b′\mathbf{b}', which is a slightly different model. Check the convention before comparing numbers across libraries.

One step with numbers

The equations run in a fixed order: both gates first, then the candidate (which needs the reset gate), then the mix (which needs the update gate and the candidate). Take a one-unit GRU with xt=1x_t = 1 and ht−1=0.5h_{t-1} = 0.5, with weights (w,u,b)(w,\allowbreak u,\allowbreak b) of (−1,2,1)(-1,\allowbreak 2,\allowbreak 1) for the update gate, (1,−2,0)(1,\allowbreak -2,\allowbreak 0) for the reset gate and (2,1,0)(2,\allowbreak 1,\allowbreak 0) for the candidate:

  • update: −1(1)+2(0.5)+1=−1+1+1=1-1(1) + 2(0.5) + 1 = -1 + 1 + 1 = 1, so zt=σ(1)=1/(1+e−1)=1/1.3679≈0.7311z_t = \sigma(1) = 1/(1 + e^{-1}) = 1/1.3679 \approx 0.7311
  • reset: 1(1)−2(0.5)+0=1−1+0=01(1) - 2(0.5) + 0 = 1 - 1 + 0 = 0, so rt=σ(0)=0.5r_t = \sigma(0) = 0.5
  • candidate: the reset gate halves the old state before the candidate sees it, rtht−1=0.5×0.5=0.25r_t h_{t-1} = 0.5 \times 0.5 = 0.25, so the pre-activation is 2(1)+1(0.25)+0=2.252(1) + 1(0.25) + 0 = 2.25 and h~t=tanh⁡(2.25)≈0.9780\tilde{h}_t = \tanh(2.25) \approx 0.9780
  • new state: keep the fraction ztz_t of the old value and fill the rest, 1−zt=1−0.7311=0.26891 - z_t = 1 - 0.7311 = 0.2689, with the candidate. With five decimals, so that the sum checks, ht=(0.73106)(0.5)+(0.26894)(0.97803)=0.36553+0.26303=0.62856≈0.6286h_t = (0.73106)(0.5) + (0.26894)(0.97803) = 0.36553 + 0.26303 = 0.62856 \approx 0.6286

The unit kept 73% of its old value and moved 27% of the way from 0.50.5 toward the candidate 0.9780.978.

Gradients, and a bound on the state

Hold the gates and the candidate fixed and differentiate the last equation, entry by entry: ht,j=zt,j ht−1,j+(1−zt,j) h~t,jh_{t,j} = z_{t,j}\,h_{t-1,j} + (1 - z_{t,j})\,\tilde{h}_{t,j} has slope zt,jz_{t,j} with respect to ht−1,jh_{t-1,j} and 0 with respect to every other entry. So ∂ht/∂ht−1\partial \mathbf{h}_t / \partial \mathbf{h}_{t-1} along the direct path is diag(zt)\mathrm{diag}(\mathbf{z}_t). That is the same structure as the LSTM's cell path, with zt\mathbf{z}_t in place of ft\mathbf{f}_t (why gradients survive), and it needs the same care at initialization: with a bias of 0, z≈0.5z \approx 0.5 and the path halves the gradient at every step. A positive keep bias on bz\mathbf{b}_z opens it, exactly as the forget bias did in the hand-design lesson. The GRU has more indirect paths than the LSTM, because ht−1\mathbf{h}_{t-1} also feeds both gates and, through the reset gate, the candidate.

The coupling has a consequence that the LSTM does not share. If ∣ht−1,j∣<1|h_{t-1,j}| < 1, then ht,jh_{t,j} is a weighted average of two numbers inside (−1,1)(-1,\allowbreak 1) (the old value, and a tanh), with weights zt,jz_{t,j} and 1−zt,j1 - z_{t,j} that are positive and sum to 1. An average of numbers inside (−1,1)(-1,\allowbreak 1) stays inside: ∣ht,j∣≤zt,j∣ht−1,j∣+(1−zt,j)∣h~t,j∣<zt,j+(1−zt,j)=1|h_{t,j}| \le z_{t,j}|h_{t-1,j}| + (1 - z_{t,j})|\tilde{h}_{t,j}| < z_{t,j} + (1 - z_{t,j}) = 1. Starting from h0=0\mathbf{h}_0 = \mathbf{0}, a GRU's state never leaves (−1,1)(-1,\allowbreak 1). The LSTM cell in the hand-design lesson counted to 12 by adding 1 at every step; a GRU unit cannot do that. Weiss, Goldberg and Yahav (2018) showed the difference empirically: LSTMs trained to recognize strings of the form anbna^n b^n (some number of a's followed by exactly as many b's) learned to count, and kept working well beyond the lengths they were trained on; GRUs did not.

Parameters

Three blocks instead of four: 3n(n+m)+3n3n(n + m) + 3n, three quarters of an LSTM of the same sizes. For n=128n = 128 and m=32m = 32 that is 3⋅128⋅160+3⋅128=61,440+384=61,8243 \cdot 128 \cdot 160 + 3 \cdot 128 = 61{,}440 + 384 = 61{,}824, against the LSTM's 82,43282{,}432. It also does three quarters of the multiply-adds per step. On many tasks the two perform about the same: an early comparison by Chung and colleagues (2014) found them comparable on polyphonic music and speech signal modeling, with both well ahead of a plain tanh RNN on the speech data.

To recap: the GRU keeps one state, updates it as a weighted average of the old value and a candidate, and has a direct gradient path whose factor is the update gate. Now count, compute and prove on paper.

On paperThe GRU on paper
  1. Count the parameters of a GRU layer and of an LSTM layer, both with hidden size n=64n = 64 and input size m=16m = 16 and one bias vector per gate.
  2. A one-unit GRU has weights (w,u,b)(w,\allowbreak u,\allowbreak b) of (1,1,0)(1,\allowbreak 1,\allowbreak 0) for the update gate, (0,2,−1)(0,\allowbreak 2,\allowbreak -1) for the reset gate and (1,−1,0.5)(1,\allowbreak -1,\allowbreak 0.5) for the candidate, in the lesson's convention ht=ztht−1+(1−zt)h~th_t = z_t h_{t-1} + (1 - z_t)\tilde{h}_t with h~t=tanh⁡(wxt+u (rtht−1)+b)\tilde{h}_t = \tanh(w x_t + u\,(r_t h_{t-1}) + b). With xt=−1x_t = -1 and ht−1=0.8h_{t-1} = 0.8, compute ztz_t, rtr_t, h~t\tilde{h}_t and hth_t to four decimals.
  3. Prove that if ∣ht−1∣<1|h_{t-1}| < 1 then ∣ht∣<1|h_t| < 1, whatever the weights. Why does the same argument fail for an LSTM's cell state?

The check at the top takes the GRU's parameter count, the LSTM's parameter count and hth_t, in that order, with hth_t to 4 decimal places.

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

GRU parameters, LSTM parameters, h_t

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

The GRU in code

With batches as rows, stack the three blocks as for the LSTM, in the order [r,z,h~][\mathbf{r},\allowbreak \mathbf{z},\allowbreak \tilde{\mathbf{h}}]: Wx=[Wr⊤∣Wz⊤∣Wc⊤]\mathbf{W}_x = [\mathbf{W}_r^\top \mid \mathbf{W}_z^\top \mid \mathbf{W}_c^\top] of shape m×3nm \times 3n for the input and Wh=[Ur⊤∣Uz⊤∣Uc⊤]\mathbf{W}_h = [\mathbf{U}_r^\top \mid \mathbf{U}_z^\top \mid \mathbf{U}_c^\top] of shape n×3nn \times 3n for the previous state, plus b\mathbf{b} of length 3n3n. As with the LSTM, the blocks are transposes of the column-form matrices above. The gates can use one product each for the input and the state. The candidate cannot share the state product, because the reset gate must multiply ht−1\mathbf{h}_{t-1} before Wh\mathbf{W}_h sees it: compute xWx+b\mathbf{x}\mathbf{W}_x + \mathbf{b} once for all three blocks, ht−1Wh\mathbf{h}_{t-1}\mathbf{W}_h for the first two blocks only, and (r⊙ht−1)Wh(\mathbf{r} \odot \mathbf{h}_{t-1})\mathbf{W}_h for the third.

The backward pass follows the same four-stage pattern as the LSTM's in the forward and backward pass, with one new twist: ht−1\mathbf{h}_{t-1} is used four times in a step (directly, in the two gates, and through r⊙ht−1\mathbf{r} \odot \mathbf{h}_{t-1}), so its gradient is a sum of four terms.

Go slower: The GRU step backward, line by line

This box uses the row convention of the code: a batch is stored as rows, the input part is xWx\mathbf{x}\mathbf{W}_x, and Wx,c\mathbf{W}_{x,c}, Wh,c\mathbf{W}_{h,c} are the candidate's column blocks of Wx\mathbf{W}_x and Wh\mathbf{W}_h (Wh,[r,z]\mathbf{W}_{h,[r,z]} is the first two blocks of Wh\mathbf{W}_h). Write ar,az,ac\mathbf{a}_r,\allowbreak \mathbf{a}_z,\allowbreak \mathbf{a}_c for the three pre-activations (so r=σ(ar)\mathbf{r} = \sigma(\mathbf{a}_r), z=σ(az)\mathbf{z} = \sigma(\mathbf{a}_z), h~=tanh⁡(ac)\tilde{\mathbf{h}} = \tanh(\mathbf{a}_c)), p=r⊙ht−1\mathbf{p} = \mathbf{r} \odot \mathbf{h}_{t-1}, and dq\mathrm{d}q for ∂L/∂q\partial L/\partial q. Given dht\mathrm{d}\mathbf{h}_t:

Stage 1: the mix ht=z⊙ht−1+(1−z)⊙h~\mathbf{h}_t = \mathbf{z} \odot \mathbf{h}_{t-1} + (1 - \mathbf{z}) \odot \tilde{\mathbf{h}}. The partial derivative with respect to z\mathbf{z} is ht−1−h~\mathbf{h}_{t-1} - \tilde{\mathbf{h}} (entry by entry), so dz=dht⊙(ht−1−h~),dh~=dht⊙(1−z),dht−1←dht⊙z.\mathrm{d}\mathbf{z} = \mathrm{d}\mathbf{h}_t \odot (\mathbf{h}_{t-1} - \tilde{\mathbf{h}}),\allowbreak \qquad \mathrm{d}\tilde{\mathbf{h}} = \mathrm{d}\mathbf{h}_t \odot (1 - \mathbf{z}),\allowbreak \qquad \mathrm{d}\mathbf{h}_{t-1} \leftarrow \mathrm{d}\mathbf{h}_t \odot \mathbf{z}.

Stage 2: the candidate. Through the tanh, dac=dh~⊙(1−h~2)\mathrm{d}\mathbf{a}_c = \mathrm{d}\tilde{\mathbf{h}} \odot (1 - \tilde{\mathbf{h}}^2). The candidate's pre-activation is xWx,c+p Wh,c+bc\mathbf{x}\mathbf{W}_{x,c} + \mathbf{p}\,\mathbf{W}_{h,c} + \mathbf{b}_c, a linear layer in p\mathbf{p}, so dWh,c=p⊤dac\mathrm{d}\mathbf{W}_{h,c} = \mathbf{p}^\top\mathrm{d}\mathbf{a}_c and dp=dac Wh,c⊤\mathrm{d}\mathbf{p} = \mathrm{d}\mathbf{a}_c\,\mathbf{W}_{h,c}^\top.

Stage 3: the reset product p=r⊙ht−1\mathbf{p} = \mathbf{r} \odot \mathbf{h}_{t-1}: dr=dp⊙ht−1\mathrm{d}\mathbf{r} = \mathrm{d}\mathbf{p} \odot \mathbf{h}_{t-1} and dht−1+=dp⊙r\mathrm{d}\mathbf{h}_{t-1} \mathrel{+}= \mathrm{d}\mathbf{p} \odot \mathbf{r}.

Stage 4: the gates. dar=dr⊙r⊙(1−r)\mathrm{d}\mathbf{a}_r = \mathrm{d}\mathbf{r} \odot \mathbf{r} \odot (1 - \mathbf{r}) and daz=dz⊙z⊙(1−z)\mathrm{d}\mathbf{a}_z = \mathrm{d}\mathbf{z} \odot \mathbf{z} \odot (1 - \mathbf{z}). Their pre-activations are linear in ht−1\mathbf{h}_{t-1} through the first two blocks of Wh\mathbf{W}_h, so dWh,[r,z]=ht−1⊤[dar,daz]\mathrm{d}\mathbf{W}_{h,[r,z]} = \mathbf{h}_{t-1}^\top[\mathrm{d}\mathbf{a}_r,\allowbreak \mathrm{d}\mathbf{a}_z] and dht−1+=[dar,daz] Wh,[r,z]⊤\mathrm{d}\mathbf{h}_{t-1} \mathrel{+}= [\mathrm{d}\mathbf{a}_r,\allowbreak \mathrm{d}\mathbf{a}_z]\,\mathbf{W}_{h,[r,z]}^\top.

The input side. All three pre-activations are linear in x\mathbf{x} through Wx\mathbf{W}_x and b\mathbf{b}: with da=[dar,daz,dac]\mathrm{d}\mathbf{a} = [\mathrm{d}\mathbf{a}_r,\allowbreak \mathrm{d}\mathbf{a}_z,\allowbreak \mathrm{d}\mathbf{a}_c], dWx=x⊤da\mathrm{d}\mathbf{W}_x = \mathbf{x}^\top\mathrm{d}\mathbf{a}, db\mathrm{d}\mathbf{b} is the column sum of da\mathrm{d}\mathbf{a}, and dx=da Wx⊤\mathrm{d}\mathbf{x} = \mathrm{d}\mathbf{a}\,\mathbf{W}_x^\top.

The four contributions to dht−1\mathrm{d}\mathbf{h}_{t-1} come from the direct mix (stage 1), the reset product (stage 3) and the two gates (stage 4).

The bound is easy to see in a run. Feed twelve 1s to the counting LSTM from the hand-design lesson and to a GRU unit that writes a candidate of +1+1 at every 1, with its update gate at one half. Each GRU step is then ht=0.5 ht−1+0.5×1h_t = 0.5\,h_{t-1} + 0.5 \times 1, so it moves halfway from where it is toward 1:

⌘+Enter runs · edit freelyPython sleeps until you run code

The LSTM cell reads 1, 2, 3, and so on up to 12. The GRU state goes 0.5, 0.75, 0.875, closing half the remaining gap to 1 at every step: after a few 1s the differences between counts are too small to use. Any other update gate changes the speed, not the limit.

Now build it, and let the gradient check tell you whether every line is right.

Code itA GRU step, forward and backward

Build a GRU cell in the stacked row layout, with column blocks in the order [r,z,h~][\mathbf{r},\allowbreak \mathbf{z},\allowbreak \tilde{\mathbf{h}}] and the lesson's convention ht=z⊙ht−1+(1−z)⊙h~\mathbf{h}_t = \mathbf{z} \odot \mathbf{h}_{t-1} + (1 - \mathbf{z}) \odot \tilde{\mathbf{h}}.

  1. gru_step(x, h_prev, Wx, Wh, b) with Wx of shape (m,3n)(m,\allowbreak 3n), Wh of shape (n,3n)(n,\allowbreak 3n) and b of shape (3n,)(3n,\allowbreak ) returns h and a cache.
  2. gru_step_backward(dh, cache) returns dx, dh_prev, dWx, dWh, db. Remember that ht−1\mathbf{h}_{t-1} reaches the output four ways.
  3. gru_param_count(n, m).

The code at the bottom gradient-checks your step and prints the errors.

⌘+Enter runsPython sleeps until you run code
Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.

Other knobs

The GRU is one change. Here are the others you will meet, each with what it costs and what is known about it.

  • Peephole connections. Gers and Schmidhuber (2000) let the gates read the cell state directly, through one extra weight per unit: the forget and input gates see ct−1\mathbf{c}_{t-1} and the output gate sees ct\mathbf{c}_t, for example ft=σ(Wfxt+Ufht−1+pf⊙ct−1+bf)\mathbf{f}_t = \sigma(\mathbf{W}_f\mathbf{x}_t + \mathbf{U}_f\mathbf{h}_{t-1} + \mathbf{p}_f \odot \mathbf{c}_{t-1} + \mathbf{b}_f). This helps when the gates need to know what is stored even while the output gate hides it, as in precise timing tasks. It adds only 3n3n parameters, and it is rarely used today.
  • Coupled input and forget gates. Set it=1−ft\mathbf{i}_t = 1 - \mathbf{f}_t, as the GRU does, and drop one gate's weights: a quarter fewer parameters.
  • Stacking layers. A second LSTM layer reads the first layer's hidden states as its input sequence: xt(2)=ht(1)\mathbf{x}^{(2)}_t = \mathbf{h}^{(1)}_t. Its input size is therefore the first layer's hidden size, so when both layers have hidden size nn the second has 4n(n+n)+4n4n(n + n) + 4n parameters. Depth in layers lets each step compute a more complex function; depth in time is what the recurrence already gives. Google's 2016 neural machine translation system stacked eight LSTM layers in its encoder and eight in its decoder, with residual connections between layers.
  • Hidden size. More units hold more at once, and cost grows with n2n^2: in parameters, in time per step and in memory for the cache.
  • Training window. With truncated backpropagation, the gradient never reaches further back than the window, so a model cannot be taught a dependency longer than its windows. Longer windows cost time and cache memory in proportion.
  • Dropout. Setting a random fraction of activations to zero during training reduces overfitting. For recurrent networks, Zaremba, Sutskever and Vinyals (2014) applied it only between layers (and to the input and output), not on the recurrent connection from ht−1\mathbf{h}_{t-1} to ht\mathbf{h}_t, where fresh noise at every step would disrupt the memory. Later work showed that dropping the same units at every step can work on the recurrent connection too.

What matters most among these? A large study by Greff and colleagues (2017) compared eight LSTM variants on speech, handwriting and music tasks. None improved significantly on the standard LSTM. Removing the forget gate or the output activation hurt the most, while coupling the input and forget gates or removing the peepholes made little difference. The broader lesson: once the essential mechanism is in place (an additive, gated memory with a forget gate), architecture details often matter less than initialization, training budget and model size.

Use those findings to pick a change under a parameter budget.

Quick checkCut a quarter of the parameters

You must shrink an LSTM layer's parameter count by about a quarter, and you care most about keeping its ability to learn long gaps. Based on the comparisons described in the lesson, which change is the safest?

Choose one answer, then check.

And count what stacking costs.

Work it outTwo stacked layers

Two LSTM layers are stacked, both with hidden size n=128n = 128. The first reads inputs of size m=32m = 32; the second reads the first layer's hidden states. With one bias vector per gate, how many parameters do the two layers have together?

parameters

Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.

An experiment you can trust

"The GRU is better" or "peepholes help" are claims about a distribution of training runs, not about one run. The training lesson showed why: on delayed recall at a gap of 40, the same RNN code succeeded in 2 runs of 24 and failed in the other 22, and the same LSTM succeeded or failed depending on one bias. A single run is a single sample. Here is a checklist for a design experiment.

  1. Ask one question with a measurable answer. Not "is the GRU good?" but "does a GRU learn delayed recall at a gap of 40 within 1,200 steps, with and without a keep bias?"
  2. Change one thing. Keep the data, the budget (steps, batch size, learning rate), the hidden size and the evaluation fixed. If you compare two architectures, decide whether you are matching hidden size or parameter count, and say which. A GRU and an LSTM with the same hidden size are not the same size.
  3. Run several seeds and report how many succeeded, or the mean and the spread. For a task where runs succeed or fail outright, a success count is more honest than an average.
  4. Measure on data the model never trained on. For recall that is fresh sequences; for text, held-out text.
  5. Report the reference points: the chance level, the baselines, and the budget. "0.98 accuracy" means little without "chance is 0.25".
  6. Say what the result does not show. A result at one gap, one size and one budget does not rank two architectures in general.

Now run one. The capstone asks whether a GRU, like the LSTM, needs its direct path held open at the start: a GRU classifier on delayed recall at a gap of 40, with a keep bias of 0 and of 3, two runs each. In our reference runs (24 seeds each), the GRU with a keep bias of 3 solved every run at gaps of 40 and 60, and with a keep bias of 0 it stayed at chance in all 24 runs at 40.

Code itCapstone: does a GRU need a keep bias?

Run a design experiment of your own. Build a GRU classifier over whole sequences and test whether its update gate, like the LSTM's forget gate, must start open for the model to learn delayed recall at a gap of 40.

The setup provides the training lesson's task and training (make_recall_batch, train_and_test, N_INPUTS = 6, N_SYMBOLS = 4), a reference gru_step(x, h_prev, Wx, Wh, b) returning (h, cache) and gru_step_backward(dh, cache) returning (dx, dh_prev, dWx, dWh, db), with blocks in the order [r,z,h~][\mathbf{r},\allowbreak \mathbf{z},\allowbreak \tilde{\mathbf{h}}], and softmax_cross_entropy.

  1. GRUClassifier(n_in, hidden, n_classes, rng, keep_bias): the interface of the training lesson's classifiers. Run the steps forward from a zero state, classify the last state, and send the gradient back through every step, summing the weight gradients.
  2. keep_bias_experiment(delay, keep_biases, runs): a dict from each keep bias to the list of accuracies of its runs, each run with its own seeded generator.
  3. param_counts(n, m): the parameters of one RNN, GRU and LSTM layer.

The run trains four GRUs, which takes about half a minute.

This one trains a model in your browser, so a run can take a minute or more. The charts update as it goes, and Stop ends it early.

⌘+Enter runsPython sleeps until you run code
Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.

Finish by writing up your experiment so that someone who did not see it could trust it.

In your own wordsWrite up the experiment

Write a short report of the keep-bias experiment for a colleague who did not see it: the question, what was held fixed and what was changed, the results with their reference points, what the result shows about the GRU's update gate, and at least one thing it does not show.

A few sentences first: 0 of 60 characters.

Saved in this browser as you type.

Next: the module checkpoint