Lesson 6 of 6
Design your own
The LSTM is one point in a space of designs. Build the gated recurrent unit from scratch, see which other changes matter and which do not, and learn to test a design idea with an experiment you can trust.
Write the GRU equations, compare them with the LSTM's, and count the parameters of both.
Implement a GRU step forward and backward and verify it with a gradient check.
Describe peephole connections, stacked layers, hidden size, training length and dropout as design decisions with costs.
Design a fair experiment (one change, several seeds, held-out measurement), run it, and write up what it shows.

Every piece of the LSTM is a decision someone made. Four gates, not three or five. A separate cell state next to the hidden state. A tanh on the way out. Gates that look at the input and the previous output but not at the cell. Each decision could have gone another way, and over the years almost every alternative has been tried.
This lesson builds the most successful alternative, the gated recurrent unit or GRU, from scratch. Then it surveys the other knobs and what is known about them. It ends with the part that transfers to any model you will ever work on: how to find out whether a design change actually helps, with an experiment whose result you can trust.
The GRU: two gates, one state
The GRU, introduced by Cho and colleagues in 2014, removes the separate cell. It keeps a single state and updates it with two gates:
The shapes follow the LSTM's: has entries, and every gate have , the matrices are and the matrices . The subscript marks the candidate's weights, and (read "h tilde") is the candidate itself. Read the equations against the LSTM's.
- The update gate plays the forget gate. Unit keeps the fraction of its old value and fills the rest, , with the candidate. In LSTM terms, the forget gate is and the input gate is forced to be : the gates are coupled, so keeping more always means writing less.
- The reset gate decides what the candidate may see. With near 0, the candidate is computed as if the past were blank, from alone. The LSTM has nothing exactly like it.
- There is no output gate and no separate cell. The whole state is always visible to the next layer and to the gates.
One warning about conventions. Sources differ on whether or multiplies the old state. This lesson follows PyTorch, where multiplies , so that lines up with the forget gate. The two conventions describe the same model with the sign of the gate's weights and bias flipped. PyTorch also applies the reset gate after the matrix product, with an extra bias vector , which is a slightly different model. Check the convention before comparing numbers across libraries.
One step with numbers
The equations run in a fixed order: both gates first, then the candidate (which needs the reset gate), then the mix (which needs the update gate and the candidate). Take a one-unit GRU with and , with weights of for the update gate, for the reset gate and for the candidate:
- update: , so
- reset: , so
- candidate: the reset gate halves the old state before the candidate sees it, , so the pre-activation is and
- new state: keep the fraction of the old value and fill the rest, , with the candidate. With five decimals, so that the sum checks,
The unit kept 73% of its old value and moved 27% of the way from toward the candidate .
Gradients, and a bound on the state
Hold the gates and the candidate fixed and differentiate the last equation, entry by entry: has slope with respect to and 0 with respect to every other entry. So along the direct path is . That is the same structure as the LSTM's cell path, with in place of (why gradients survive), and it needs the same care at initialization: with a bias of 0, and the path halves the gradient at every step. A positive keep bias on opens it, exactly as the forget bias did in the hand-design lesson. The GRU has more indirect paths than the LSTM, because also feeds both gates and, through the reset gate, the candidate.
The coupling has a consequence that the LSTM does not share. If , then is a weighted average of two numbers inside (the old value, and a tanh), with weights and that are positive and sum to 1. An average of numbers inside stays inside: . Starting from , a GRU's state never leaves . The LSTM cell in the hand-design lesson counted to 12 by adding 1 at every step; a GRU unit cannot do that. Weiss, Goldberg and Yahav (2018) showed the difference empirically: LSTMs trained to recognize strings of the form (some number of a's followed by exactly as many b's) learned to count, and kept working well beyond the lengths they were trained on; GRUs did not.
Parameters
Three blocks instead of four: , three quarters of an LSTM of the same sizes. For and that is , against the LSTM's . It also does three quarters of the multiply-adds per step. On many tasks the two perform about the same: an early comparison by Chung and colleagues (2014) found them comparable on polyphonic music and speech signal modeling, with both well ahead of a plain tanh RNN on the speech data.
To recap: the GRU keeps one state, updates it as a weighted average of the old value and a candidate, and has a direct gradient path whose factor is the update gate. Now count, compute and prove on paper.
- Count the parameters of a GRU layer and of an LSTM layer, both with hidden size and input size and one bias vector per gate.
- A one-unit GRU has weights of for the update gate, for the reset gate and for the candidate, in the lesson's convention with . With and , compute , , and to four decimals.
- Prove that if then , whatever the weights. Why does the same argument fail for an LSTM's cell state?
The check at the top takes the GRU's parameter count, the LSTM's parameter count and , in that order, with to 4 decimal places.
Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
The GRU in code
With batches as rows, stack the three blocks as for the LSTM, in the order : of shape for the input and of shape for the previous state, plus of length . As with the LSTM, the blocks are transposes of the column-form matrices above. The gates can use one product each for the input and the state. The candidate cannot share the state product, because the reset gate must multiply before sees it: compute once for all three blocks, for the first two blocks only, and for the third.
The backward pass follows the same four-stage pattern as the LSTM's in the forward and backward pass, with one new twist: is used four times in a step (directly, in the two gates, and through ), so its gradient is a sum of four terms.
Go slower: The GRU step backward, line by line
This box uses the row convention of the code: a batch is stored as rows, the input part is , and , are the candidate's column blocks of and ( is the first two blocks of ). Write for the three pre-activations (so , , ), , and for . Given :
Stage 1: the mix . The partial derivative with respect to is (entry by entry), so
Stage 2: the candidate. Through the tanh, . The candidate's pre-activation is , a linear layer in , so and .
Stage 3: the reset product : and .
Stage 4: the gates. and . Their pre-activations are linear in through the first two blocks of , so and .
The input side. All three pre-activations are linear in through and : with , , is the column sum of , and .
The four contributions to come from the direct mix (stage 1), the reset product (stage 3) and the two gates (stage 4).
The bound is easy to see in a run. Feed twelve 1s to the counting LSTM from the hand-design lesson and to a GRU unit that writes a candidate of at every 1, with its update gate at one half. Each GRU step is then , so it moves halfway from where it is toward 1:
The LSTM cell reads 1, 2, 3, and so on up to 12. The GRU state goes 0.5, 0.75, 0.875, closing half the remaining gap to 1 at every step: after a few 1s the differences between counts are too small to use. Any other update gate changes the speed, not the limit.
Now build it, and let the gradient check tell you whether every line is right.
Build a GRU cell in the stacked row layout, with column blocks in the order and the lesson's convention .
gru_step(x, h_prev, Wx, Wh, b)withWxof shape ,Whof shape andbof shape returnshand a cache.gru_step_backward(dh, cache)returnsdx, dh_prev, dWx, dWh, db. Remember that reaches the output four ways.gru_param_count(n, m).
The code at the bottom gradient-checks your step and prints the errors.
raise NotImplementedError, then press Run tests. Each check says what it expects.Other knobs
The GRU is one change. Here are the others you will meet, each with what it costs and what is known about it.
- Peephole connections. Gers and Schmidhuber (2000) let the gates read the cell state directly, through one extra weight per unit: the forget and input gates see and the output gate sees , for example . This helps when the gates need to know what is stored even while the output gate hides it, as in precise timing tasks. It adds only parameters, and it is rarely used today.
- Coupled input and forget gates. Set , as the GRU does, and drop one gate's weights: a quarter fewer parameters.
- Stacking layers. A second LSTM layer reads the first layer's hidden states as its input sequence: . Its input size is therefore the first layer's hidden size, so when both layers have hidden size the second has parameters. Depth in layers lets each step compute a more complex function; depth in time is what the recurrence already gives. Google's 2016 neural machine translation system stacked eight LSTM layers in its encoder and eight in its decoder, with residual connections between layers.
- Hidden size. More units hold more at once, and cost grows with : in parameters, in time per step and in memory for the cache.
- Training window. With truncated backpropagation, the gradient never reaches further back than the window, so a model cannot be taught a dependency longer than its windows. Longer windows cost time and cache memory in proportion.
- Dropout. Setting a random fraction of activations to zero during training reduces overfitting. For recurrent networks, Zaremba, Sutskever and Vinyals (2014) applied it only between layers (and to the input and output), not on the recurrent connection from to , where fresh noise at every step would disrupt the memory. Later work showed that dropping the same units at every step can work on the recurrent connection too.
What matters most among these? A large study by Greff and colleagues (2017) compared eight LSTM variants on speech, handwriting and music tasks. None improved significantly on the standard LSTM. Removing the forget gate or the output activation hurt the most, while coupling the input and forget gates or removing the peepholes made little difference. The broader lesson: once the essential mechanism is in place (an additive, gated memory with a forget gate), architecture details often matter less than initialization, training budget and model size.
Use those findings to pick a change under a parameter budget.
You must shrink an LSTM layer's parameter count by about a quarter, and you care most about keeping its ability to learn long gaps. Based on the comparisons described in the lesson, which change is the safest?
Choose one answer, then check.
And count what stacking costs.
Two LSTM layers are stacked, both with hidden size . The first reads inputs of size ; the second reads the first layer's hidden states. With one bias vector per gate, how many parameters do the two layers have together?
Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.
An experiment you can trust
"The GRU is better" or "peepholes help" are claims about a distribution of training runs, not about one run. The training lesson showed why: on delayed recall at a gap of 40, the same RNN code succeeded in 2 runs of 24 and failed in the other 22, and the same LSTM succeeded or failed depending on one bias. A single run is a single sample. Here is a checklist for a design experiment.
- Ask one question with a measurable answer. Not "is the GRU good?" but "does a GRU learn delayed recall at a gap of 40 within 1,200 steps, with and without a keep bias?"
- Change one thing. Keep the data, the budget (steps, batch size, learning rate), the hidden size and the evaluation fixed. If you compare two architectures, decide whether you are matching hidden size or parameter count, and say which. A GRU and an LSTM with the same hidden size are not the same size.
- Run several seeds and report how many succeeded, or the mean and the spread. For a task where runs succeed or fail outright, a success count is more honest than an average.
- Measure on data the model never trained on. For recall that is fresh sequences; for text, held-out text.
- Report the reference points: the chance level, the baselines, and the budget. "0.98 accuracy" means little without "chance is 0.25".
- Say what the result does not show. A result at one gap, one size and one budget does not rank two architectures in general.
Now run one. The capstone asks whether a GRU, like the LSTM, needs its direct path held open at the start: a GRU classifier on delayed recall at a gap of 40, with a keep bias of 0 and of 3, two runs each. In our reference runs (24 seeds each), the GRU with a keep bias of 3 solved every run at gaps of 40 and 60, and with a keep bias of 0 it stayed at chance in all 24 runs at 40.
Run a design experiment of your own. Build a GRU classifier over whole sequences and test whether its update gate, like the LSTM's forget gate, must start open for the model to learn delayed recall at a gap of 40.
The setup provides the training lesson's task and training (make_recall_batch, train_and_test, N_INPUTS = 6, N_SYMBOLS = 4), a reference gru_step(x, h_prev, Wx, Wh, b) returning (h, cache) and gru_step_backward(dh, cache) returning (dx, dh_prev, dWx, dWh, db), with blocks in the order , and softmax_cross_entropy.
GRUClassifier(n_in, hidden, n_classes, rng, keep_bias): the interface of the training lesson's classifiers. Run the steps forward from a zero state, classify the last state, and send the gradient back through every step, summing the weight gradients.keep_bias_experiment(delay, keep_biases, runs): a dict from each keep bias to the list of accuracies of its runs, each run with its own seeded generator.param_counts(n, m): the parameters of one RNN, GRU and LSTM layer.
The run trains four GRUs, which takes about half a minute.
This one trains a model in your browser, so a run can take a minute or more. The charts update as it goes, and Stop ends it early.
raise NotImplementedError, then press Run tests. Each check says what it expects.Finish by writing up your experiment so that someone who did not see it could trust it.
Write a short report of the keep-bias experiment for a colleague who did not see it: the question, what was held fixed and what was changed, the results with their reference points, what the result shows about the GRU's update gate, and at least one thing it does not show.
Saved in this browser as you type.