Checkpoint
LSTMs
Twelve questions across the whole module: the cell and its gates, the gradient along the cell path, hand-designed weights and parameter counts, the backward pass, training, the GRU and how to read an experiment. The numbers are new, and several questions aim at the classic confusions. Work on paper before answering, and write the code without looking back at the lessons if you can. You need 80% to pass.
- Question 1 of 12.
An LSTM unit should hold a value for many steps without letting the rest of the network see it, and reveal it only when a query symbol arrives. Which gate makes the holding silent?
- Question 2 of 12.
A one-unit LSTM has weights ( on , on , ): forget , input , candidate , output . With , and , find and . Enter them as a column, first, to 3 decimal places.
c_t, h_tOne entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.
- Question 3 of 12.
A gradient of 2 arrives at . The forget gate of this unit is at step , at step , at step and at step . How large is the part of the gradient that reaches along the direct cell path? Give 4 decimal places.
gradient at c_{t-4}Type a number: 0.25, -2, 3/4 and sqrt(2) all work.
- Question 4 of 12.
Which of these statements are true? Select every true one.
Choose every option that applies.
- Question 5 of 12.
How many parameters does an LSTM layer with hidden size and input size have, with one bias vector per gate?
parametersType a number: 0.25, -2, 3/4 and sqrt(2) all work.
- Question 6 of 12.
At initialization a forget gate sits at about . What fraction of a gradient survives 30 steps along the cell path? Give 3 decimal places.
fractionType a number: 0.25, -2, 3/4 and sqrt(2) all work.
- Question 7 of 12.
During the backward pass at step , a total gradient arrives at the output and arrives along the cell path. The saved values are , and . What gradient does this step send to ? Give 4 decimal places.
dc_{t-1}Type a number: 0.25, -2, 3/4 and sqrt(2) all work.
- Question 8 of 12.
Your gradient check for a full LSTM forward and backward pass passes for , , and , and fails only for . Which bug fits?
- Question 9 of 12.
Which of these can a one-layer LSTM do by construction that a GRU of the same size cannot?
- Question 10 of 12.
A character model over a 50-character vocabulary reaches a validation loss of nats per character. What is its perplexity? Give 2 decimal places.
perplexityType a number: 0.25, -2, 3/4 and sqrt(2) all work.
- Question 11 of 12.
A colleague trains one GRU and one LSTM on a memory task, once each, with the same settings. The GRU reaches 0.98 accuracy and the LSTM 0.25, which is chance. What is the most defensible conclusion?
- Question 12 of 12.
Run the tests before you submit. This question counts as correct if your latest run passes every check.
Code it Write
cell_path_gains(F).Fhas shape and row holds the forget-gate values for steps . ReturnGof the same shape, where row is the factor by which a gradient on is multiplied when it reaches along the direct cell path: , which is 1 for . Do not modifyF.Write your code where the starter saysraise NotImplementedError, then press Run tests. Each check says what it expects.