ml.lab
Python sleeps until you run code
08 LSTMs

Checkpoint

LSTMs

Twelve questions across the whole module: the cell and its gates, the gradient along the cell path, hand-designed weights and parameter counts, the backward pass, training, the GRU and how to read an experiment. The numbers are new, and several questions aim at the classic confusions. Work on paper before answering, and write the code without looking back at the lessons if you can. You need 80% to pass.

  1. Question 1 of 12.

    An LSTM unit should hold a value for many steps without letting the rest of the network see it, and reveal it only when a query symbol arrives. Which gate makes the holding silent?

  2. Question 2 of 12.

    A one-unit LSTM has weights (ww on xtx_t, uu on ht−1h_{t-1}, bb): forget (0,2,2)(0,\allowbreak 2,\allowbreak 2), input (1,0,1)(1,\allowbreak 0,\allowbreak 1), candidate (−2,0,1)(-2,\allowbreak 0,\allowbreak 1), output (1,2,0)(1,\allowbreak 2,\allowbreak 0). With xt=1x_t = 1, ht−1=−0.5h_{t-1} = -0.5 and ct−1=2c_{t-1} = 2, find ctc_t and hth_t. Enter them as a column, ctc_t first, to 3 decimal places.

    c_t, h_t

    One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.

  3. Question 3 of 12.

    A gradient of 2 arrives at ctc_t. The forget gate of this unit is 0.90.9 at step tt, 0.50.5 at step t−1t-1, 0.990.99 at step t−2t-2 and 0.80.8 at step t−3t-3. How large is the part of the gradient that reaches ct−4c_{t-4} along the direct cell path? Give 4 decimal places.

    gradient at c_{t-4}

    Type a number: 0.25, -2, 3/4 and sqrt(2) all work.

  4. Question 4 of 12.

    Which of these statements are true? Select every true one.

    Choose every option that applies.

  5. Question 5 of 12.

    How many parameters does an LSTM layer with hidden size n=100n = 100 and input size m=50m = 50 have, with one bias vector per gate?

    parameters

    Type a number: 0.25, -2, 3/4 and sqrt(2) all work.

  6. Question 6 of 12.

    At initialization a forget gate sits at about σ(2)\sigma(2). What fraction of a gradient survives 30 steps along the cell path? Give 3 decimal places.

    fraction

    Type a number: 0.25, -2, 3/4 and sqrt(2) all work.

  7. Question 7 of 12.

    During the backward pass at step tt, a total gradient dht=0.5\mathrm{d}h_t = 0.5 arrives at the output and dct=1\mathrm{d}c_t = 1 arrives along the cell path. The saved values are ot=0.6o_t = 0.6, tanh⁡(ct)=0.8\tanh(c_t) = 0.8 and ft=0.9f_t = 0.9. What gradient does this step send to ct−1c_{t-1}? Give 4 decimal places.

    dc_{t-1}

    Type a number: 0.25, -2, 3/4 and sqrt(2) all work.

  8. Question 8 of 12.

    Your gradient check for a full LSTM forward and backward pass passes for W\mathbf{W}, b\mathbf{b}, X\mathbf{X} and h0\mathbf{h}_0, and fails only for c0\mathbf{c}_0. Which bug fits?

  9. Question 9 of 12.

    Which of these can a one-layer LSTM do by construction that a GRU of the same size cannot?

  10. Question 10 of 12.

    A character model over a 50-character vocabulary reaches a validation loss of 1.81.8 nats per character. What is its perplexity? Give 2 decimal places.

    perplexity

    Type a number: 0.25, -2, 3/4 and sqrt(2) all work.

  11. Question 11 of 12.

    A colleague trains one GRU and one LSTM on a memory task, once each, with the same settings. The GRU reaches 0.98 accuracy and the LSTM 0.25, which is chance. What is the most defensible conclusion?

  12. Question 12 of 12.

    Run the tests before you submit. This question counts as correct if your latest run passes every check.

    Code it

    Write cell_path_gains(F). F has shape (T,n)(T,\allowbreak n) and row t−1t - 1 holds the forget-gate values ft\mathbf{f}_t for steps t=1,…,Tt = 1,\allowbreak \ldots,\allowbreak T. Return G of the same shape, where row t−1t - 1 is the factor by which a gradient on cT\mathbf{c}_T is multiplied when it reaches ct\mathbf{c}_t along the direct cell path: ft+1⊙⋯⊙fT\mathbf{f}_{t+1} \odot \cdots \odot \mathbf{f}_T, which is 1 for t=Tt = T. Do not modify F.

    ⌘+Enter runsPython sleeps until you run code
    Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.
0 of 12 answered. Unanswered questions count as wrong.