Checkpoint

LSTMs

Twelve questions across the whole module: the cell and its gates, the gradient along the cell path, hand-designed weights and parameter counts, the backward pass, training, the GRU and how to read an experiment. The numbers are new, and several questions aim at the classic confusions. Work on paper before answering, and write the code without looking back at the lessons if you can. You need 80% to pass.

1
An LSTM unit should hold a value for many steps without letting the rest of the network see it, and reveal it only when a query symbol arrives. Which gate makes the holding silent?
2
A one-unit LSTM has weights ( $w$ on $x_t$ , $u$ on $h_{t-1}$ , $b$ ): forget $(0, 2, 2)$ , input $(1, 0, 1)$ , candidate $(-2, 0, 1)$ , output $(1, 2, 0)$ . With $x_t = 1$ , $h_{t-1} = -0.5$ and $c_{t-1} = 2$ , find $c_t$ and $h_t$ . Enter them as a column, $c_t$ first.
c_t, h_t
3
A gradient of 2 arrives at $c_t$ . The forget gate of this unit is $0.9$ at step $t$ , $0.5$ at step $t-1$ , $0.99$ at step $t-2$ and $0.8$ at step $t-3$ . How large is the part of the gradient that reaches $c_{t-4}$ along the direct cell path?
gradient at c_{t-4}
4
Which of these statements are true? Select every true one.
5
How many parameters does an LSTM layer with hidden size $n = 100$ and input size $m = 50$ have, with one bias vector per gate?
parameters
6
At initialization a forget gate sits at about $\sigma(2)$ . What fraction of a gradient survives 30 steps along the cell path? Give 3 decimal places.
fraction
7
During the backward pass at step $t$ , a total gradient $\mathrm{d}h_t = 0.5$ arrives at the output and $\mathrm{d}c_t = 1$ arrives along the cell path. The saved values are $o_t = 0.6$ , $\tanh(c_t) = 0.8$ and $f_t = 0.9$ . What gradient does this step send to $c_{t-1}$ ?
dc_{t-1}
8
Your gradient check for a full LSTM forward and backward pass passes for $\mathbf{W}$ , $\mathbf{b}$ , $\mathbf{X}$ and $\mathbf{h}_0$ , and fails only for $\mathbf{c}_0$ . Which bug fits?
9
Which of these can a one-layer LSTM do by construction that a GRU of the same size cannot?
10
A character model over a 50-character vocabulary reaches a validation loss of $1.8$ nats per character. What is its perplexity? Give 2 decimal places.
perplexity
11
A colleague trains one GRU and one LSTM on a memory task, once each, with the same settings. The GRU reaches 0.98 accuracy and the LSTM 0.25, which is chance. What is the most defensible conclusion?
12
Code it
Write cell_path_gains(F). F has shape $(T, n)$ and row $t - 1$ holds the forget-gate values $\mathbf{f}_t$ for steps $t = 1, \ldots, T$ . Return G of the same shape, where row $t - 1$ is the factor by which a gradient on $\mathbf{c}_T$ is multiplied when it reaches $\mathbf{c}_t$ along the direct cell path: $\mathbf{f}_{t+1} \odot \cdots \odot \mathbf{f}_T$ , which is 1 for $t = T$ . Do not modify F.
⌘↵ runsPython sleeps until you run code

12 still unanswered. Unanswered questions count as wrong.