Checkpoint

Recurrent networks

Twelve questions across the module: the recurrent step, one-hot inputs, parameter counts, the language-model loss, windows and sampling, and backpropagation through time with its vanishing and exploding gradients. The numbers are new, and several questions aim at the classic confusions. Work on paper before answering. You need 80% to pass.

1
A character RNN reads one-hot characters from a vocabulary of 90, has hidden size 200, and outputs logits over the same 90 characters. How many parameters does it have, biases included?
parameters
2
A two-unit RNN reads one-hot characters from a vocabulary of three. Its weights are $\mathbf{W}_{hh} = \begin{bmatrix} 1 & 0.5 \\ -0.5 & 0 \end{bmatrix}$ , $\mathbf{W}_{xh} = \begin{bmatrix} 0 & 1 & -1 \\ 2 & 0 & 1 \end{bmatrix}$ and $\mathbf{b} = (0.3, -1)$ . The previous state is $\mathbf{h}_{t-1} = (0.2, 0.6)$ , and it now reads the second character of the vocabulary. Find $\mathbf{h}_t$ to 3 decimal places.
hₜ
3
In code, Wxh has shape (V, H) and X is a (B, T) array of character ids. What is Wxh[X]?
4
A character model's validation loss is 1.5 nats per character. What is its perplexity? Give 2 decimal places.
perplexity
5
The encoded text is ids = [5, 9, 2, 7, 1, 8, 3]. For training windows of length 3, which is the window with the largest valid start?
6
A model's next-character probabilities are $(0.7, 0.2, 0.1)$ . Find the distribution at temperature $\tau = 2$ , to 3 decimal places.
probabilities
7
You generate from a trained character model at temperature 1 and at temperature 0. Which statement is true?
8
An RNN reads 10 characters and has a loss only at the last step. In $\partial L/\partial\mathbf{W}_{hh} = \sum_{t=1}^{10}\boldsymbol{\delta}_t\mathbf{h}_{t-1}^\top$ , which of the $\boldsymbol{\delta}_t$ can be nonzero?
9
A scalar RNN has $w_{hh} = 1.25$ , and its state stays near $h = 0.6$ . By roughly what factor is a gradient multiplied on its way back 30 steps? Give 4 decimal places.
factor
10
The gradients of a model's two parameter arrays are $(6, -8)$ and $(0, 24)$ . You clip the combined gradient to norm 13. Enter the four clipped entries in order.
clipped gradient
11
The recurrent matrix of an RNN has largest singular value $\sigma_1(\mathbf{W}_{hh}) = 0.9$ . Which statements are guaranteed? Select all that apply.
12
Code it
Write state_gradients(W_hh, hs, g_T) for an RNN written with column vectors, $\mathbf{h}_t = \tanh(\mathbf{W}_{hh}\mathbf{h}_{t-1} + \ldots)$ . hs has shape (T + 1, n) with row t equal to $\mathbf{h}_t$ for $t = 0, \ldots, T$ , and g_T is $\partial L/\partial\mathbf{h}_T$ , where the loss depends on the states only through $\mathbf{h}_T$ . Return G of shape (T + 1, n) with G[t] equal to $\partial L/\partial\mathbf{h}_t$ .
⌘↵ runsPython sleeps until you run code

12 still unanswered. Unanswered questions count as wrong.