Checkpoint
Recurrent networks
Twelve questions across the module: the recurrent step, one-hot inputs, parameter counts, the language-model loss, windows and sampling, and backpropagation through time with its vanishing and exploding gradients. The numbers are new, and several questions aim at the classic confusions. Work on paper before answering. You need 80% to pass.
- Question 1 of 12.
A character RNN reads one-hot characters from a vocabulary of 90, has hidden size 200, and outputs logits over the same 90 characters. How many parameters does it have, biases included? Enter the whole number, without commas.
parametersType a number: 0.25, -2, 3/4 and sqrt(2) all work.
- Question 2 of 12.
A two-unit RNN reads one-hot characters from a vocabulary of three. Its weights are , and . The previous state is , and it now reads the second character of the vocabulary. Find to 3 decimal places and enter it as a column, unit 1 first.
hₜOne entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.
- Question 3 of 12.
In code,
Wxhhas shape(V, H)andXis a(B, T)array of character ids. What isWxh[X]? - Question 4 of 12.
A character model's validation loss is 1.5 nats per character. What is its perplexity? Give 2 decimal places.
perplexityType a number: 0.25, -2, 3/4 and sqrt(2) all work.
- Question 5 of 12.
The encoded text is
ids = [5, 9, 2, 7, 1, 8, 3]. For training windows of length 3, which is the window with the largest valid start? - Question 6 of 12.
A model's next-character probabilities are . Find the distribution at temperature , to 3 decimal places, and enter it as a column in the same order.
probabilitiesOne entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.
- Question 7 of 12.
You generate from a trained character model at temperature 1 and at temperature 0. Which statement is true?
- Question 8 of 12.
An RNN reads 10 characters and has a loss only at the last step. In , which of the can be nonzero?
- Question 9 of 12.
A scalar RNN has , and its state stays near . By roughly what factor is a gradient multiplied on its way back 30 steps? Give 4 decimal places.
factorType a number: 0.25, -2, 3/4 and sqrt(2) all work.
- Question 10 of 12.
The gradients of a model's two parameter arrays are and . You clip the combined gradient to norm 13. Enter the four clipped entries as a column, in the same order.
clipped gradientOne entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.
- Question 11 of 12.
The recurrent matrix of an RNN has largest singular value . Which statements are guaranteed? Select all that apply.
Choose every option that applies.
- Question 12 of 12.
Run the tests before you submit. This question counts as correct if your latest run passes every check.
Code it Write
state_gradients(W_hh, hs, g_T)for an RNN written with column vectors, .hshas shape(T + 1, n)with rowtequal to for , andg_Tis , where the loss depends on the states only through . ReturnGof shape(T + 1, n)withG[t]equal to .Write your code where the starter saysraise NotImplementedError, then press Run tests. Each check says what it expects.