ml.lab
Python sleeps until you run code
07 Recurrent networks

Checkpoint

Recurrent networks

Twelve questions across the module: the recurrent step, one-hot inputs, parameter counts, the language-model loss, windows and sampling, and backpropagation through time with its vanishing and exploding gradients. The numbers are new, and several questions aim at the classic confusions. Work on paper before answering. You need 80% to pass.

  1. Question 1 of 12.

    A character RNN reads one-hot characters from a vocabulary of 90, has hidden size 200, and outputs logits over the same 90 characters. How many parameters does it have, biases included? Enter the whole number, without commas.

    parameters

    Type a number: 0.25, -2, 3/4 and sqrt(2) all work.

  2. Question 2 of 12.

    A two-unit RNN reads one-hot characters from a vocabulary of three. Its weights are Whh=[10.5−0.50]\mathbf{W}_{hh} = \begin{bmatrix} 1 & 0.5 \\ -0.5 & 0 \end{bmatrix}, Wxh=[01−1201]\mathbf{W}_{xh} = \begin{bmatrix} 0 & 1 & -1 \\ 2 & 0 & 1 \end{bmatrix} and b=(0.3,−1)\mathbf{b} = (0.3,\allowbreak -1). The previous state is ht−1=(0.2,0.6)\mathbf{h}_{t-1} = (0.2,\allowbreak 0.6), and it now reads the second character of the vocabulary. Find ht\mathbf{h}_t to 3 decimal places and enter it as a column, unit 1 first.

    hₜ

    One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.

  3. Question 3 of 12.

    In code, Wxh has shape (V, H) and X is a (B, T) array of character ids. What is Wxh[X]?

  4. Question 4 of 12.

    A character model's validation loss is 1.5 nats per character. What is its perplexity? Give 2 decimal places.

    perplexity

    Type a number: 0.25, -2, 3/4 and sqrt(2) all work.

  5. Question 5 of 12.

    The encoded text is ids = [5, 9, 2, 7, 1, 8, 3]. For training windows of length 3, which is the window with the largest valid start?

  6. Question 6 of 12.

    A model's next-character probabilities are (0.7,0.2,0.1)(0.7,\allowbreak 0.2,\allowbreak 0.1). Find the distribution at temperature τ=2\tau = 2, to 3 decimal places, and enter it as a column in the same order.

    probabilities

    One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.

  7. Question 7 of 12.

    You generate from a trained character model at temperature 1 and at temperature 0. Which statement is true?

  8. Question 8 of 12.

    An RNN reads 10 characters and has a loss only at the last step. In ∂L/∂Whh=∑t=110δtht−1⊤\partial L/\partial\mathbf{W}_{hh} = \sum_{t=1}^{10}\boldsymbol{\delta}_t\mathbf{h}_{t-1}^\top, which of the δt\boldsymbol{\delta}_t can be nonzero?

  9. Question 9 of 12.

    A scalar RNN has whh=1.25w_{hh} = 1.25, and its state stays near h=0.6h = 0.6. By roughly what factor is a gradient multiplied on its way back 30 steps? Give 4 decimal places.

    factor

    Type a number: 0.25, -2, 3/4 and sqrt(2) all work.

  10. Question 10 of 12.

    The gradients of a model's two parameter arrays are (6,−8)(6,\allowbreak -8) and (0,24)(0,\allowbreak 24). You clip the combined gradient to norm 13. Enter the four clipped entries as a column, in the same order.

    clipped gradient

    One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.

  11. Question 11 of 12.

    The recurrent matrix of an RNN has largest singular value σ1(Whh)=0.9\sigma_1(\mathbf{W}_{hh}) = 0.9. Which statements are guaranteed? Select all that apply.

    Choose every option that applies.

  12. Question 12 of 12.

    Run the tests before you submit. This question counts as correct if your latest run passes every check.

    Code it

    Write state_gradients(W_hh, hs, g_T) for an RNN written with column vectors, ht=tanh⁡(Whhht−1+…)\mathbf{h}_t = \tanh(\mathbf{W}_{hh}\mathbf{h}_{t-1} + \ldots). hs has shape (T + 1, n) with row t equal to ht\mathbf{h}_t for t=0,…,Tt = 0,\allowbreak \ldots,\allowbreak T, and g_T is ∂L/∂hT\partial L/\partial\mathbf{h}_T, where the loss depends on the states only through hT\mathbf{h}_T. Return G of shape (T + 1, n) with G[t] equal to ∂L/∂ht\partial L/\partial\mathbf{h}_t.

    ⌘+Enter runsPython sleeps until you run code
    Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.
0 of 12 answered. Unanswered questions count as wrong.