ml.lab
Python sleeps until you run code
06 Neural networks from scratch

Checkpoint

Neural networks from scratch

Twelve questions across the whole module: layers and their parameters, why the nonlinearity matters, softmax and cross-entropy, backpropagation through a linear layer, and reading and optimizing a training run. The numbers are new, and several questions aim at the classic confusions. Work on paper before answering. You need 80% to pass.

  1. Question 1 of 12.

    How many parameters, weights and biases together, does a network with layer sizes 100-64-64-5 have?

    parameters

    Type a number: 0.25, -2, 3/4 and sqrt(2) all work.

  2. Question 2 of 12.

    In the column convention, a network computes h=W1x\mathbf{h} = \mathbf{W}_1\mathbf{x} and then y=W2h\mathbf{y} = \mathbf{W}_2\mathbf{h}, with no activation and no biases, where

    W1=(1201),W2=(1−120).\mathbf{W}_1 = \begin{pmatrix} 1 & 2 \\ 0 & 1 \end{pmatrix}, \qquad \mathbf{W}_2 = \begin{pmatrix} 1 & -1 \\ 2 & 0 \end{pmatrix}.

    Find the single matrix A\mathbf{A} with y=Ax\mathbf{y} = \mathbf{A}\mathbf{x} for every x\mathbf{x}, and enter it row by row.

    A

    One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.

  3. Question 3 of 12.

    A layer computes h=ReLU⁡(Wx+b)\mathbf{h} = \operatorname{ReLU}(\mathbf{W}\mathbf{x} + \mathbf{b}) with

    W=(2−111−13),b=(1−40),x=(12).\mathbf{W} = \begin{pmatrix} 2 & -1 \\ 1 & 1 \\ -1 & 3 \end{pmatrix}, \qquad \mathbf{b} = \begin{pmatrix} 1 \\ -4 \\ 0 \end{pmatrix}, \qquad \mathbf{x} = \begin{pmatrix} 1 \\ 2 \end{pmatrix}.

    Compute h\mathbf{h} and enter it as a column, h1h_1 at the top.

    h

    One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.

  4. Question 4 of 12.

    Compute softmax⁡(1001,1002,1003)\operatorname{softmax}(1001,\allowbreak 1002,\allowbreak 1003). Enter the three probabilities as a column, in the order of the logits, to 3 decimal places.

    p

    One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.

  5. Question 5 of 12.

    A 10-class classifier reports a cross-entropy loss of 23.0 on its very first batch, before any update. What is the most likely explanation?

  6. Question 6 of 12.

    A character-level language model has a mean cross-entropy of 1.5 nats per character on validation text. What is its perplexity? Give 2 decimal places.

    perplexity

    Type a number: 0.25, -2, 3/4 and sqrt(2) all work.

  7. Question 7 of 12.

    A classifier outputs logits z=(0,ln⁡3,0)\mathbf{z} = (0,\allowbreak \ln 3,\allowbreak 0), and the correct class is the first one. The loss is softmax followed by cross-entropy. Find ∂L/∂z\partial L/\partial\mathbf{z} and enter it as a column of three decimals, the first class at the top.

    ∂L/∂z

    One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.

  8. Question 8 of 12.

    A layer Y=XW+b\mathbf{Y} = \mathbf{X}\mathbf{W} + \mathbf{b} receives a batch of two examples with three inputs, and backprop delivers δ=∂L/∂Y\boldsymbol{\delta} = \partial L/\partial\mathbf{Y}:

    X=(1−12021),δ=(12−10).\mathbf{X} = \begin{pmatrix} 1 & -1 & 2 \\ 0 & 2 & 1 \end{pmatrix}, \qquad \boldsymbol{\delta} = \begin{pmatrix} 1 & 2 \\ -1 & 0 \end{pmatrix}.

    Find ∂L/∂W\partial L/\partial\mathbf{W} and enter it row by row.

    ∂L/∂W

    One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.

  9. Question 9 of 12.

    A graph computes u=3xu = 3x and v=x2v = x^2 from the same input, then L=u vL = u\,v. Backpropagate to find dL/dxdL/dx at x=2x = 2.

    dL/dx

    Type a number: 0.25, -2, 3/4 and sqrt(2) all work.

  10. Question 10 of 12.

    A training run's loss falls for 200 steps, then starts jumping up and down by large amounts, and becomes nan at step 350. What should you try first?

  11. Question 11 of 12.

    Adam runs with η=0.001\eta = 0.001 and the default β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999. At the very first step, one parameter's gradient is 0.000030.00003. By how much does that parameter change? Ignore ϵ\epsilon and give the size of the change.

    size of the change

    Type a number: 0.25, -2, 3/4 and sqrt(2) all work.

  12. Question 12 of 12.

    Run the tests before you submit. This question counts as correct if your latest run passes every check.

    Code it

    Write linear_backward(X, W, dY) for a layer Y=XW+b\mathbf{Y} = \mathbf{X}\mathbf{W} + \mathbf{b} in the row convention. Return the tuple (dX, dW, db), with the shapes of X, W and b.

    ⌘+Enter runsPython sleeps until you run code
    Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.
0 of 12 answered. Unanswered questions count as wrong.