Checkpoint

Neural networks from scratch

Twelve questions across the whole module: layers and their parameters, why the nonlinearity matters, softmax and cross-entropy, backpropagation through a linear layer, and reading and optimizing a training run. The numbers are new, and several questions aim at the classic confusions. Work on paper before answering. You need 80% to pass.

1
How many parameters, weights and biases together, does a network with layer sizes 100-64-64-5 have?
parameters
2
In the column convention, a network computes $\mathbf{h} = \mathbf{W}_1\mathbf{x}$ and then $\mathbf{y} = \mathbf{W}_2\mathbf{h}$ , with no activation and no biases, where
$\mathbf{W}_1 = \begin{pmatrix} 1 & 2 \\ 0 & 1 \end{pmatrix}, \qquad \mathbf{W}_2 = \begin{pmatrix} 1 & -1 \\ 2 & 0 \end{pmatrix}.$
Find the single matrix $\mathbf{A}$ with $\mathbf{y} = \mathbf{A}\mathbf{x}$ for every $\mathbf{x}$ .
A
3
A layer computes $\mathbf{h} = \operatorname{ReLU}(\mathbf{W}\mathbf{x} + \mathbf{b})$ with
$\mathbf{W} = \begin{pmatrix} 2 & -1 \\ 1 & 1 \\ -1 & 3 \end{pmatrix}, \qquad \mathbf{b} = \begin{pmatrix} 1 \\ -4 \\ 0 \end{pmatrix}, \qquad \mathbf{x} = \begin{pmatrix} 1 \\ 2 \end{pmatrix}.$
Compute $\mathbf{h}$ .
h
4
Compute $\operatorname{softmax}(1001, 1002, 1003)$ , to 3 decimal places.
p
5
A 10-class classifier reports a cross-entropy loss of 23.0 on its very first batch, before any update. What is the most likely explanation?
6
A character-level language model has a mean cross-entropy of 1.5 nats per character on validation text. What is its perplexity? Give 2 decimal places.
perplexity
7
A classifier outputs logits $\mathbf{z} = (0, \ln 3, 0)$ , and the correct class is the first one. The loss is softmax followed by cross-entropy. Find $\partial L/\partial\mathbf{z}$ .
∂L/∂z
8
A layer $\mathbf{Y} = \mathbf{X}\mathbf{W} + \mathbf{b}$ receives a batch of two examples with three inputs, and backprop delivers $\boldsymbol{\delta} = \partial L/\partial\mathbf{Y}$ :
$\mathbf{X} = \begin{pmatrix} 1 & -1 & 2 \\ 0 & 2 & 1 \end{pmatrix}, \qquad \boldsymbol{\delta} = \begin{pmatrix} 1 & 2 \\ -1 & 0 \end{pmatrix}.$
Find $\partial L/\partial\mathbf{W}$ .
∂L/∂W
9
A graph computes $u = 3x$ and $v = x^2$ from the same input, then $L = u\,v$ . Backpropagate to find $dL/dx$ at $x = 2$ .
dL/dx
10
A training run's loss falls for 200 steps, then starts jumping up and down by large amounts, and becomes nan at step 350. What should you try first?
11
Adam runs with $\eta = 0.001$ and the default $\beta_1 = 0.9$ , $\beta_2 = 0.999$ . At the very first step, one parameter's gradient is $0.00003$ . By how much does that parameter change? Ignore $\epsilon$ and give the size of the change.
size of the change
12
Code it
Write linear_backward(X, W, dY) for a layer $\mathbf{Y} = \mathbf{X}\mathbf{W} + \mathbf{b}$ in the row convention. Return the tuple (dX, dW, db), with the shapes of X, W and b.
⌘↵ runsPython sleeps until you run code

12 still unanswered. Unanswered questions count as wrong.