Checkpoint
Neural networks from scratch
Twelve questions across the whole module: layers and their parameters, why the nonlinearity matters, softmax and cross-entropy, backpropagation through a linear layer, and reading and optimizing a training run. The numbers are new, and several questions aim at the classic confusions. Work on paper before answering. You need 80% to pass.
- Question 1 of 12.
How many parameters, weights and biases together, does a network with layer sizes 100-64-64-5 have?
parametersType a number: 0.25, -2, 3/4 and sqrt(2) all work.
- Question 2 of 12.
In the column convention, a network computes and then , with no activation and no biases, where
Find the single matrix with for every , and enter it row by row.
AOne entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.
- Question 3 of 12.
A layer computes with
Compute and enter it as a column, at the top.
hOne entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.
- Question 4 of 12.
Compute . Enter the three probabilities as a column, in the order of the logits, to 3 decimal places.
pOne entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.
- Question 5 of 12.
A 10-class classifier reports a cross-entropy loss of 23.0 on its very first batch, before any update. What is the most likely explanation?
- Question 6 of 12.
A character-level language model has a mean cross-entropy of 1.5 nats per character on validation text. What is its perplexity? Give 2 decimal places.
perplexityType a number: 0.25, -2, 3/4 and sqrt(2) all work.
- Question 7 of 12.
A classifier outputs logits , and the correct class is the first one. The loss is softmax followed by cross-entropy. Find and enter it as a column of three decimals, the first class at the top.
∂L/∂zOne entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.
- Question 8 of 12.
A layer receives a batch of two examples with three inputs, and backprop delivers :
Find and enter it row by row.
∂L/∂WOne entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.
- Question 9 of 12.
A graph computes and from the same input, then . Backpropagate to find at .
dL/dxType a number: 0.25, -2, 3/4 and sqrt(2) all work.
- Question 10 of 12.
A training run's loss falls for 200 steps, then starts jumping up and down by large amounts, and becomes nan at step 350. What should you try first?
- Question 11 of 12.
Adam runs with and the default , . At the very first step, one parameter's gradient is . By how much does that parameter change? Ignore and give the size of the change.
size of the changeType a number: 0.25, -2, 3/4 and sqrt(2) all work.
- Question 12 of 12.
Run the tests before you submit. This question counts as correct if your latest run passes every check.
Code it Write
linear_backward(X, W, dY)for a layer in the row convention. Return the tuple(dX, dW, db), with the shapes ofX,Wandb.Write your code where the starter saysraise NotImplementedError, then press Run tests. Each check says what it expects.