Practice
Reps until it is automatic
Practice sets generate a fresh problem every time, check your answer, and show the worked solution. Review brings back the exercises you have missed so far.
Each set deals a fresh problem every time. Get enough right in a row to complete it, then keep going for as long as it helps.
Multiply matching entries and add. Vectors with 2 to 4 entries and small integers.
Compute for and .
Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.
Project a onto the line through b with the formula (a·b / b·b) b.
Find for and .
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Scale vectors and add them entry by entry: linear combinations of two or three vectors, and solving for an unknown vector.
Find the vector that satisfies , where and . Enter as a column.
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Norms from Pythagoras, distances between points, unit vectors, the L1 norm, and cosine similarity.
How far apart are the points and ? Find the distance .
Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.
Compute Ax with the row view (dot products), then check it with the column view (a combination of columns).
Compute for and .
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Multiply matrices entry by entry: each entry is a row of the left matrix dotted with a column of the right one.
Compute for and .
One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Predict shapes before you compute: products, transposes, dense layers, numpy broadcasting, parameter counts and multiply-add counts.
has shape and has shape . What is the shape of ?
2x2 determinants as ad - bc, 3x3 by cofactor expansion, and the rules for products, transposes, inverses and scaling.
Find for by cofactor expansion along the first row.
Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.
Invert 2x2 matrices with the swap-negate-divide formula, and use the inverse to find which input lands on a given output.
Solve for and . In words: which input does send to ?
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Solve det(A - λI) = 0 for a 2x2 matrix, check with the trace and determinant, and see the eigenvectors that go with each eigenvalue.
Find the eigenvalues of . Enter them as a column, largest first.
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Compute A^k v by writing v in the eigenbasis, and predict which eigenvalue takes over after many steps.
The matrix has eigenvectors with and with . Use them to compute for .
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Power, product and chain rules, and the derivatives of exp, ln, sigmoid, tanh and ReLU, evaluated at a point.
Let . Find .
Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.
Partial derivatives collected into a gradient: polynomials, exponentials, quadratic forms, linear-regression loss, and directional derivatives.
Let . Find the gradient at .
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Matrices of partial derivatives: polynomial maps, linear layers, elementwise activations, softmax, and the chain rule as a matrix product.
Let . Find the Jacobian at .
One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Push numbers through small networks: a ReLU hidden layer, a linear or sigmoid output, and a batch in the row convention used by code.
In code, a layer processes a batch with one example per row: , where the bias row is added to every row. Compute for , and .
One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Turn logits into probabilities the stable way, with and without temperature, then compute the cross-entropy loss and its gradient p - y.
A freshly initialized classifier gives all classes the same logit. What cross-entropy loss (natural log) should the first training step report? Give the answer to 3 decimal places.
Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.
Gradients through a linear layer with a batch (dL/dW, dL/dX, dL/db), through a ReLU, and where a value is used twice.
A single ReLU unit and a squared-error loss: , , . With , , and target , find .
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Run a recurrent network forward by hand: two steps of a scalar RNN, or one step of a two-unit RNN with weight matrices.
A two-unit RNN computes with applied to each entry. Use , , , the previous state and the input . Find to 3 decimal places.
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Compute the four gates of a scalar LSTM, then the new cell state and hidden state, by hand.
A scalar LSTM (hidden size 1) has these weights:
| gate | (times ) | (times ) | (bias) |
|---|---|---|---|
| forget | -1 | 0 | 0 |
| input | 1 | 0 | 1 |
| candidate | 2 | 0 | 0 |
| output | 2 | -1 | 1 |
With , and , find and to 3 decimal places. Enter them as a column, first.
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Count the weights of LSTM layers, compare with a plain RNN, and count whole character-level models.
An LSTM layer has hidden size and input size , with one bias vector per gate. How many parameters does it have?
Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.
Scaled dot-product attention for one query and three keys: scores, softmax weights, the weighted average of values, and a causal mask.
A query attends over three keys , , , in dimensions. Find the attention weights . Give each to 3 decimal places.
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Slide a small kernel over a small image by hand, with stride 1 or 2, and predict output sizes with padding and stride.
Convolve the image with the kernel , stride 1, no padding. As in deep-learning libraries, slide the kernel without flipping it: each output is the sum of the kernel times the patch under it, entry by entry.
One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.