Checkpoint
From LSTMs to today's models
Twelve questions across the module: attention and its mask, the size of a transformer, the cost of pretraining, the objectives of post-training, LoRA, and convolution and detection. The numbers are new, and several items target the classic confusions. Work on paper where it helps. You need 80% to pass, and passing means you can read and reason about how today's models are built and adapted, not that you have trained one at scale.
- Question 1 of 12.
A query attends to two keys in dimensions. The raw dot products are and . What weight does scaled dot-product attention give to key 1? Answer to 3 decimal places.
a_1Type a number: 0.25, -2, 3/4 and sqrt(2) all work.
- Question 2 of 12.
A colleague implements the causal mask by setting the blocked scores to 0 before the softmax. What goes wrong?
- Question 3 of 12.
A transformer's context grows from 2,048 tokens to 8,192 tokens. By what factor does the number of attention scores computed per head per layer grow?
factorType a number: 0.25, -2, 3/4 and sqrt(2) all work.
- Question 4 of 12.
A transformer has and 24 blocks, each with the usual four attention matrices and an MLP from to and back. How many numbers do the weight matrices of all its blocks hold? Ignore biases, normalization and the embedding table, and answer in millions, to one decimal place.
millionsType a number: 0.25, -2, 3/4 and sqrt(2) all work.
- Question 5 of 12.
Estimate the pretraining compute, in FLOPs, for a hypothetical 3-billion-parameter model trained on 600 billion tokens. Scientific notation such as 2.5e21 is accepted.
FLOPsType a number: 0.25, -2, 3/4 and sqrt(2) all work.
- Question 6 of 12.
A weight matrix of shape gets a rank-16 LoRA adapter. How many numbers does the adapter train?
trainable parametersType a number: 0.25, -2, 3/4 and sqrt(2) all work.
- Question 7 of 12.
Annotators compared pairs of answers and marked the better one. The team trains the model directly on those pairs, with a loss built from the model's log-probability ratios against a frozen copy of the starting model. No reward model is trained and nothing is sampled during training. What method is this?
- Question 8 of 12.
After fine-tuning on a customer's support conversations, the model's scores on general benchmarks drop. Which changes are reasonable ways to reduce this? Select all that apply.
Choose every option that applies.
- Question 9 of 12.
Convolve with , stride 1, no padding, without flipping the kernel. Enter the output row by row, with at the top left.
YOne entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.
- Question 10 of 12.
A feature map is . A convolutional layer uses kernels with stride 2 and padding 1. What is the height of its output?
heightType a number: 0.25, -2, 3/4 and sqrt(2) all work.
- Question 11 of 12.
You must count the apples on a conveyor belt and measure each apple's area in pixels, even when apples touch. Which approach fits?
- Question 12 of 12.
Run the tests before you submit. This question counts as correct if your latest run passes every check.
Code it Write
causal_weights(S): given a matrix of already-scaled attention scores, return the causal attention weights. Row is a softmax over columns to , and the later columns get exactly 0. Keep it numerically stable.Write your code where the starter saysraise NotImplementedError, then press Run tests. Each check says what it expects.