ml.lab
Python sleeps until you run code
09 From LSTMs to today's models

Checkpoint

From LSTMs to today's models

Twelve questions across the module: attention and its mask, the size of a transformer, the cost of pretraining, the objectives of post-training, LoRA, and convolution and detection. The numbers are new, and several items target the classic confusions. Work on paper where it helps. You need 80% to pass, and passing means you can read and reason about how today's models are built and adapted, not that you have trained one at scale.

  1. Question 1 of 12.

    A query attends to two keys in d=16d = 16 dimensions. The raw dot products are q⋅k1=12\mathbf{q}\cdot\mathbf{k}_1 = 12 and q⋅k2=4\mathbf{q}\cdot\mathbf{k}_2 = 4. What weight does scaled dot-product attention give to key 1? Answer to 3 decimal places.

    a_1

    Type a number: 0.25, -2, 3/4 and sqrt(2) all work.

  2. Question 2 of 12.

    A colleague implements the causal mask by setting the blocked scores to 0 before the softmax. What goes wrong?

  3. Question 3 of 12.

    A transformer's context grows from 2,048 tokens to 8,192 tokens. By what factor does the number of attention scores computed per head per layer grow?

    factor

    Type a number: 0.25, -2, 3/4 and sqrt(2) all work.

  4. Question 4 of 12.

    A transformer has d=1024d = 1024 and 24 blocks, each with the usual four d×dd \times d attention matrices and an MLP from dd to 4d4d and back. How many numbers do the weight matrices of all its blocks hold? Ignore biases, normalization and the embedding table, and answer in millions, to one decimal place.

    millions

    Type a number: 0.25, -2, 3/4 and sqrt(2) all work.

  5. Question 5 of 12.

    Estimate the pretraining compute, in FLOPs, for a hypothetical 3-billion-parameter model trained on 600 billion tokens. Scientific notation such as 2.5e21 is accepted.

    FLOPs

    Type a number: 0.25, -2, 3/4 and sqrt(2) all work.

  6. Question 6 of 12.

    A weight matrix of shape 2048×81922048 \times 8192 gets a rank-16 LoRA adapter. How many numbers does the adapter train?

    trainable parameters

    Type a number: 0.25, -2, 3/4 and sqrt(2) all work.

  7. Question 7 of 12.

    Annotators compared pairs of answers and marked the better one. The team trains the model directly on those pairs, with a loss built from the model's log-probability ratios against a frozen copy of the starting model. No reward model is trained and nothing is sampled during training. What method is this?

  8. Question 8 of 12.

    After fine-tuning on a customer's support conversations, the model's scores on general benchmarks drop. Which changes are reasonable ways to reduce this? Select all that apply.

    Choose every option that applies.

  9. Question 9 of 12.

    Convolve X=[102310021]\mathbf{X} = \begin{bmatrix} 1 & 0 & 2 \\ 3 & 1 & 0 \\ 0 & 2 & 1 \end{bmatrix} with K=[120−1]\mathbf{K} = \begin{bmatrix} 1 & 2 \\ 0 & -1 \end{bmatrix}, stride 1, no padding, without flipping the kernel. Enter the output Y\mathbf{Y} row by row, with Y00Y_{00} at the top left.

    Y

    One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box.

  10. Question 10 of 12.

    A feature map is 56×5656 \times 56. A convolutional layer uses 3×33 \times 3 kernels with stride 2 and padding 1. What is the height of its output?

    height

    Type a number: 0.25, -2, 3/4 and sqrt(2) all work.

  11. Question 11 of 12.

    You must count the apples on a conveyor belt and measure each apple's area in pixels, even when apples touch. Which approach fits?

  12. Question 12 of 12.

    Run the tests before you submit. This question counts as correct if your latest run passes every check.

    Code it

    Write causal_weights(S): given a (T,T)(T,\allowbreak T) matrix of already-scaled attention scores, return the causal attention weights. Row tt is a softmax over columns 00 to tt, and the later columns get exactly 0. Keep it numerically stable.

    ⌘+Enter runsPython sleeps until you run code
    Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.
0 of 12 answered. Unanswered questions count as wrong.