Checkpoint

From LSTMs to today's models

Twelve questions across the module: attention and its mask, the cost of pretraining, the objectives of post-training, LoRA, and convolution and detection. The numbers are new, and several items target the classic confusions. Work on paper where it helps. You need 80% to pass, and passing means you can read and reason about how today's models are built and adapted, not that you have trained one at scale.

1
A query attends to two keys in $d = 16$ dimensions. The raw dot products are $\mathbf{q}\cdot\mathbf{k}_1 = 12$ and $\mathbf{q}\cdot\mathbf{k}_2 = 4$ . What weight does scaled dot-product attention give to key 1? Answer to 3 decimal places.
a_1
2
A colleague implements the causal mask by setting the blocked scores to 0 before the softmax. What goes wrong?
3
A transformer's context grows from 2,048 tokens to 8,192 tokens. By what factor does the number of attention scores computed per head per layer grow?
factor
4
Estimate the pretraining compute, in FLOPs, for a hypothetical 3-billion-parameter model trained on 600 billion tokens. Scientific notation such as 2.5e21 is accepted.
FLOPs
5
A weight matrix of shape $2048 \times 8192$ gets a rank-16 LoRA adapter. How many numbers does the adapter train?
trainable parameters
6
Annotators compared pairs of answers and marked the better one. The team trains the model directly on those pairs, with a loss built from the model's log-probability ratios against a frozen copy of the starting model. No reward model is trained and nothing is sampled during training. What method is this?
7
In a DPO run with $\beta = 0.5$ , for one pair the policy's log-ratio against the reference is $-1$ for the preferred response and $+1$ for the rejected one: training has so far moved it the wrong way. Each pair's gradient is scaled by $\sigma(-m)$ , where $m$ is the quantity inside the loss's sigmoid. What is that weight for this pair? Answer to 3 decimal places.
σ(−m)
8
After fine-tuning on a customer's support conversations, the model's scores on general benchmarks drop. Which changes are reasonable ways to reduce this? Select all that apply.
9
Convolve $\mathbf{X} = \begin{bmatrix} 1 & 0 & 2 \\ 3 & 1 & 0 \\ 0 & 2 & 1 \end{bmatrix}$ with $\mathbf{K} = \begin{bmatrix} 1 & 2 \\ 0 & -1 \end{bmatrix}$ , stride 1, no padding, without flipping the kernel.
Y
10
A feature map is $56 \times 56$ . A convolutional layer uses $3 \times 3$ kernels with stride 2 and padding 1. What is the height of its output?
height
11
You must count the apples on a conveyor belt and measure each apple's area in pixels, even when apples touch. Which approach fits?
12
Code it
Write causal_weights(S): given a $(T, T)$ matrix of already-scaled attention scores, return the causal attention weights. Row $t$ is a softmax over columns $0$ to $t$ , and the later columns get exactly 0. Keep it numerically stable.
⌘↵ runsPython sleeps until you run code

12 still unanswered. Unanswered questions count as wrong.