ml.lab
Python sleeps until you run code
09 From LSTMs to today's models

Lesson 1 of 5

Attention

Attention lets every position look directly at every other position. Score with dot products, turn the scores into weights with a softmax, and average the values. It is the operation that replaced recurrence and the core of every transformer.

About 70 minutes
By the end you can
  • Compute attention scores, weights and the output by hand for one query and a few keys.

  • Derive why dot-product scores are divided by d\sqrt{d}.

  • Write self-attention in matrix form with its shapes, and follow every matrix of a three-token example.

  • Say what queries, keys and values are each for, and why a language model needs the causal mask.

  • Implement scaled dot-product attention and causal self-attention in numpy.

  • Explain why attention replaced recurrence, and what it costs.

It is 2014 and you are translating with LSTMs. An encoder LSTM reads the English sentence one word at a time. Its final state, a vector of fixed size (say a thousand numbers), goes to a decoder LSTM, which writes the French sentence from that vector alone. Every word, its order and its role in the sentence has to fit into those thousand numbers, and has to survive the whole trip through the encoder's steps. Researchers found that such systems got worse as sentences got longer.

An encoder squeezes a whole sentence through one narrow vector, while a decoder with attention reaches back to every encoder state along lines of different thicknessLeft: everything the decoder knows must pass through one fixed-size vector. Right: with attention, each output step reaches back to every input, more strongly to the ones that matter now.

The fix is easy to state: let the decoder look back. At each step of writing, it scores every encoder state for how relevant it is to the word it is about to write, and takes a weighted average of them. That is attention. Within three years, a model built from attention alone had replaced the recurrence itself.

This lesson builds attention in four moves. First, one lookup with numbers small enough to check on paper. Second, why the scores are divided by d\sqrt{d}. Third, the same lookup for every token of a sentence at once, drawn matrix by matrix, with what queries, keys and values are each for and the mask that hides the future. Fourth, why this replaced recurrence and what it costs. The next lesson, Inside a transformer, assembles a whole model from it.

A soft dictionary lookup

Attention is a kind of lookup, so start with the most ordinary lookup there is. In Python, prices = {"apple": 3, "pear": 7} and prices["apple"] gives 3. The lookup compares your query, "apple", with every key, finds the one that matches exactly, and returns that key's value.

Attention keeps the three roles and softens the matching:

  • A query q\mathbf{q}: what the current position is looking for.
  • Keys k1,…,kn\mathbf{k}_1,\allowbreak \ldots,\allowbreak \mathbf{k}_n: what each position offers to be matched against.
  • Values v1,…,vn\mathbf{v}_1,\allowbreak \ldots,\allowbreak \mathbf{v}_n: what each position hands over when it is chosen.

All three are vectors, and queries and keys have the same length dd so that they can be compared. In the translation model, the query is computed from the decoder's current state, and every encoder state supplies one key and one value. The lookup has three steps.

  1. Score every key by its dot product with the query: sj=q⋅kjs_j = \mathbf{q}\cdot\mathbf{k}_j. As you saw in module 1, the dot product is large when two vectors point the same way.
  2. Turn scores into weights with the softmax of module 6: aj=esj/∑iesia_j = e^{s_j} / \sum_i e^{s_i}. The weights are positive and add up to 1.
  3. Average the values with those weights: o=∑jajvj\mathbf{o} = \sum_j a_j\mathbf{v}_j. The output o\mathbf{o} is what the lookup returns.

One lookup on paper

Here is the smallest example that shows every step. The query is q=(2,1)\mathbf{q} = (2,\allowbreak 1). The keys are k1=(1,0)\mathbf{k}_1 = (1,\allowbreak 0), k2=(0,1)\mathbf{k}_2 = (0,\allowbreak 1) and k3=(−1,0)\mathbf{k}_3 = (-1,\allowbreak 0). The values are plain numbers, v1=10v_1 = 10, v2=20v_2 = 20 and v3=30v_3 = 30, so the output is a number too.

Step 1, the scores. Multiply the query and each key entry by entry, then add:

s1=(2)(1)+(1)(0)=2,s2=(2)(0)+(1)(1)=1,s3=(2)(−1)+(1)(0)=−2.s_1 = (2)(1) + (1)(0) = 2, \qquad s_2 = (2)(0) + (1)(1) = 1, \qquad s_3 = (2)(-1) + (1)(0) = -2 .

The query points mostly along k1\mathbf{k}_1, so k1\mathbf{k}_1 scores highest. It points away from k3\mathbf{k}_3, so k3\mathbf{k}_3 scores below zero.

Step 2, the weights. Exponentiate each score. This makes every number positive and keeps their order:

e2≈7.3891,e1≈2.7183,e−2≈0.1353.e^{2} \approx 7.3891, \qquad e^{1} \approx 2.7183, \qquad e^{-2} \approx 0.1353 .

Add them: 7.3891+2.7183+0.1353=10.24277.3891 + 2.7183 + 0.1353 = 10.2427. Divide each by that sum:

a1=7.389110.2427≈0.7214,a2=2.718310.2427≈0.2654,a3=0.135310.2427≈0.0132.a_1 = \frac{7.3891}{10.2427} \approx 0.7214, \qquad a_2 = \frac{2.7183}{10.2427} \approx 0.2654, \qquad a_3 = \frac{0.1353}{10.2427} \approx 0.0132 .

Check: 0.7214+0.2654+0.0132=10.7214 + 0.2654 + 0.0132 = 1.

Step 3, the output. Multiply each value by its weight and add:

o=0.7214×10+0.2654×20+0.0132×30=7.214+5.308+0.396=12.918≈12.92.o = 0.7214 \times 10 + 0.2654 \times 20 + 0.0132 \times 30 = 7.214 + 5.308 + 0.396 = 12.918 \approx 12.92 .

The same computation as one table, laid out the way the widget below lays it out:

keykj\mathbf{k}_jscore sjs_jesje^{s_j}weight aja_jvalue vjv_jajvja_j v_j
k1\mathbf{k}_1(1,0)(1,\allowbreak 0)227.38917.38910.72140.721410107.2147.214
k2\mathbf{k}_2(0,1)(0,\allowbreak 1)112.71832.71830.26540.265420205.3085.308
k3\mathbf{k}_3(−1,0)(-1,\allowbreak 0)−2-20.13530.13530.01320.013230300.3960.396
sum10.242710.24271112.91812.918

Two things are different from the dictionary. Every key contributes something, in proportion to how well it matches, so the output is a blend that leans toward the best match: 12.92 sits between the three values, closest to v1=10v_1 = 10. And everything is differentiable. A hard lookup that returns only the best key has zero gradient almost everywhere: nudging the query does not change which key wins, until suddenly it does. The softmax makes every weight a smooth function of the query and the keys, so gradient descent can learn what queries and keys should look like.

Before you try it live, predict what a longer query does. The next question doubles the query and keeps everything else.

Quick checkDouble the query

In the worked example, the query q=(2,1)\mathbf{q} = (2,\allowbreak 1) attends over k1=(1,0)\mathbf{k}_1 = (1,\allowbreak 0), k2=(0,1)\mathbf{k}_2 = (0,\allowbreak 1) and k3=(−1,0)\mathbf{k}_3 = (-1,\allowbreak 0) with values 10,20,3010,\allowbreak 20,\allowbreak 30, and the output is about 12.9212.92. Now double the query to (4,2)(4,\allowbreak 2) and keep the keys and values. What happens? Decide before you check it in the widget below.

Choose one answer, then check.

The widget below runs the same lookup and answers questions like that one. On the plane, the blue arrow is the query q\mathbf{q} and the three coral arrows are the keys, each tagged with its weight. The values 10, 20 and 30 sit on a number line under the plane as white ticks, and the lime marker on that line is the output oo. The table under it has the columns of the table above.

Why divide by the square root of d

The widget showed that bigger scores give sharper weights. In real models, queries and keys typically have 64 or 128 entries, not 2, and that alone makes the scores large, for a reason that has nothing to do with how well a key matches. The fix is to divide every score by d\sqrt{d}, where dd is the length of the query and key vectors:

sj=q⋅kjd.s_j = \frac{\mathbf{q}\cdot\mathbf{k}_j}{\sqrt{d}} .

Here is where the d\sqrt{d} comes from. Suppose the entries of a query and a key behave like independent random numbers with mean 0 and variance 1, which is roughly what careful initialization aims for. (The variance of a random quantity is the average of its squared distance from its mean. Its standard deviation, the square root of the variance, is its typical distance from the mean.) The dot product q⋅k=q1k1+⋯+qdkd\mathbf{q}\cdot\mathbf{k} = q_1 k_1 + \cdots + q_d k_d is a sum of dd terms, each with mean 0 and variance 1. Its variance is dd, so its typical size is d\sqrt{d}. For d=64d = 64, raw scores have a standard deviation of 8.

Go slower: The variance of a dot product

Assume every qiq_i and kik_i is independent of the others, with E[qi]=E[ki]=0\mathbb{E}[q_i] = \mathbb{E}[k_i] = 0 and E[qi2]=E[ki2]=1\mathbb{E}[q_i^2] = \mathbb{E}[k_i^2] = 1. Here E\mathbb{E} means the average over many random draws, and Var⁡(X)=E[X2]−(E[X])2\operatorname{Var}(X) = \mathbb{E}[X^2] - (\mathbb{E}[X])^2.

One term, its mean. For independent variables, the average of a product is the product of the averages: E[qiki]=E[qi] E[ki]=0⋅0=0.\mathbb{E}[q_i k_i] = \mathbb{E}[q_i]\,\mathbb{E}[k_i] = 0 \cdot 0 = 0 .

One term, its variance. Use the definition of variance with the mean just found: Var⁡(qiki)=E[qi2ki2]−02=E[qi2ki2].\operatorname{Var}(q_i k_i) = \mathbb{E}[q_i^2 k_i^2] - 0^2 = \mathbb{E}[q_i^2 k_i^2] . Squares of independent variables are independent too, so the average of the product splits again: E[qi2ki2]=E[qi2] E[ki2]=1⋅1=1.\mathbb{E}[q_i^2 k_i^2] = \mathbb{E}[q_i^2]\,\mathbb{E}[k_i^2] = 1 \cdot 1 = 1 .

The sum. The terms q1k1,…,qdkdq_1k_1,\allowbreak \ldots,\allowbreak q_dk_d are independent of each other, and variances of independent terms add: Var⁡(q⋅k)=∑i=1dVar⁡(qiki)=1+1+⋯+1⏟d terms=d.\operatorname{Var}(\mathbf{q}\cdot\mathbf{k}) = \sum_{i=1}^{d}\operatorname{Var}(q_i k_i) = \underbrace{1 + 1 + \cdots + 1}_{d \text{ terms}} = d .

The fix. Dividing a random quantity by a constant cc divides its variance by c2c^2. With c=dc = \sqrt{d}: Var⁡ ⁣(q⋅kd)=d(d)2=dd=1,\operatorname{Var}\!\left(\frac{\mathbf{q}\cdot\mathbf{k}}{\sqrt{d}}\right) = \frac{d}{(\sqrt{d})^2} = \frac{d}{d} = 1, whatever the dimension.

Why do large scores hurt? Suppose one score is 8 and seven others are 0. The softmax gives the first key e8/(e8+7)≈2981/2988≈0.998e^8 / (e^8 + 7) \approx 2981 / 2988 \approx 0.998: almost all the weight, from a difference that came from the dimension rather than from meaning. Worse, a saturated softmax stops learning. In the Jacobians lesson you derived the softmax's Jacobian, diag⁡(p)−pp⊤\operatorname{diag}(\mathbf{p}) - \mathbf{p}\mathbf{p}^\top. When p\mathbf{p} is exactly one-hot, say (1,0,…,0)(1,\allowbreak 0,\allowbreak \ldots,\allowbreak 0), both diag⁡(p)\operatorname{diag}(\mathbf{p}) and pp⊤\mathbf{p}\mathbf{p}^\top are the matrix with a single 1 in the top-left corner, so their difference is all zeros. When p\mathbf{p} is nearly one-hot, every entry is nearly 0, so almost no gradient flows back into the scores, and the queries and keys that produced them barely change. It is the same saturation you met with sigmoid and tanh.

Run the experiment: 500 random queries, each compared with its own 8 random keys, for three sizes of dd. The table prints the spread (standard deviation) of the scores before and after dividing by d\sqrt{d}, and the average weight the softmax gives the winning key.

⌘+Enter runs · edit freelyPython sleeps until you run code

The raw spread grows like d\sqrt{d}: about 2, 8 and 23, against 4=2\sqrt{4} = 2, 64=8\sqrt{64} = 8 and 512≈22.6\sqrt{512} \approx 22.6. The winning key's average weight climbs from 0.51 to 0.96. After dividing by d\sqrt{d} the spread stays near 1 and the winning weight stays near 0.35 at every size. Softness is not the goal in itself: a trained model can still learn sharp attention by making its queries and keys large along the directions that matter. The division only stops the dimension from deciding that for it.

To recap: a dot product of two random dd-entry vectors has spread d\sqrt{d}, which would push the softmax toward a single key and stall its gradient. Dividing by d\sqrt{d} brings the spread back to 1. This is scaled dot-product attention, the form the transformer paper introduced and most models since have kept.

The next exercise is one full lookup with d=4d = 4, where the division by 4=2\sqrt{4} = 2 matters.

Work it outOne attention output

With d=4d = 4, the query q=(1,0,1,0)\mathbf{q} = (1,\allowbreak 0,\allowbreak 1,\allowbreak 0) attends over the keys k1=(2,0,2,0)\mathbf{k}_1 = (2,\allowbreak 0,\allowbreak 2,\allowbreak 0), k2=(0,1,0,1)\mathbf{k}_2 = (0,\allowbreak 1,\allowbreak 0,\allowbreak 1) and k3=(0,1,0,−1)\mathbf{k}_3 = (0,\allowbreak 1,\allowbreak 0,\allowbreak -1), whose values are v1=(1,0)\mathbf{v}_1 = (1,\allowbreak 0), v2=(0,1)\mathbf{v}_2 = (0,\allowbreak 1) and v3=(1,1)\mathbf{v}_3 = (1,\allowbreak 1). Compute the scaled dot-product attention output o=∑jajvj\mathbf{o} = \sum_j a_j\mathbf{v}_j. Enter it as a column, first entry on top, each entry to 2 decimal places.

output

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Every token at once: self-attention

So far one query looked at a list of keys. In a language model, every token of a sentence needs its own lookup: in "the cat sat", the token "sat" should be able to find "cat", and "cat" should be able to find "the". In self-attention, the queries, keys and values all come from the same sequence, so every token is at once a query and a key and value for the others. Done one token at a time this would be slow. Stacked as matrices, all the lookups become three matrix products.

The tokens. Stack the TT token vectors of a sequence as the rows of a matrix X\mathbf{X} of shape (T,dmodel)(T,\allowbreak d_{\text{model}}), the row convention from module 2. Row tt is token tt's vector xt\mathbf{x}_t, and dmodeld_{\text{model}} is how many numbers each token carries.

Queries, keys and values. Three learned matrices turn every token into its query, key and value:

Q=XWQ,K=XWK,V=XWV.\mathbf{Q} = \mathbf{X}\mathbf{W}_Q, \qquad \mathbf{K} = \mathbf{X}\mathbf{W}_K, \qquad \mathbf{V} = \mathbf{X}\mathbf{W}_V .

WQ\mathbf{W}_Q and WK\mathbf{W}_K have shape (dmodel,dk)(d_{\text{model}},\allowbreak d_k) and WV\mathbf{W}_V has shape (dmodel,dv)(d_{\text{model}},\allowbreak d_v). Row tt of Q\mathbf{Q} is xtWQ\mathbf{x}_t\mathbf{W}_Q, the query qt\mathbf{q}_t of token tt, and likewise row tt of K\mathbf{K} is its key kt\mathbf{k}_t and row tt of V\mathbf{V} its value vt\mathbf{v}_t. Queries and keys have length dkd_k, which plays the role of dd from the previous section.

Every score at once. Transposing K\mathbf{K} turns its rows into columns, so column ss of K⊤\mathbf{K}^\top is the key ks\mathbf{k}_s. Entry (t,s)(t,\allowbreak s) of the product QK⊤\mathbf{Q}\mathbf{K}^\top is row tt of Q\mathbf{Q} times column ss of K⊤\mathbf{K}^\top, which is qt⋅ks\mathbf{q}_t\cdot\mathbf{k}_s. Divide everything by dk\sqrt{d_k}:

S=QK⊤dk,Sts=qt⋅ksdk.\mathbf{S} = \frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d_k}}, \qquad S_{ts} = \frac{\mathbf{q}_t\cdot\mathbf{k}_s}{\sqrt{d_k}} .

S\mathbf{S} is T×TT \times T: row tt holds token tt's scores for every token ss.

Weights and output. A softmax along each row of S\mathbf{S} gives the weight matrix A\mathbf{A}, whose rows each add up to 1. The output is O=AV\mathbf{O} = \mathbf{A}\mathbf{V}, of shape (T,dv)(T,\allowbreak d_v). Row tt of O\mathbf{O} is ∑sAtsvs\sum_s A_{ts}\mathbf{v}_s: the lookup from the first section, done for every token in parallel.

The diagram below computes every one of these matrices for three tokens, "the cat sat", with dmodel=dk=dv=2d_{\text{model}} = d_k = d_v = 2 and small whole-number weights, so that every number can be checked by hand. It has eight steps: the formula at the top lights up the part each step computes, the new matrix is drawn in lime, and the text under the figure gives the arithmetic. Step 5 adds a causal mask, which the section after next explains.

To recap: self-attention is the lookup of the first section run for every token at once. QK⊤\mathbf{Q}\mathbf{K}^\top holds every score, a row softmax turns each row into weights, and AV\mathbf{A}\mathbf{V} averages the values. For a sequence of TT tokens, the score matrix alone has T2T^2 entries, which matters at the end of this lesson.

What queries, keys and values are for

Why three learned matrices, rather than one vector per token used for all three jobs? Try the simplest alternative and see what goes wrong. If every token used its own vector xt\mathbf{x}_t as its query, key and value, the score of token tt for token ss would be xt⋅xs\mathbf{x}_t\cdot\mathbf{x}_s. That is large for tokens whose vectors point the same way as xt\mathbf{x}_t, so every token would mostly look for tokens like itself, and could only hand over its own vector. Separate matrices remove both limits.

The query and the key can differ. A query says what a token is looking for; a key says what a token can be found by. The verb "sat" looks for its subject, while "cat" is not looking for its verb in the same way. With separate WQ\mathbf{W}_Q and WK\mathbf{W}_K the scores need not be symmetric, and in the diagram they are not: step 3 has qcat⋅ksat=3\mathbf{q}_{\text{cat}}\cdot\mathbf{k}_{\text{sat}} = 3 but qsat⋅kcat=2\mathbf{q}_{\text{sat}}\cdot\mathbf{k}_{\text{cat}} = 2. If the two matrices were equal, every score matrix would be symmetric. Transposing a product reverses the order of the factors and transposes each one, so the raw scores are QK⊤=XWQ(XWK)⊤=XWQWK⊤X⊤\mathbf{Q}\mathbf{K}^\top = \mathbf{X}\mathbf{W}_Q(\mathbf{X}\mathbf{W}_K)^\top = \mathbf{X}\mathbf{W}_Q\mathbf{W}_K^\top\mathbf{X}^\top. With WQ=WK=W\mathbf{W}_Q = \mathbf{W}_K = \mathbf{W} this is XWW⊤X⊤\mathbf{X}\mathbf{W}\mathbf{W}^\top\mathbf{X}^\top, and transposing it by the same rule gives it back unchanged.

The value can differ from the key. Being found and being useful are different jobs. In "the cat that the dog chased sat down", the token "sat" has to find its subject, "cat", by matching on grammatical role, but what it needs to take over is what "cat" means. Separate WV\mathbf{W}_V lets a token be matched on one kind of information and hand over another.

A reader holds a glowing request card toward a shelf of books; the spine of each book glows in proportion to how well it matches the card, and streams of pages flow from every book toward the reader, thickest from the best-matching books, merging into one blended pageThe query is the request, the keys are the labels on the spines, and the values are what the books hand over. Every book contributes, in proportion to how well its label matches.

None of these roles is programmed. WQ\mathbf{W}_Q, WK\mathbf{W}_K and WV\mathbf{W}_V start random and are learned by gradient descent through the softmax, which is why the smoothness of the soft lookup mattered.

The causal mask

A language model is trained to predict token t+1t + 1 from tokens 11 to tt, at every position of a training sequence at once, exactly as your character models were in module 7. If position tt could attend to position t+1t + 1, it could read the answer instead of predicting it. Training loss would fall to nearly zero, and the model would be useless for generation, where the future does not exist yet.

The fix is a mask M\mathbf{M} added to the scores before the softmax:

Mts={0if s≤t (the same or an earlier token),−∞if s>t (a later token).M_{ts} = \begin{cases} 0 & \text{if } s \le t \text{ (the same or an earlier token)}, \\ -\infty & \text{if } s > t \text{ (a later token)}. \end{cases}

Any number plus −∞-\infty is −∞-\infty, and e−∞=0e^{-\infty} = 0, so blocked positions get weight exactly 0, and each row's softmax spreads its weight over the visible positions only. The full formula becomes softmax⁡(QK⊤/dk+M) V\operatorname{softmax}(\mathbf{Q}\mathbf{K}^\top/\sqrt{d_k} + \mathbf{M})\,\mathbf{V}.

Go back to steps 5 and 6 of the diagram to see what the mask changes. Without it, the row for "the" would be softmax⁡(−1.41,0,−1.41)≈(0.164,0.673,0.164)\operatorname{softmax}(-1.41,\allowbreak 0,\allowbreak -1.41) \approx (0.164,\allowbreak 0.673,\allowbreak 0.164): the first token would take 67% of its information from "cat", the very word it is trained to predict. With the mask it is (1,0,0)(1,\allowbreak 0,\allowbreak 0). In the same way, "cat" would put 0.576 of its weight on "sat", its own prediction target; with the mask, that weight is 0 and the rest is renormalized to (0.330,0.670,0)(0.330,\allowbreak 0.670,\allowbreak 0).

The picture below uses the reference implementation in mlref.attention on six random token vectors. The first heatmap has no mask; the second has the causal mask, so everything above the diagonal is 0.

⌘+Enter runs · edit freelyPython sleeps until you run code

The mask has one more consequence: since token tt reads only tokens 1 to tt, its output never changes when later tokens are added. The code exercise below tests exactly that, and the next lesson uses it to make generation fast. First, what happens when the mask is forgotten?

Quick checkForgetting the mask

You train a character language model built from self-attention, but you forget the causal mask. Training runs on windows of text with the targets shifted by one, as in module 7. What do you expect?

Choose one answer, then check.

Now put the whole computation together by hand: scaled scores, weights, the output, the causal mask, and what happens without the d\sqrt{d}.

On paperAttention for three keys

A query attends over three keys in d=4d = 4 dimensions: q=(1,1,1,1)\mathbf{q} = (1,\allowbreak 1,\allowbreak 1,\allowbreak 1), k1=(1,1,0,0)\mathbf{k}_1 = (1,\allowbreak 1,\allowbreak 0,\allowbreak 0), k2=(1,−1,0,0)\mathbf{k}_2 = (1,\allowbreak -1,\allowbreak 0,\allowbreak 0), k3=(1,1,1,1)\mathbf{k}_3 = (1,\allowbreak 1,\allowbreak 1,\allowbreak 1), with values v1=(4,0)\mathbf{v}_1 = (4,\allowbreak 0), v2=(0,4)\mathbf{v}_2 = (0,\allowbreak 4), v3=(2,2)\mathbf{v}_3 = (2,\allowbreak 2).

  1. Compute the scaled scores q⋅kj/d\mathbf{q}\cdot\mathbf{k}_j/\sqrt{d} and the attention weights, to 3 decimal places.
  2. Compute the output ∑jajvj\sum_j a_j\mathbf{v}_j, to 3 decimal places.
  3. The query now belongs to position 2 of a causal language model, and k3\mathbf{k}_3 to position 3, while k1\mathbf{k}_1 and k2\mathbf{k}_2 belong to positions 1 and 2. Recompute the weights and the output.
  4. Redo part 1 without dividing by d\sqrt{d}. What changes, and why?

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

The three weights from part 1, in the order k1, k2, k3

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

A few more lookups for practice, each with new numbers:

Practice setAttention weights reps
3 correct in a row completes the set. A miss starts the count again.

A query q=(2,2,1,0)\mathbf{q} = (2,\allowbreak 2,\allowbreak 1,\allowbreak 0) attends over three keys k1=(0,1,2,2)\mathbf{k}_1 = (0,\allowbreak 1,\allowbreak 2,\allowbreak 2), k2=(1,1,−1,2)\mathbf{k}_2 = (1,\allowbreak 1,\allowbreak -1,\allowbreak 2), k3=(0,−1,−1,2)\mathbf{k}_3 = (0,\allowbreak -1,\allowbreak -1,\allowbreak 2), in d=4d = 4 dimensions. Find the attention weights aj=softmax⁡j(q⋅kj/d)a_j = \operatorname{softmax}_j(\mathbf{q}\cdot\mathbf{k}_j/\sqrt{d}). Give each to 3 decimal places.

a

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Then build it in code. The exercise checks your attention against the reference, reproduces the diagram's weights for "the cat sat", and tests causality directly: changing later tokens must not change earlier outputs.

Code itBuild attention and self-attention

Implement attention in numpy, one sequence at a time (no batch dimension), with rows as tokens.

  1. causal_mask(T): a (T,T)(T,\allowbreak T) boolean array, True where a query may not look, which is strictly above the diagonal.
  2. attention(Q, K, V, mask=None): scaled dot-product attention. Blocked scores become -np.inf, and the softmax subtracts each row's maximum before exponentiating. Return the output and the weights, in that order.
  3. self_attention(X, Wq, Wk, Wv, causal=True): project the tokens to queries, keys and values, then attend, with the causal mask when causal is true.

The tests compare with the reference mlref.attention, reproduce the diagram's weights for "the cat sat", check that the mask gives later positions exactly zero weight, and check causality directly: changing later tokens must not change earlier outputs. The next lesson builds several heads and a whole block on top of this.

⌘+Enter runsPython sleeps until you run code
Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.

A transformer layer runs several of these attentions side by side, each with its own WQ\mathbf{W}_Q, WK\mathbf{W}_K and WV\mathbf{W}_V, so that different heads can look for different things. That, and everything else around attention, is the next lesson.

Why attention replaced recurrence

Attention entered in 2014 as a helper for recurrent networks. In 2017 the transformer dropped the recurrence and kept only attention, and within a few years it had replaced recurrent networks for most sequence tasks. Three facts explain why, and the third is the price.

Short paths. In a recurrent network, information from position ss reaches position tt only through t−st - s steps, and the gradient that trains the connection is a product of t−st - s Jacobians. As you derived in backpropagation through time, such a product shrinks or grows geometrically, governed by the eigenvalues and singular values of module 4. In self-attention, position tt reads position ss directly, in one layer, whatever the distance. There is no long product to vanish.

Two rows of eight glowing beads: in the top row each bead connects only to its neighbor, and a signal fades as it passes link by link from the first bead to the last; in the bottom row the last bead connects directly to every earlier bead by one line eachA recurrent network passes information link by link, and the signal weakens at every hop. Attention connects the last position to every earlier one in a single step.

Parallel training. A recurrent network must compute h1\mathbf{h}_1 before h2\mathbf{h}_2, and h2\mathbf{h}_2 before h3\mathbf{h}_3. A transformer computes every position of a training sequence at once, with a few large matrix multiplications, which is exactly the work GPUs are built for. The causal mask is what makes this legal for next-token prediction.

The cost. Using the cost rule from module 2, QK⊤\mathbf{Q}\mathbf{K}^\top takes T⋅dk⋅TT \cdot d_k \cdot T multiply-adds and AV\mathbf{A}\mathbf{V} takes T⋅T⋅dvT \cdot T \cdot d_v. The score matrix has T2T^2 entries per head per layer, so doubling the context quadruples that work and memory. For T=8,192T = 8{,}192 and 32 heads, a single layer computes 32×81922=32×67,108,864=2,147,483,64832 \times 8192^2 = 32 \times 67{,}108{,}864 = 2{,}147{,}483{,}648 scores. A recurrent network's cost per step does not depend on how much it has read, and it carries a fixed-size state, while a transformer that generates text must keep something for every earlier token (the next lesson shows what).

To recap: attention replaced recurrence because every position reaches every other in one step and a whole sequence trains in parallel. It pays with a cost that grows with the square of the sequence length. Much engineering goes into that trade: attention kernels that never store the full score matrix at once, and a revival of recurrent and state-space designs, often gated, that carry a fixed-size state again.

The last exercise asks you to put this argument in your own words, for someone who knows recurrent networks well.

In your own wordsWhy attention replaced recurrence

A colleague who trained LSTMs years ago asks why transformers replaced them. Explain the two main reasons and what attention costs in exchange, using what you know from modules 4 and 7 about long products of Jacobians. Then say honestly what recurrent designs still offer.

A few sentences first: 0 of 60 characters.

Saved in this browser as you type.

Next: Inside a transformer