ml.lab
Python sleeps until you run code
01 Vectors and the dot product

Lesson 4 of 4

Length and similarity

How long is a vector, how far apart are two of them, and how closely do their directions agree? Cosine similarity is the dot product with length divided out, and it is how embedding search compares meaning.

About 80 minutes
By the end you can
  • Compute the L2 and L1 lengths of a vector in any number of dimensions, and normalize a vector to length 1.

  • Choose between distance and angle for a comparison, and compute cosine similarity by hand and in code.

  • Prove the Cauchy-Schwarz inequality and explain why it keeps cosine similarity between -1 and 1.

  • Build nearest-neighbor search and word analogies over a small embedding table.

Suppose you are building search for a help center. As a first attempt, you describe each article by how many times three words appear in it: password, reset and invoice. A user types "reset password", which becomes the query vector q=(1,1,0)\mathbf{q} = (1,\allowbreak 1,\allowbreak 0).

Article a\mathbf{a} is a short note on resetting your password: a=(2,2,0)\mathbf{a} = (2,\allowbreak 2,\allowbreak 0). Article f\mathbf{f} is a long billing FAQ that mentions passwords along the way: f=(6,1,9)\mathbf{f} = (6,\allowbreak 1,\allowbreak 9). Score both with the dot product:

q⋅a=(1)(2)+(1)(2)+(0)(0)=4,q⋅f=(1)(6)+(1)(1)+(0)(9)=7.\mathbf{q}\cdot\mathbf{a} = (1)(2) + (1)(2) + (0)(0) = 4, \qquad \mathbf{q}\cdot\mathbf{f} = (1)(6) + (1)(1) + (0)(9) = 7.

The billing FAQ wins, mostly because it is long. Worse, if the password note is accidentally saved with its text pasted in twice, its counts double to (4,4,0)(4,\allowbreak 4,\allowbreak 0) and its score doubles to 88, although it has not become any more relevant. The dot product mixes two things: how closely two vectors point the same way, and how long they are. For search we usually want only the first.

This lesson pulls the two apart. First length, then direction, then the inequality that makes "angle" meaningful in any number of dimensions, and finally a small embedding search you build yourself.

How long is a vector

Before we can divide length out of a comparison, we need a way to measure it. In the plane, a vector and its two entries form a right triangle: the entries are the two short sides (the legs), and the arrow is the longest side (the hypotenuse). Pythagoras gives its length.

The figure shows exactly that triangle. The vector v\mathbf{v} is drawn from the origin, its dashed legs show the two entries, and the second formula under the sentence computes the length by Pythagoras with the current numbers.

For v=(3,4)\mathbf{v} = (3,\allowbreak 4) that is 32+42=9+16=25=5\sqrt{3^2 + 4^2} = \sqrt{9 + 16} = \sqrt{25} = 5. In three dimensions, Pythagoras is applied twice: v=(1,2,2)\mathbf{v} = (1,\allowbreak 2,\allowbreak 2) has length 1+4+4=9=3\sqrt{1 + 4 + 4} = \sqrt{9} = 3. The picture below shows why twice is enough.

A rectangular box drawn from one corner at the origin, with the diagonal across its floor and the diagonal through the box from the origin to the opposite top corner, forming a right triangle with one vertical edgePythagoras twice. The floor diagonal and a vertical edge meet at a right angle, so together with the arrow through the box they form a right triangle.

Go slower: Pythagoras twice, for three dimensions

Take v=(v1,v2,v3)\mathbf{v} = (v_1,\allowbreak v_2,\allowbreak v_3). Its shadow on the floor (the plane where the third entry is 0) is (v1,v2,0)(v_1,\allowbreak v_2,\allowbreak 0).

Step 1, the floor. The shadow lies in a plane, so ordinary Pythagoras gives its length: v12+v22\sqrt{v_1^2 + v_2^2}.

Step 2, straight up. From the tip of the shadow, go straight up by v3v_3 to reach the tip of v\mathbf{v}. Straight up is perpendicular to the floor, so the shadow, the vertical segment and v\mathbf{v} form a right triangle with v\mathbf{v} as its hypotenuse. Pythagoras once more: ∣v∣2=(v12+v22)2+v32.|\mathbf{v}|^2 = \left(\sqrt{v_1^2 + v_2^2}\right)^2 + v_3^2. Squaring a square root gives back what was under it: ∣v∣2=v12+v22+v32.|\mathbf{v}|^2 = v_1^2 + v_2^2 + v_3^2.

Step 3, the pattern. For (1,2,2)(1,\allowbreak 2,\allowbreak 2): the shadow has length 1+4=5\sqrt{1 + 4} = \sqrt{5}, and then ∣v∣=(5)2+22=5+4=3|\mathbf{v}| = \sqrt{(\sqrt{5})^2 + 2^2} = \sqrt{5 + 4} = 3. Each extra dimension adds one more perpendicular leg and one more squared entry under the root. Beyond three dimensions there is nothing to draw, so we take the pattern as the definition: every axis is perpendicular to all the others, and each one adds its squared entry.

The length (also called the Euclidean length, the magnitude or the L2 norm) of a vector with nn entries is

∣v∣=v12+v22+⋯+vn2=v⋅v.|\mathbf{v}| = \sqrt{v_1^2 + v_2^2 + \cdots + v_n^2} = \sqrt{\mathbf{v}\cdot\mathbf{v}}.

The second form is the one to remember: the dot product of a vector with itself is the sum of its squared entries. Many books and papers write the length with double bars, ∥v∥\|\mathbf{v}\| or ∥v∥2\|\mathbf{v}\|_2, to keep it apart from the absolute value of a number. It means the same thing as our ∣v∣|\mathbf{v}|.

Two properties follow from the formula. First, the length is zero only for the zero vector, because a sum of squares is zero only when every entry is zero. Second, scaling a vector by cc scales its length by ∣c∣|c|, the size of cc without its sign. One move per line:

  1. By the definition, ∣c v∣=(cv1)2+⋯+(cvn)2|c\,\mathbf{v}| = \sqrt{(c v_1)^2 + \cdots + (c v_n)^2}.
  2. Square each product, (cvi)2=c2vi2(c v_i)^2 = c^2 v_i^2: this is c2v12+⋯+c2vn2\sqrt{c^2 v_1^2 + \cdots + c^2 v_n^2}.
  3. Factor out c2c^2: c2 (v12+⋯+vn2)\sqrt{c^2\,(v_1^2 + \cdots + v_n^2)}.
  4. The square root of a product is the product of the square roots: c2  v12+⋯+vn2\sqrt{c^2}\;\sqrt{v_1^2 + \cdots + v_n^2}.
  5. c2=∣c∣\sqrt{c^2} = |c|, and the second root is ∣v∣|\mathbf{v}|: so ∣c v∣=∣c∣ ∣v∣|c\,\mathbf{v}| = |c|\,|\mathbf{v}|.

The step c2=∣c∣\sqrt{c^2} = |c| matters: lengths are never negative, so −2v-2\mathbf{v} is twice as long as v\mathbf{v}, not −2-2 times. Check: −2 (3,4)=(−6,−8)-2\,(3,\allowbreak 4) = (-6,\allowbreak -8) has length 36+64=100=10=2×5\sqrt{36 + 64} = \sqrt{100} = 10 = 2 \times 5.

Unit vectors

Often only the direction of a vector matters. A unit vector has length 1, so it carries a direction and nothing else. Dividing any nonzero vector by its length gives the unit vector pointing the same way, a step called normalizing:

v^=v∣v∣.\hat{\mathbf{v}} = \frac{\mathbf{v}}{|\mathbf{v}|}.

The hat means "the unit vector in the direction of". Its length is 1 by the scaling rule with c=1/∣v∣c = 1/|\mathbf{v}|, which is positive: ∣v^∣=1∣v∣ ∣v∣=1|\hat{\mathbf{v}}| = \frac{1}{|\mathbf{v}|}\,|\mathbf{v}| = 1. For v=(3,4)\mathbf{v} = (3,\allowbreak 4), v^=(35,45)=(0.6,0.8)\hat{\mathbf{v}} = \left(\frac{3}{5},\allowbreak \frac{4}{5}\right) = (0.6,\allowbreak 0.8), and indeed 0.62+0.82=0.36+0.64=10.6^2 + 0.8^2 = 0.36 + 0.64 = 1.

Arrows of different lengths leaving the origin in different directions, and a circle of radius 1 centered on the origin. Each arrow has a bright dot on the circle in its own direction: where a long arrow crosses the circle, or straight beyond the tip of a short arrow that ends inside itNormalizing moves each tip along its own arrow onto the circle of radius 1. The direction stays; the length becomes 1.

In the plane, every unit vector ends on the circle of radius 1 around the origin, the unit circle. Normalizing slides a tip along its own arrow, in or out, until it reaches that circle.

Normalizing is two steps: find the length, then divide every entry by it. The next vector has three entries and a zero among them.

Work it outNormalize a vector

Normalize v=(4,0,−3)\mathbf{v} = (4,\allowbreak 0,\allowbreak -3): find the unit vector v^\hat{\mathbf{v}} that points the same way. Enter its three entries top to bottom, as decimals or fractions.

unit vector

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Another way to measure size

The L2 length measures the straight line from the origin to the tip. The L1 length measures the walk if you may only move along the axes, like a taxi on a grid of streets:

∣v∣1=∣v1∣+∣v2∣+⋯+∣vn∣.|\mathbf{v}|_1 = |v_1| + |v_2| + \cdots + |v_n|.

A grid of city streets with a straight diagonal line from one corner to another and several staircase routes along the streets between the same two corners, all of the same lengthThe straight line is the L2 length. Every route along the streets without backtracking has the same, longer length: the L1 length.

For (3,−4)(3,\allowbreak -4) the L1 length is ∣3∣+∣−4∣=3+4=7|3| + |-4| = 3 + 4 = 7, while the L2 length is 9+16=5\sqrt{9 + 16} = 5. The walk along the grid is never shorter than the straight line, so ∣v∣1≥∣v∣|\mathbf{v}|_1 \ge |\mathbf{v}| always. To see why in the plane, square the L1 length: (∣v1∣+∣v2∣)2=v12+2 ∣v1∣ ∣v2∣+v22(|v_1| + |v_2|)^2 = v_1^2 + 2\,|v_1|\,|v_2| + v_2^2, which is the squared L2 length plus a term that is never negative. For (3,−4)(3,\allowbreak -4): 72=49=25+2(3)(4)7^2 = 49 = 25 + 2(3)(4). (Other sources write ∥v∥1\|\mathbf{v}\|_1.) In the plane, the vectors with L2 length 1 form the unit circle; those with L1 length 1 form a diamond with corners at (±1,0)(\pm 1,\allowbreak 0) and (0,±1)(0,\allowbreak \pm 1).

You will mostly meet the L1 length as a penalty during training. Adding λ∑i∣wi∣\lambda\sum_i |w_i| to a model's loss, where λ\lambda (the Greek letter lambda) is a small positive number that sets the strength, favors solutions in which many weights are exactly zero. Adding λ∑iwi2\lambda\sum_i w_i^2 instead (the squared L2 length, the usual choice in neural networks and closely related to what optimizers call weight decay) shrinks every weight but rarely makes any of them exactly zero.

The difference is in the pull, meaning the slope of the penalty, which is what gradient descent follows (module 5). The L1 penalty λ∣w∣\lambda|w| pulls a weight toward zero with the same strength λ\lambda however small the weight already is. The squared penalty λw2\lambda w^2 pulls with strength 2λ∣w∣2\lambda|w|, which fades to nothing as ww approaches zero. With λ=0.1\lambda = 0.1 and a weight of 0.010.01, the L1 pull is still 0.10.1, while the squared penalty pulls with 2(0.1)(0.01)=0.0022(0.1)(0.01) = 0.002, fifty times weaker. At the best weights the penalty's pull must balance the pull of the rest of the loss. Under L1, whenever the rest of the loss pulls a weight away from zero with strength less than λ\lambda, the penalty wins and the weight stays at exactly zero. Under the squared penalty the pull near zero is tiny, so any pull at all from the rest of the loss moves the weight to a small nonzero value.

To recap the section: the length ∣v∣=v⋅v|\mathbf{v}| = \sqrt{\mathbf{v}\cdot\mathbf{v}} is Pythagoras in any number of dimensions, scaling by cc multiplies it by ∣c∣|c|, and dividing by it gives a unit vector. The L1 length adds the sizes of the entries instead, and it is never smaller.

Distance or direction

With length in hand, we can compare two vectors in two different ways, and they do not always agree.

The distance between a\mathbf{a} and b\mathbf{b} is the length of their difference, ∣a−b∣|\mathbf{a} - \mathbf{b}|: the length of the arrow from the tip of b\mathbf{b} to the tip of a\mathbf{a} that you met in Arrows and lists.

Take a=(1,1)\mathbf{a} = (1,\allowbreak 1), b=(4,4)\mathbf{b} = (4,\allowbreak 4) and c=(1,−1)\mathbf{c} = (1,\allowbreak -1):

∣a−b∣=∣(−3,−3)∣=9+9=18≈4.243,∣a−c∣=∣(0,2)∣=0+4=2.|\mathbf{a} - \mathbf{b}| = |(-3, -3)| = \sqrt{9 + 9} = \sqrt{18} \approx 4.243, \qquad |\mathbf{a} - \mathbf{c}| = |(0, 2)| = \sqrt{0 + 4} = 2.

By distance, c\mathbf{c} is nearer to a\mathbf{a} than b\mathbf{b} is. Yet b=4a\mathbf{b} = 4\mathbf{a} points exactly the same way as a\mathbf{a}, while c\mathbf{c} is perpendicular to it. Distance mixes direction and length. Sometimes that is right: two apartments with similar area, bedrooms and age should count as close, and their sizes matter. Sometimes it is wrong: a query and a long document about the same thing should count as similar however long the document is.

For direction alone, use the angle. The dot product lesson showed that a⋅b=∣a∣ ∣b∣cos⁡θ\mathbf{a}\cdot\mathbf{b} = |\mathbf{a}|\,|\mathbf{b}|\cos\theta, where θ\theta is the angle between the arrows. Divide both sides by ∣a∣ ∣b∣|\mathbf{a}|\,|\mathbf{b}| (for nonzero vectors):

cos⁡θ=a⋅b∣a∣ ∣b∣.\cos\theta = \frac{\mathbf{a}\cdot\mathbf{b}}{|\mathbf{a}|\,|\mathbf{b}|}.

This number is the cosine similarity of a\mathbf{a} and b\mathbf{b}. Linearity of the dot product shows what it really is. Dividing by ∣a∣ ∣b∣|\mathbf{a}|\,|\mathbf{b}| is multiplying by 1∣a∣\frac{1}{|\mathbf{a}|} and by 1∣b∣\frac{1}{|\mathbf{b}|}. Linearity lets the first factor scale a\mathbf{a} and the second scale b\mathbf{b}, since (c a)⋅b=c (a⋅b)(c\,\mathbf{a})\cdot\mathbf{b} = c\,(\mathbf{a}\cdot\mathbf{b}) and the same holds for the second vector:

cos⁡θ=1∣a∣⋅1∣b∣ (a⋅b)=a∣a∣⋅b∣b∣=a^⋅b^.\cos\theta = \frac{1}{|\mathbf{a}|}\cdot\frac{1}{|\mathbf{b}|}\,(\mathbf{a}\cdot\mathbf{b}) = \frac{\mathbf{a}}{|\mathbf{a}|}\cdot\frac{\mathbf{b}}{|\mathbf{b}|} = \hat{\mathbf{a}}\cdot\hat{\mathbf{b}}.

Cosine similarity is the dot product of the normalized vectors. Normalize both, then dot. Stretching either vector changes nothing, because the stretch cancels in the division. The value is 11 when the vectors point the same way, 00 when they are perpendicular and −1-1 when they point in opposite directions. In the example, a⋅b=(1)(4)+(1)(4)=8\mathbf{a}\cdot\mathbf{b} = (1)(4) + (1)(4) = 8 with ∣a∣=2|\mathbf{a}| = \sqrt{2} and ∣b∣=32|\mathbf{b}| = \sqrt{32}, so

cos⁡(a,b)=82 32=864=88=1,cos⁡(a,c)=(1)(1)+(1)(−1)2 2=02=0.\cos(\mathbf{a}, \mathbf{b}) = \frac{8}{\sqrt{2}\,\sqrt{32}} = \frac{8}{\sqrt{64}} = \frac{8}{8} = 1, \qquad \cos(\mathbf{a}, \mathbf{c}) = \frac{(1)(1) + (1)(-1)}{\sqrt{2}\,\sqrt{2}} = \frac{0}{2} = 0.

The figure from the dot product lesson shows the difference between length and angle directly. It draws a\mathbf{a} (blue) and b\mathbf{b} (coral) from the origin, the lime shadow of a\mathbf{a} on the line of b\mathbf{b}, and the amber arc for the angle, with readouts for a⋅b\mathbf{a}\cdot\mathbf{b}, the angle, cos⁡θ\cos\theta and the shadow. The question here: when you stretch one arrow, what changes and what does not?

Back to the help center. With cosine similarity, the password note scores

q⋅a∣q∣ ∣a∣=42 8=416=44=1,\frac{\mathbf{q}\cdot\mathbf{a}}{|\mathbf{q}|\,|\mathbf{a}|} = \frac{4}{\sqrt{2}\,\sqrt{8}} = \frac{4}{\sqrt{16}} = \frac{4}{4} = 1,

a perfect match in direction. The billing FAQ has ∣f∣=36+1+81=118|\mathbf{f}| = \sqrt{36 + 1 + 81} = \sqrt{118}, so it scores 72 118=7236≈0.456\frac{7}{\sqrt{2}\,\sqrt{118}} = \frac{7}{\sqrt{236}} \approx 0.456. The ranking now follows the topic, and pasting the note twice changes nothing: (4,4,0)(4,\allowbreak 4,\allowbreak 0) points the same way as (2,2,0)(2,\allowbreak 2,\allowbreak 0), so its cosine with q\mathbf{q} is still 1.

Distance and angle are not unrelated. Expanding ∣a−b∣2|\mathbf{a} - \mathbf{b}|^2 with the dot product gives ∣a∣2+∣b∣2−2 a⋅b|\mathbf{a}|^2 + |\mathbf{b}|^2 - 2\,\mathbf{a}\cdot\mathbf{b}, as in the dot product lesson's derivation. When both vectors have length 1, the lengths drop out and only the angle is left.

Go slower: Distance between unit vectors

Let a^\hat{\mathbf{a}} and b^\hat{\mathbf{b}} have length 1, so a^⋅a^=1\hat{\mathbf{a}}\cdot\hat{\mathbf{a}} = 1 and b^⋅b^=1\hat{\mathbf{b}}\cdot\hat{\mathbf{b}} = 1, and let θ\theta be the angle between them, so a^⋅b^=cos⁡θ\hat{\mathbf{a}}\cdot\hat{\mathbf{b}} = \cos\theta.

Step 1. Squared length is the dot product with itself: ∣a^−b^∣2=(a^−b^)⋅(a^−b^).|\hat{\mathbf{a}} - \hat{\mathbf{b}}|^2 = (\hat{\mathbf{a}} - \hat{\mathbf{b}})\cdot(\hat{\mathbf{a}} - \hat{\mathbf{b}}). Step 2. Expand with linearity, as you would multiply out (x−y)(x−y)(x - y)(x - y): =a^⋅a^−a^⋅b^−b^⋅a^+b^⋅b^.= \hat{\mathbf{a}}\cdot\hat{\mathbf{a}} - \hat{\mathbf{a}}\cdot\hat{\mathbf{b}} - \hat{\mathbf{b}}\cdot\hat{\mathbf{a}} + \hat{\mathbf{b}}\cdot\hat{\mathbf{b}}. Step 3. Order does not matter, so b^⋅a^=a^⋅b^\hat{\mathbf{b}}\cdot\hat{\mathbf{a}} = \hat{\mathbf{a}}\cdot\hat{\mathbf{b}} and the middle terms combine: =a^⋅a^−2 a^⋅b^+b^⋅b^.= \hat{\mathbf{a}}\cdot\hat{\mathbf{a}} - 2\,\hat{\mathbf{a}}\cdot\hat{\mathbf{b}} + \hat{\mathbf{b}}\cdot\hat{\mathbf{b}}. Step 4. Substitute the three known values: ∣a^−b^∣2=1−2cos⁡θ+1=2−2cos⁡θ.|\hat{\mathbf{a}} - \hat{\mathbf{b}}|^2 = 1 - 2\cos\theta + 1 = 2 - 2\cos\theta.

The right side goes down exactly when cos⁡θ\cos\theta goes up. So among unit vectors, the nearest one by distance is always the most similar one by cosine, and sorting by either gives the same order. Two unit vectors 60°60° apart have cos⁡θ=0.5\cos\theta = 0.5, so ∣a^−b^∣2=2−1=1|\hat{\mathbf{a}} - \hat{\mathbf{b}}|^2 = 2 - 1 = 1 and their distance is 1=1\sqrt{1} = 1.

So the choice is not really between two unrelated measures. It is whether length should count. If it should not, normalize, and then distance and cosine agree.

To recap: distance ∣a−b∣|\mathbf{a} - \mathbf{b}| counts both length and direction. Cosine similarity a^⋅b^\hat{\mathbf{a}}\cdot\hat{\mathbf{b}} counts direction only. For unit vectors the two agree, because ∣a^−b^∣2=2−2cos⁡θ|\hat{\mathbf{a}} - \hat{\mathbf{b}}|^2 = 2 - 2\cos\theta.

The next exercise works every measure from this lesson so far on one pair of 3D vectors, and checks the distance two ways.

On paperLengths, cosine and distance by hand

Let a=(2,−1,2)\mathbf{a} = (2,\allowbreak -1,\allowbreak 2) and b=(1,2,2)\mathbf{b} = (1,\allowbreak 2,\allowbreak 2). By hand:

  1. The L2 length ∣a∣|\mathbf{a}| and the L1 length ∣a∣1|\mathbf{a}|_1.
  2. The unit vector a^\hat{\mathbf{a}}.
  3. The cosine similarity of a\mathbf{a} and b\mathbf{b}, as a fraction and as a decimal.
  4. The distance ∣a−b∣|\mathbf{a} - \mathbf{b}|, two ways: directly, and from ∣a−b∣2=∣a∣2+∣b∣2−2 a⋅b|\mathbf{a} - \mathbf{b}|^2 = |\mathbf{a}|^2 + |\mathbf{b}|^2 - 2\,\mathbf{a}\cdot\mathbf{b}.

Enter four numbers in the check box, top to bottom: ∣a∣|\mathbf{a}|, ∣a∣1|\mathbf{a}|_1, the cosine similarity and the distance, the last two to 3 decimal places.

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

|a|, L1 length of a, cosine, distance

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Now a cosine on its own, in four dimensions. Compute the three ingredients separately, then divide.

Work it outCosine to three decimals

Find the cosine similarity of a=(2,1,0,3)\mathbf{a} = (2,\allowbreak 1,\allowbreak 0,\allowbreak 3) and b=(1,1,1,1)\mathbf{b} = (1,\allowbreak 1,\allowbreak 1,\allowbreak 1). Enter it as a decimal to 3 decimal places.

cos θ

Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.

Distance and cosine can disagree when lengths differ a lot. The next question has one query and two documents, one of them three times as long as the query.

Quick checkNearest by distance, nearest by angle

A query vector is q=(2,1)\mathbf{q} = (2,\allowbreak 1). Document A has vector (6,3)(6,\allowbreak 3) and document B has vector (1,2)(1,\allowbreak 2). Which document does each method rank closer to the query?

Choose one answer, then check.

Why cosine stays between −1 and 1

In the plane, cos⁡θ\cos\theta is the cosine of an angle you can see, so it lies between −1-1 and 11 like every cosine. But for two 4,096-entry embeddings there is no angle to see. There, the formula cos⁡θ=a⋅b/(∣a∣ ∣b∣)\cos\theta = \mathbf{a}\cdot\mathbf{b}/(|\mathbf{a}|\,|\mathbf{b}|) is how we define the angle, and that only works if the ratio can never leave the range from −1-1 to 11. The guarantee is the Cauchy-Schwarz inequality:

∣a⋅b∣≤∣a∣ ∣b∣,|\mathbf{a}\cdot\mathbf{b}| \le |\mathbf{a}|\,|\mathbf{b}|,

with equality exactly when one vector is a multiple of the other. Divide both sides by ∣a∣ ∣b∣|\mathbf{a}|\,|\mathbf{b}| (for nonzero vectors) and it says ∣cos⁡θ∣≤1|\cos\theta| \le 1. For the pair (3,1)(3,\allowbreak 1) and (1,2)(1,\allowbreak 2): ∣a⋅b∣=5|\mathbf{a}\cdot\mathbf{b}| = 5 and ∣a∣ ∣b∣=10 5=50≈7.07|\mathbf{a}|\,|\mathbf{b}| = \sqrt{10}\,\sqrt{5} = \sqrt{50} \approx 7.07, and indeed 5≤7.075 \le 7.07.

The picture behind it is the shadow from the dot product lesson. The shadow of a\mathbf{a} on the line through b\mathbf{b} has length ∣a⋅b∣/∣b∣|\mathbf{a}\cdot\mathbf{b}|/|\mathbf{b}| (ignoring its sign), and a shadow is never longer than the arrow that casts it: the shadow and the drop form a right triangle with a\mathbf{a} as the hypotenuse. "Shadow at most ∣a∣|\mathbf{a}|", multiplied through by ∣b∣|\mathbf{b}|, is Cauchy-Schwarz.

Go slower: Proof of Cauchy-Schwarz from the projection

If b=0\mathbf{b} = \mathbf{0}, both sides are 00 and there is nothing to prove. Otherwise, split a\mathbf{a} into its projection on b\mathbf{b} and a leftover, exactly as in the dot product lesson: c=a⋅bb⋅b,p=c b,r=a−p.c = \frac{\mathbf{a}\cdot\mathbf{b}}{\mathbf{b}\cdot\mathbf{b}},\allowbreak \qquad \mathbf{p} = c\,\mathbf{b},\allowbreak \qquad \mathbf{r} = \mathbf{a} - \mathbf{p}.

Step 1, the leftover is perpendicular. Dot r=a−c b\mathbf{r} = \mathbf{a} - c\,\mathbf{b} with b\mathbf{b} and use linearity: r⋅b=a⋅b−c (b⋅b).\mathbf{r}\cdot\mathbf{b} = \mathbf{a}\cdot\mathbf{b} - c\,(\mathbf{b}\cdot\mathbf{b}). Substitute cc. The b⋅b\mathbf{b}\cdot\mathbf{b} on top cancels the one underneath: r⋅b=a⋅b−a⋅bb⋅b (b⋅b)=a⋅b−a⋅b=0.\mathbf{r}\cdot\mathbf{b} = \mathbf{a}\cdot\mathbf{b} - \frac{\mathbf{a}\cdot\mathbf{b}}{\mathbf{b}\cdot\mathbf{b}}\,(\mathbf{b}\cdot\mathbf{b}) = \mathbf{a}\cdot\mathbf{b} - \mathbf{a}\cdot\mathbf{b} = 0. Since p\mathbf{p} is a multiple of b\mathbf{b}, linearity gives r⋅p=c (r⋅b)=0\mathbf{r}\cdot\mathbf{p} = c\,(\mathbf{r}\cdot\mathbf{b}) = 0 too.

Step 2, Pythagoras. Since a=p+r\mathbf{a} = \mathbf{p} + \mathbf{r}, the squared length of a\mathbf{a} is ∣a∣2=(p+r)⋅(p+r).|\mathbf{a}|^2 = (\mathbf{p} + \mathbf{r})\cdot(\mathbf{p} + \mathbf{r}). Expand with linearity, as you would multiply out (x+y)(x+y)(x + y)(x + y): =p⋅p+p⋅r+r⋅p+r⋅r.= \mathbf{p}\cdot\mathbf{p} + \mathbf{p}\cdot\mathbf{r} + \mathbf{r}\cdot\mathbf{p} + \mathbf{r}\cdot\mathbf{r}. Order does not matter, so r⋅p=p⋅r\mathbf{r}\cdot\mathbf{p} = \mathbf{p}\cdot\mathbf{r} and the two middle terms combine: =p⋅p+2 p⋅r+r⋅r.= \mathbf{p}\cdot\mathbf{p} + 2\,\mathbf{p}\cdot\mathbf{r} + \mathbf{r}\cdot\mathbf{r}. The middle term is 00 by step 1, and each self dot product is a squared length: =∣p∣2+∣r∣2.= |\mathbf{p}|^2 + |\mathbf{r}|^2. A squared length is never negative, so dropping ∣r∣2|\mathbf{r}|^2 can only make the right side smaller: ∣a∣2≥∣p∣2.|\mathbf{a}|^2 \ge |\mathbf{p}|^2.

Step 3, the length of the shadow. By the scaling rule, ∣c b∣2=c2 ∣b∣2=c2 (b⋅b)|c\,\mathbf{b}|^2 = c^2\,|\mathbf{b}|^2 = c^2\,(\mathbf{b}\cdot\mathbf{b}). Substitute cc: ∣p∣2=(a⋅b)2(b⋅b)2 (b⋅b).|\mathbf{p}|^2 = \frac{(\mathbf{a}\cdot\mathbf{b})^2}{(\mathbf{b}\cdot\mathbf{b})^2}\,(\mathbf{b}\cdot\mathbf{b}). One factor of b⋅b\mathbf{b}\cdot\mathbf{b} cancels, and b⋅b=∣b∣2\mathbf{b}\cdot\mathbf{b} = |\mathbf{b}|^2: ∣p∣2=(a⋅b)2b⋅b=(a⋅b)2∣b∣2.|\mathbf{p}|^2 = \frac{(\mathbf{a}\cdot\mathbf{b})^2}{\mathbf{b}\cdot\mathbf{b}} = \frac{(\mathbf{a}\cdot\mathbf{b})^2}{|\mathbf{b}|^2}.

Step 4, combine. Steps 2 and 3 give (a⋅b)2∣b∣2≤∣a∣2\frac{(\mathbf{a}\cdot\mathbf{b})^2}{|\mathbf{b}|^2} \le |\mathbf{a}|^2. Multiply both sides by ∣b∣2|\mathbf{b}|^2, which is positive: (a⋅b)2≤∣a∣2 ∣b∣2(\mathbf{a}\cdot\mathbf{b})^2 \le |\mathbf{a}|^2\,|\mathbf{b}|^2. Both sides are nonnegative, so taking square roots keeps the inequality: ∣a⋅b∣≤∣a∣ ∣b∣|\mathbf{a}\cdot\mathbf{b}| \le |\mathbf{a}|\,|\mathbf{b}|.

When is it an equality? Steps 3 and 4 only rewrite, multiply by a positive number and take square roots, which keep an equality an equality. So equality holds exactly when the one thing dropped in step 2, ∣r∣2|\mathbf{r}|^2, is zero. That means r=0\mathbf{r} = \mathbf{0}, so a=c b\mathbf{a} = c\,\mathbf{b}. Conversely, if a=k b\mathbf{a} = k\,\mathbf{b} for some number kk, then a⋅b=k (b⋅b)\mathbf{a}\cdot\mathbf{b} = k\,(\mathbf{b}\cdot\mathbf{b}), so c=kc = k, p=a\mathbf{p} = \mathbf{a} and r=0\mathbf{r} = \mathbf{0}. (The case b=0\mathbf{b} = \mathbf{0} set aside at the start is also an equality, and 0=0 a\mathbf{0} = 0\,\mathbf{a} is a multiple of a\mathbf{a}.) So equality means one vector is a multiple of the other: the two lie on one line, pointing the same way (cos⁡θ=1\cos\theta = 1) or opposite ways (cos⁡θ=−1\cos\theta = -1) when both are nonzero.

Nothing here depended on the number of entries. The proof is as valid for 4,096 dimensions as for 2.

A second classic inequality falls out of Cauchy-Schwarz in three lines: the triangle inequality, ∣a+b∣≤∣a∣+∣b∣|\mathbf{a} + \mathbf{b}| \le |\mathbf{a}| + |\mathbf{b}|. A detour is never shorter than the direct route. The next exercise proves it; the hint is to compare the squares of the two sides.

On paperThe triangle inequality

The triangle inequality says a detour is never shorter than the direct route: ∣a+b∣≤∣a∣+∣b∣|\mathbf{a} + \mathbf{b}| \le |\mathbf{a}| + |\mathbf{b}|.

  1. Prove it from Cauchy-Schwarz. Both sides are nonnegative, so it is enough to compare their squares.
  2. Check it on a=(3,4)\mathbf{a} = (3,\allowbreak 4) and b=(−6,8)\mathbf{b} = (-6,\allowbreak 8) by computing both sides.
  3. When is it an equality? Compare with the equality case of Cauchy-Schwarz.

Enter both sides from part 2 in the check box, ∣a+b∣|\mathbf{a} + \mathbf{b}| on top, to 3 decimal places.

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

|a + b| and |a| + |b| for part 2

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Searching a small embedding table

Real embeddings are learned by a model and have hundreds or thousands of entries whose individual meanings are unclear. To see the ideas without that fog, here is a table we invent by hand: six words, each described by four features we chose (royalty, gender with male positive and female negative, adulthood, and whether it is food). The cell normalizes every row, computes every cosine at once, prints each word's two nearest neighbors, and tries the analogy king−man+woman\text{king} - \text{man} + \text{woman}. Run it, then look at the heatmap and the printed neighbors.

⌘+Enter runs · edit freelyPython sleeps until you run code

A few things to notice in the output.

  • Neighbors follow the features we chose. The nearest word to king is prince (0.8540.854), then man (0.8420.842); queen comes only third (0.4100.410). In this table gender is the feature that varies most, so it dominates the angles. A learned embedding has no designer: which words end up close depends on what the training made useful.
  • Cosine 0 means nothing in common, not opposite. Apple shares no feature with any person, so its arrow is perpendicular to all of theirs and every cosine with it is 00. A negative cosine needs features that disagree in sign, like the gender feature: woman and prince have cosine −0.302-0.302.
  • The analogy works here by construction. king−man+woman=(0.9−0.1+0.1, 0.8−0.9−0.8, 0.9−0.8+0.9, 0)=(0.9,−0.9,1.0,0.0)\text{king} - \text{man} + \text{woman} = (0.9 - 0.1 + 0.1,\allowbreak \ 0.8 - 0.9 - 0.8,\allowbreak \ 0.9 - 0.8 + 0.9,\allowbreak \ 0) = (0.9,\allowbreak -0.9,\allowbreak 1.0,\allowbreak 0.0): subtracting man removes the male part of king and adding woman puts a female part in its place. The nearest word is queen (0.9960.996). Notice that woman, one of the inputs, comes a strong second (0.8740.874).

That last point is the honest caveat about analogies. In real learned embeddings, king−man+woman\text{king} - \text{man} + \text{woman} does land near queen, but the nearest word to it is often one of the three input words, so the standard analogy benchmarks leave the input words out of the candidates. Even then, many analogies miss. Vector arithmetic on embeddings reflects a real, useful regularity, but an approximate one.

Now build the search yourself: length, normalization, cosine, top-kk neighbors, and the analogy solver. Each function builds on the one before, so write them in order.

Code itBuild an embedding search

Build the pieces of embedding search, from length up to analogies. A table is a dict that maps each word to a 1D numpy array.

  1. norm(v) from the definition, and normalize(v), which returns a new unit vector and raises ValueError for the zero vector.
  2. cosine(a, b), the cosine similarity.
  3. nearest(query, table, k=1, exclude=()): the k words whose vectors have the highest cosine with query, best first, never returning a word in exclude.
  4. analogy(a, b, c, table): the answer to "a is to b as c is to ?", the word nearest to table[b] - table[a] + table[c] other than a, b and c.
⌘+Enter runsPython sleeps until you run code
Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.

The practice set below mixes lengths (some exact, some to 3 decimals), distances, unit vectors, cosines and L1 lengths, as many as you want.

Practice setLength and cosine reps
3 correct in a row completes the set. A miss starts the count again.

Normalize v=(−6,−2,3)\mathbf{v} = (-6,\allowbreak -2,\allowbreak 3): find the unit vector v^=v/∣v∣\hat{\mathbf{v}} = \mathbf{v}/|\mathbf{v}| that points the same way. Give each entry to 3 decimal places (fractions are accepted).

v̂

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Why do embedding searches normalize? Three reasons, one of them a warning.

  1. Meaning lives in direction. Many embedding models are trained so that related texts have high cosine similarity, which means direction is what training shaped. Length can vary for reasons unrelated to the topic, as document length did with our word counts.
  2. Normalizing once makes every comparison cheap. Store unit vectors, and cosine similarity with a query is a single dot product, with no lengths to recompute. And since ∣a^−b^∣2=2−2cos⁡θ|\hat{\mathbf{a}} - \hat{\mathbf{b}}|^2 = 2 - 2\cos\theta, an index that ranks by distance returns the same results.
  3. Normalizing throws information away. If a model was trained to compare vectors with raw dot products, their lengths may carry something it learned on purpose, and normalizing discards it. Check how the embeddings you use were trained.

Those three reasons are the whole argument. The last exercise asks you to make it in your own words, to a colleague who ranks by raw dot products.

In your own wordsWhy normalize before comparing

A colleague ranks documents by the raw dot product between a query embedding and each document embedding. Explain why they should normalize first (or use cosine similarity), what can go wrong without it, and one situation in which normalizing would throw away something useful.

A few sentences first: 0 of 60 characters.

Saved in this browser as you type.

Next: the module checkpoint