ml.lab
Python sleeps until you run code
01 Vectors and the dot product

Lesson 3 of 4

The dot product

One small operation, multiply matching entries and add them up, measures how much two vectors point the same way. It is the most used calculation in machine learning.

About 65 minutes
By the end you can
  • Compute a dot product by hand for vectors of any length.

  • Explain what the sign and size of a dot product say about two arrows.

  • Derive why a⋅b=∣a∣ ∣b∣cos⁡θ\mathbf{a}\cdot\mathbf{b} = |\mathbf{a}|\,|\mathbf{b}|\cos\theta from the law of cosines.

  • Compute the shadow (projection) of one vector on another, by hand and in code.

Here is a small recommender. You describe your taste in movies with three numbers: how much you like action, romance and comedy. Say your taste is t=(2,−1,3)\mathbf{t} = (2,\allowbreak -1,\allowbreak 3): you like action, dislike romance, and love comedy. A movie gets the same three numbers for how much of each it contains, say m=(1,2,1)\mathbf{m} = (1,\allowbreak 2,\allowbreak 1).

How good a match is this movie? A sensible score: for each feature, multiply how much you care by how much the movie has, then add it all up.

score=(2)(1)+(−1)(2)+(3)(1)=2−2+3=3\text{score} = (2)(1) + (-1)(2) + (3)(1) = 2 - 2 + 3 = 3

The romance term counts against the movie because you dislike romance. The comedy term helps the most because you care about comedy most. A pure romance, (0,3,0)(0,\allowbreak 3,\allowbreak 0), would score (2)(0)+(−1)(3)+(3)(0)=−3(2)(0) + (-1)(3) + (3)(0) = -3: a negative score for a movie that has only what you dislike.

That multiply-then-add score is the dot product. This lesson computes it, then reads its sign and size as geometry (an angle, and a shadow one arrow casts on another), and ends with where it appears in machine learning: inside every neuron, inside attention, and inside every embedding search.

Multiply matching entries, then add

The movie score is one example of a recipe that works for any two lists of the same length.

For two vectors a\mathbf{a} and b\mathbf{b} with nn entries each, the dot product is

a⋅b=a1b1+a2b2+⋯+anbn=∑i=1naibi.\mathbf{a}\cdot\mathbf{b} = a_1 b_1 + a_2 b_2 + \cdots + a_n b_n = \sum_{i=1}^{n} a_i b_i .

The symbol ∑\sum (a capital Greek sigma) means "add up". The whole expression ∑i=1naibi\sum_{i=1}^{n} a_i b_i reads: for each position i=1,2,…,ni = 1,\allowbreak 2,\allowbreak \ldots,\allowbreak n, form the product aibia_i b_i, then add all nn products. For example,

(1,2,3)⋅(4,5,6)=(1)(4)+(2)(5)+(3)(6)=4+10+18=32.(1, 2, 3)\cdot(4, 5, 6) = (1)(4) + (2)(5) + (3)(6) = 4 + 10 + 18 = 32.

Three things to notice:

  • The answer is one number, not a vector. Two lists go in, a single score comes out.
  • Both vectors must have the same length. There is no way to pair up entries otherwise. In code, a mismatch here is one of the most common errors you will see.
  • Order does not matter: a⋅b=b⋅a\mathbf{a}\cdot\mathbf{b} = \mathbf{b}\cdot\mathbf{a}, because each product aibia_i b_i equals biaib_i a_i. For example, (4,5,6)⋅(1,2,3)=4+10+18=32(4,\allowbreak 5,\allowbreak 6)\cdot(1,\allowbreak 2,\allowbreak 3) = 4 + 10 + 18 = 32 again.

Linear in each vector

A fourth property matters more than it looks. The dot product is linear in each vector: scaling a vector scales the score, and splitting a vector into two parts splits the score into two parts.

(c a)⋅b=c (a⋅b),(a+a′)⋅b=a⋅b+a′⋅b.(c\,\mathbf{a})\cdot\mathbf{b} = c\,(\mathbf{a}\cdot\mathbf{b}), \qquad (\mathbf{a} + \mathbf{a}')\cdot\mathbf{b} = \mathbf{a}\cdot\mathbf{b} + \mathbf{a}'\cdot\mathbf{b}.

Check both with the movie. Double your enthusiasm for every genre, 2t=(4,−2,6)2\mathbf{t} = (4,\allowbreak -2,\allowbreak 6), and the score doubles: (4)(1)+(−2)(2)+(6)(1)=4−4+6=6=2×3(4)(1) + (-2)(2) + (6)(1) = 4 - 4 + 6 = 6 = 2 \times 3. Now split the movie into its action-and-comedy part and its romance part, m=(1,0,1)+(0,2,0)\mathbf{m} = (1,\allowbreak 0,\allowbreak 1) + (0,\allowbreak 2,\allowbreak 0). The two parts score t⋅(1,0,1)=2+0+3=5\mathbf{t}\cdot(1,\allowbreak 0,\allowbreak 1) = 2 + 0 + 3 = 5 and t⋅(0,2,0)=0−2+0=−2\mathbf{t}\cdot(0,\allowbreak 2,\allowbreak 0) = 0 - 2 + 0 = -2, and 5+(−2)=35 + (-2) = 3, the whole movie's score.

Why it always works: every entry of c ac\,\mathbf{a} is c aic\,a_i, so every product becomes (c ai) bi=c (aibi)(c\,a_i)\,b_i = c\,(a_i b_i), and pulling the common factor cc out of the sum gives c (a⋅b)c\,(\mathbf{a}\cdot\mathbf{b}). For splitting, every entry of a+a′\mathbf{a} + \mathbf{a}' is ai+ai′a_i + a'_i, so every product becomes (ai+ai′) bi=aibi+ai′bi(a_i + a'_i)\,b_i = a_i b_i + a'_i b_i. Sorting those terms into two sums gives a⋅b+a′⋅b\mathbf{a}\cdot\mathbf{b} + \mathbf{a}'\cdot\mathbf{b}. Since order does not matter, the same two rules hold for the second vector.

You will lean on linearity constantly. In module 2 a matrix times a vector turns out to be many dot products at once, and linearity is what makes it behave predictably.

Three dot products by hand come next, one of them in four dimensions. Write every product before adding: most slips are a sign error in a single product.

On paperThree dot products

Compute each dot product by hand. Write every product before adding.

  1. (1,4)⋅(3,−2)(1,\allowbreak 4)\cdot(3,\allowbreak -2)
  2. (2,0,−1)⋅(5,7,3)(2,\allowbreak 0,\allowbreak -1)\cdot(5,\allowbreak 7,\allowbreak 3)
  3. (1,1,1,1)⋅(2,−3,4,1)(1,\allowbreak 1,\allowbreak 1,\allowbreak 1)\cdot(2,\allowbreak -3,\allowbreak 4,\allowbreak 1)

Enter your three answers in the check box, top to bottom.

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

Your three answers

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

What the sign says: the angle between two arrows

So far the dot product is pure arithmetic. But a vector with two entries is also an arrow, and the dot product turns out to measure something you can see: how much two arrows point the same way.

The angle θ\theta (the Greek letter theta) between two arrows drawn from the origin is how far you would turn one to line it up with the other, measured the short way round. So it is always between 0°0° (same direction) and 180°180° (opposite directions).

The figure answers one question: how does a⋅b\mathbf{a}\cdot\mathbf{b} change as the arrows turn? It shows a\mathbf{a} (blue) and b\mathbf{b} (coral) from the origin, and the faint line through b\mathbf{b}. On that line a thick lime band, the shadow of a\mathbf{a}, runs from the origin to a dot; a dashed drop from the tip of a\mathbf{a} meets the line there at a right angle, marked by a small square. An amber arc marks θ\theta. The readouts give a⋅b\mathbf{a}\cdot\mathbf{b}, the angle, cos⁡θ\cos\theta and the shadow's signed length. Under the sentence, the formulas compute a⋅b\mathbf{a}\cdot\mathbf{b} twice: entry by entry, and as ∣a∣ ∣b∣cos⁡θ|\mathbf{a}|\,|\mathbf{b}|\cos\theta, which this section explains. The tips snap to half units.

The figure shows three facts:

  1. When the arrows are less than 90° apart, the dot product is positive.
  2. When they are perpendicular, it is exactly zero.
  3. When they are more than 90° apart, it is negative.

And its size grows with both lengths. One formula captures all of this:

a⋅b=∣a∣ ∣b∣cos⁡θ.\mathbf{a}\cdot\mathbf{b} = |\mathbf{a}|\,|\mathbf{b}|\cos\theta.

Here ∣a∣=a12+a22+⋯+an2|\mathbf{a}| = \sqrt{a_1^2 + a_2^2 + \cdots + a_n^2} is the length of a\mathbf{a}, from Pythagoras (the entries are the legs of a right triangle and the arrow is its longest side; the lesson on length works this out in full). And cos⁡θ\cos\theta is the cosine of the angle: 11 at 0°0°, about 0.7070.707 at 45°45°, 00 at 90°90°, about −0.707-0.707 at 135°135° and −1-1 at 180°180°. Between 0°0° and 180°180° the cosine falls steadily, and it is positive below 90°90° and negative above, which is exactly the sign pattern in the three facts. Lengths are never negative, so they cannot change the sign.

Check the formula on the figure's starting pair, a=(3,1)\mathbf{a} = (3,\allowbreak 1) and b=(1,2)\mathbf{b} = (1,\allowbreak 2), where the angle is 45°45° and cos⁡45°=22\cos 45° = \frac{\sqrt{2}}{2}:

∣a∣=32+12=10,∣b∣=12+22=5,|\mathbf{a}| = \sqrt{3^2 + 1^2} = \sqrt{10}, \qquad |\mathbf{b}| = \sqrt{1^2 + 2^2} = \sqrt{5}, ∣a∣ ∣b∣cos⁡θ=10 5⋅22=10⋅5⋅22=1002=102=5,|\mathbf{a}|\,|\mathbf{b}|\cos\theta = \sqrt{10}\,\sqrt{5}\cdot\frac{\sqrt{2}}{2} = \frac{\sqrt{10 \cdot 5 \cdot 2}}{2} = \frac{\sqrt{100}}{2} = \frac{10}{2} = 5,

the same as (3)(1)+(1)(2)=5(3)(1) + (1)(2) = 5 entry by entry. (The second step uses the rule that a product of square roots is the square root of the product, x y=xy\sqrt{x}\,\sqrt{y} = \sqrt{xy} for numbers that are not negative.)

Nothing in "multiply matching entries and add" mentions an angle, so why should the two formulas agree? The reason is a fact about triangles from school geometry, the law of cosines: in a triangle where two sides of lengths pp and qq meet at an angle θ\theta, the third side rr satisfies

r2=p2+q2−2pqcos⁡θ.r^2 = p^2 + q^2 - 2pq\cos\theta.

At θ=90°\theta = 90° the cosine is 00 and this is Pythagoras, r2=p2+q2r^2 = p^2 + q^2. The last term corrects for an angle that is not a right angle. The box below applies it to the triangle made by a\mathbf{a} and b\mathbf{b}.

Go slower: Why the two formulas agree

Draw a\mathbf{a} and b\mathbf{b} from the origin. The third side of that triangle runs from the tip of b\mathbf{b} to the tip of a\mathbf{a}, and in Arrows and lists you saw that this arrow is a−b\mathbf{a} - \mathbf{b}.

Step 1, with geometry. The two sides from the origin have lengths ∣a∣|\mathbf{a}| and ∣b∣|\mathbf{b}| and meet at the angle θ\theta, so the law of cosines says ∣a−b∣2=∣a∣2+∣b∣2−2 ∣a∣ ∣b∣cos⁡θ.|\mathbf{a} - \mathbf{b}|^2 = |\mathbf{a}|^2 + |\mathbf{b}|^2 - 2\,|\mathbf{a}|\,|\mathbf{b}|\cos\theta.

Step 2, with algebra. First a fact: the squared length of any vector is its dot product with itself, because v⋅v=v1v1+v2v2+⋯=v12+v22+⋯=∣v∣2\mathbf{v}\cdot\mathbf{v} = v_1 v_1 + v_2 v_2 + \cdots = v_1^2 + v_2^2 + \cdots = |\mathbf{v}|^2. So: ∣a−b∣2=(a−b)⋅(a−b).|\mathbf{a} - \mathbf{b}|^2 = (\mathbf{a} - \mathbf{b})\cdot(\mathbf{a} - \mathbf{b}). Split the first vector, using linearity: =a⋅(a−b)−b⋅(a−b).= \mathbf{a}\cdot(\mathbf{a} - \mathbf{b}) - \mathbf{b}\cdot(\mathbf{a} - \mathbf{b}). Split the second vector in each term, the same way: =a⋅a−a⋅b−b⋅a+b⋅b.= \mathbf{a}\cdot\mathbf{a} - \mathbf{a}\cdot\mathbf{b} - \mathbf{b}\cdot\mathbf{a} + \mathbf{b}\cdot\mathbf{b}. Order does not matter, so b⋅a=a⋅b\mathbf{b}\cdot\mathbf{a} = \mathbf{a}\cdot\mathbf{b} and the two middle terms combine: =a⋅a−2 a⋅b+b⋅b.= \mathbf{a}\cdot\mathbf{a} - 2\,\mathbf{a}\cdot\mathbf{b} + \mathbf{b}\cdot\mathbf{b}. Replace each self dot product with a squared length: =∣a∣2+∣b∣2−2 a⋅b.= |\mathbf{a}|^2 + |\mathbf{b}|^2 - 2\,\mathbf{a}\cdot\mathbf{b}.

Step 3, compare. Steps 1 and 2 both give ∣a−b∣2|\mathbf{a} - \mathbf{b}|^2, so their right sides are equal: ∣a∣2+∣b∣2−2 ∣a∣ ∣b∣cos⁡θ=∣a∣2+∣b∣2−2 a⋅b.|\mathbf{a}|^2 + |\mathbf{b}|^2 - 2\,|\mathbf{a}|\,|\mathbf{b}|\cos\theta = |\mathbf{a}|^2 + |\mathbf{b}|^2 - 2\,\mathbf{a}\cdot\mathbf{b}. Subtract ∣a∣2+∣b∣2|\mathbf{a}|^2 + |\mathbf{b}|^2 from both sides: −2 ∣a∣ ∣b∣cos⁡θ=−2 a⋅b.-2\,|\mathbf{a}|\,|\mathbf{b}|\cos\theta = -2\,\mathbf{a}\cdot\mathbf{b}. Divide both sides by −2-2: ∣a∣ ∣b∣cos⁡θ=a⋅b.|\mathbf{a}|\,|\mathbf{b}|\cos\theta = \mathbf{a}\cdot\mathbf{b}.

A check with numbers. For a=(3,1)\mathbf{a} = (3,\allowbreak 1) and b=(1,2)\mathbf{b} = (1,\allowbreak 2), the third side is a−b=(2,−1)\mathbf{a} - \mathbf{b} = (2,\allowbreak -1), so ∣a−b∣2=4+1=5|\mathbf{a} - \mathbf{b}|^2 = 4 + 1 = 5. Step 1 gives 10+5−2 10 5⋅2210 + 5 - 2\,\sqrt{10}\,\sqrt{5}\cdot\frac{\sqrt{2}}{2}. The 22 in front cancels the 22 underneath, leaving 15−10 5 2=15−100=15−10=515 - \sqrt{10}\,\sqrt{5}\,\sqrt{2} = 15 - \sqrt{100} = 15 - 10 = 5. Step 2 gives 10+5−2(5)=510 + 5 - 2(5) = 5.

Any number of dimensions. The three points (the origin and the two tips) always lie in one flat plane, even when the vectors have 4,096 entries, so the triangle is an ordinary triangle and the law of cosines applies to it. Nothing else in the argument used two dimensions. That is how we can talk about "the angle" between two word embeddings that live in spaces we cannot picture. (The lesson on length shows why the formula always gives a cosine between −1-1 and 11, so the angle always exists.)

Because lengths are never negative, the sign of a⋅b\mathbf{a}\cdot\mathbf{b} is the sign of cos⁡θ\cos\theta. That is enough to answer the next question without drawing anything.

Quick checkRead the angle from the sign

Let u=(4,1)\mathbf{u} = (4,\allowbreak 1) and v=(−1,5)\mathbf{v} = (-1,\allowbreak 5). Without drawing them, what can you say about the angle between them?

Choose one answer, then check.

To recap: a⋅b=∣a∣ ∣b∣cos⁡θ\mathbf{a}\cdot\mathbf{b} = |\mathbf{a}|\,|\mathbf{b}|\cos\theta. The sign says which side of 90°90° the angle is on, a zero means the arrows are perpendicular, and the size grows with both lengths.

The shadow: projection

The lime band in the figure has a precise meaning, and it is one of the most useful things a dot product gives you: how much of one vector lies along the direction of another. It is how a vector gets split into a part along a direction and a part across it, the same split that least-squares regression is built on. In this course it comes back in module 4, where principal component analysis projects data onto the directions along which it varies most.

A blue arrow a and a faint coral line through b, with light falling at a right angle onto the line; the shadow of a is a lime segment on the line from the origin to the foot of a dashed drop from the tip of aLight falls at a right angle onto the line through b. The shadow of a runs from the origin to the point directly under the tip of a.

Shine a light onto the line through b\mathbf{b} so that it falls at a right angle to that line. The shadow of a\mathbf{a} is the piece of the line from the origin to the point directly under the tip of a\mathbf{a}: the foot of the drop. How long is it? One step at a time:

  1. The arrow a\mathbf{a}, its shadow and the drop form a right triangle. The right angle is where the drop meets the line, so a\mathbf{a} is the longest side (the hypotenuse), with length ∣a∣|\mathbf{a}|.
  2. In a right triangle, the cosine of an angle is the side next to it divided by the hypotenuse. The side next to θ\theta is the shadow, so the shadow's length is ∣a∣cos⁡θ|\mathbf{a}|\cos\theta.
  3. Divide both sides of a⋅b=∣a∣ ∣b∣cos⁡θ\mathbf{a}\cdot\mathbf{b} = |\mathbf{a}|\,|\mathbf{b}|\cos\theta by ∣a∣ ∣b∣|\mathbf{a}|\,|\mathbf{b}| to get cos⁡θ=a⋅b∣a∣ ∣b∣\cos\theta = \frac{\mathbf{a}\cdot\mathbf{b}}{|\mathbf{a}|\,|\mathbf{b}|}, and put it into step 2: the shadow's length is ∣a∣⋅a⋅b∣a∣ ∣b∣|\mathbf{a}|\cdot\frac{\mathbf{a}\cdot\mathbf{b}}{|\mathbf{a}|\,|\mathbf{b}|}.
  4. The ∣a∣|\mathbf{a}| in front cancels the ∣a∣|\mathbf{a}| underneath, leaving a⋅b∣b∣\frac{\mathbf{a}\cdot\mathbf{b}}{|\mathbf{b}|}.

When θ\theta is more than 90°90°, the shadow falls on the far side of the origin, pointing away from b\mathbf{b}, and the figure colors it coral. The right triangle is still there, but its angle at the origin is now 180°−θ180° - \theta, so the shadow's size is ∣a∣cos⁡(180°−θ)|\mathbf{a}|\cos(180° - \theta). Two angles that add up to 180°180° have opposite cosines (you saw 0.7070.707 at 45°45° and −0.707-0.707 at 135°135°), so cos⁡(180°−θ)=−cos⁡θ\cos(180° - \theta) = -\cos\theta and the size is −∣a∣cos⁡θ-|\mathbf{a}|\cos\theta, a positive number because cos⁡θ\cos\theta is negative. The single expression ∣a∣cos⁡θ|\mathbf{a}|\cos\theta carries both facts: its size is the shadow's length, and its sign (negative here) says the shadow points away from b\mathbf{b}. That is why it is called a signed length:

shadow length=∣a∣cos⁡θ=a⋅b∣b∣.\text{shadow length} = |\mathbf{a}|\cos\theta = \frac{\mathbf{a}\cdot\mathbf{b}}{|\mathbf{b}|}.

In the figure's starting picture, a⋅b=5\mathbf{a}\cdot\mathbf{b} = 5 and ∣b∣=5|\mathbf{b}| = \sqrt{5}, so the shadow is 55\frac{5}{\sqrt{5}} long. Since 5=5 55 = \sqrt{5}\,\sqrt{5}, that is 5 55=5≈2.24\frac{\sqrt{5}\,\sqrt{5}}{\sqrt{5}} = \sqrt{5} \approx 2.24, exactly the length of b\mathbf{b}. That is why the figure said the shadow ends at the tip of b\mathbf{b}.

Multiply both sides by ∣b∣|\mathbf{b}| and you get another way to say what the dot product is: the length of a\mathbf{a}'s shadow on b\mathbf{b}, times the length of b\mathbf{b}. If b\mathbf{b} has length 1, the dot product is the shadow length. That is why we so often shrink or stretch vectors to length 1 before comparing them, the subject of the next lesson.

To get the shadow as a vector, not only its length, walk that far along the direction of b\mathbf{b}:

  1. The vector b∣b∣\frac{\mathbf{b}}{|\mathbf{b}|} points the same way as b\mathbf{b} and has length 1, because multiplying a vector by a positive number multiplies its length by that number, and here the number is 1∣b∣\frac{1}{|\mathbf{b}|}.
  2. Walk the shadow's length along it: a⋅b∣b∣⋅b∣b∣\frac{\mathbf{a}\cdot\mathbf{b}}{|\mathbf{b}|}\cdot\frac{\mathbf{b}}{|\mathbf{b}|}.
  3. Multiply the two numbers in front: a⋅b∣b∣ ∣b∣ b=a⋅b∣b∣2 b\frac{\mathbf{a}\cdot\mathbf{b}}{|\mathbf{b}|\,|\mathbf{b}|}\,\mathbf{b} = \frac{\mathbf{a}\cdot\mathbf{b}}{|\mathbf{b}|^2}\,\mathbf{b}.
  4. A squared length is a self dot product, ∣b∣2=b⋅b|\mathbf{b}|^2 = \mathbf{b}\cdot\mathbf{b}.

The result is the projection of a\mathbf{a} onto b\mathbf{b}:

proj⁡ba=a⋅bb⋅b b.\operatorname{proj}_{\mathbf{b}}\mathbf{a} = \frac{\mathbf{a}\cdot\mathbf{b}}{\mathbf{b}\cdot\mathbf{b}}\,\mathbf{b}.

This form needs no square roots, which is how you should compute it by hand: two dot products, one division, then scale b\mathbf{b}. Two examples:

  • The figure's pair: a⋅b=5\mathbf{a}\cdot\mathbf{b} = 5 and b⋅b=(1)(1)+(2)(2)=5\mathbf{b}\cdot\mathbf{b} = (1)(1) + (2)(2) = 5, so the projection is 55 (1,2)=(1,2)\frac{5}{5}\,(1,\allowbreak 2) = (1,\allowbreak 2), which is b\mathbf{b} itself. What is left over, a−(1,2)=(3−1, 1−2)=(2,−1)\mathbf{a} - (1,\allowbreak 2) = (3 - 1,\allowbreak \ 1 - 2) = (2,\allowbreak -1), is the dashed drop in the figure, read from its foot up to the tip of a\mathbf{a}, and it is perpendicular to b\mathbf{b}: (2)(1)+(−1)(2)=0(2)(1) + (-1)(2) = 0.
  • a=(3,4)\mathbf{a} = (3,\allowbreak 4) and b=(2,0)\mathbf{b} = (2,\allowbreak 0), which lies along the horizontal axis: a⋅b=(3)(2)+(4)(0)=6\mathbf{a}\cdot\mathbf{b} = (3)(2) + (4)(0) = 6 and b⋅b=(2)(2)+(0)(0)=4\mathbf{b}\cdot\mathbf{b} = (2)(2) + (0)(0) = 4, so the projection is 64 (2,0)=1.5 (2,0)=(3,0)\frac{6}{4}\,(2,\allowbreak 0) = 1.5\,(2,\allowbreak 0) = (3,\allowbreak 0). The shadow on the horizontal axis keeps the first entry of a\mathbf{a} and drops the second. What is left over, a−(3,0)=(0,4)\mathbf{a} - (3,\allowbreak 0) = (0,\allowbreak 4), is perpendicular to b\mathbf{b}: (0)(2)+(4)(0)=0(0)(2) + (4)(0) = 0.

Use the square-root-free form on the next one: two dot products, one division, then scale b\mathbf{b}.

Work it outProject a onto b

Find the projection of a=(4,2)\mathbf{a} = (4,\allowbreak 2) onto the line through b=(1,1)\mathbf{b} = (1,\allowbreak 1), that is proj⁡ba\operatorname{proj}_{\mathbf{b}}\mathbf{a}. Enter its two entries top to bottom.

proj

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

In both examples above the leftover, a\mathbf{a} minus its projection, was perpendicular to b\mathbf{b}. The next exercise proves that it always is, using nothing but linearity.

On paperWhy the leftover is perpendicular

Projection splits a\mathbf{a} into two pieces: the part along b\mathbf{b}, and the leftover r=a−proj⁡ba\mathbf{r} = \mathbf{a} - \operatorname{proj}_{\mathbf{b}}\mathbf{a}.

  1. Prove that r⋅b=0\mathbf{r}\cdot\mathbf{b} = 0 for any a\mathbf{a} and any nonzero b\mathbf{b}. Use only linearity.
  2. Check it on a=(1,3)\mathbf{a} = (1,\allowbreak 3), b=(2,1)\mathbf{b} = (2,\allowbreak 1) by computing r\mathbf{r}.

Enter r\mathbf{r} from part 2 in the check box, top to bottom.

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

r for part 2

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

To recap: the shadow of a\mathbf{a} on the line of b\mathbf{b} has signed length a⋅b/∣b∣\mathbf{a}\cdot\mathbf{b}/|\mathbf{b}|, the projection vector is a⋅bb⋅b b\frac{\mathbf{a}\cdot\mathbf{b}}{\mathbf{b}\cdot\mathbf{b}}\,\mathbf{b}, and the leftover is perpendicular to b\mathbf{b}.

Where this shows up

You now know the operation that most of a neural network's time is spent on.

The vectors in these systems have hundreds or thousands of entries, but the operation is exactly the one you did on paper.

In numpy, a @ b computes the dot product of two 1D arrays. The cell below computes the figure's pair both ways:

⌘+Enter runs · edit freelyPython sleeps until you run code

Now write both operations yourself: the dot product with a plain loop, so every multiply and add is visible, then the projection built on top of it.

Code itWrite dot and project

Implement dot(a, b) with a plain loop over Python lists (no numpy inside it), and raise ValueError if the lengths differ. Then implement project(a, b), which returns the projection of a onto the line through b as a list. Reuse your dot.

⌘+Enter runsPython sleeps until you run code
Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.

The practice set below generates fresh dot products, with 2 to 4 entries and small whole numbers, as many as you want.

Practice setDot product reps
3 correct in a row completes the set. A miss starts the count again.

Compute a⋅b\mathbf{a}\cdot\mathbf{b} for a=(6,1)\mathbf{a} = (6,\allowbreak 1) and b=(−3,5)\mathbf{b} = (-3,\allowbreak 5).

a · b

Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.

Finally, put the geometry into words. If you can explain the sign of a dot product with the shadow picture and no formulas, you have the idea this lesson is about.

In your own wordsExplain the sign

Without formulas, explain to a friend why the dot product of two perpendicular arrows is zero, and what a negative dot product tells you. Use the shadow picture.

A few sentences first: 0 of 60 characters.

Saved in this browser as you type.

Next: Length and similarity