The dot product

Here is a small recommender. You describe your taste in movies with three numbers: how much you like action, romance and comedy. Say your taste is $\mathbf{t} = (2, -1, 3)$: you like action, dislike romance, and love comedy. A movie gets the same three numbers for how much of each it contains, say $\mathbf{m} = (1, 2, 1)$.

How good a match is this movie? A sensible score: for each feature, multiply how much you care by how much the movie has, then add it all up.

$$\text{score} = (2)(1) + (-1)(2) + (3)(1) = 2 - 2 + 3 = 3$$

The romance term counts against the movie because you dislike romance. The comedy term helps the most because you care about comedy most. That multiply-then-add score is the dot product, and by the end of this lesson you will see it everywhere in machine learning: inside every neuron, inside attention, and inside every embedding search.

#Multiply matching entries, then add

For two vectors of the same length $n$, the dot product is

$$\mathbf{a}\cdot\mathbf{b} = a_1 b_1 + a_2 b_2 + \cdots + a_n b_n = \sum_{i=1}^{n} a_i b_i .$$

The symbol $\sum_{i=1}^{n}$ just means "add up the following for $i = 1, 2, \ldots, n$". Three things to notice:

• The answer is one number, not a vector. Two lists go in, a single score comes out.
• Both vectors must have the same length. There is no way to pair up entries otherwise. In code, a mismatch here is one of the most common errors you will see.
• Order does not matter: $\mathbf{a}\cdot\mathbf{b} = \mathbf{b}\cdot\mathbf{a}$, because $a_i b_i = b_i a_i$.

A second property matters more than it looks. The dot product is linear in each argument: scaling a vector scales the score, and splitting a vector splits the score.

$$(c\,\mathbf{a})\cdot\mathbf{b} = c\,(\mathbf{a}\cdot\mathbf{b}), \qquad (\mathbf{a} + \mathbf{a}')\cdot\mathbf{b} = \mathbf{a}\cdot\mathbf{b} + \mathbf{a}'\cdot\mathbf{b}.$$

Double your enthusiasm for every genre and every score doubles. You will lean on this property constantly once we reach matrices, which are nothing more than many dot products at once.

#What the number means

The algebra says nothing about arrows. But a 2D vector is also an arrow from the origin, and the dot product turns out to measure something visual. Drag the two arrows below and watch the number.

Play with it until these three facts feel obvious:

1. When the arrows point roughly the same way, the dot product is positive. The angle between them is under 90°.
2. When they are perpendicular, it is exactly zero. Rotate $\mathbf{a}$ until the shadow shrinks to a point.
3. When they point in roughly opposite directions, it is negative. The shadow lands on the far side of the origin.

And the size of the number grows with both lengths. The formula that captures all of this is

$$\mathbf{a}\cdot\mathbf{b} = |\mathbf{a}|\,|\mathbf{b}|\cos\theta,$$

where $|\mathbf{a}| = \sqrt{a_1^2 + a_2^2 + \cdots}$ is the length of $\mathbf{a}$ and $\theta$ is the angle between the arrows. The cosine is $1$ at $0°$, $0$ at $90°$ and $-1$ at $180°$, which is exactly the sign behavior you just saw.

It is not obvious that these two formulas agree. Multiplying coordinates and adding them has no visible angle in it. Here is why they must be the same.

Go slower: Why the two formulas agree

Draw $\mathbf{a}$ and $\mathbf{b}$ from the origin. The third side of that triangle is the vector $\mathbf{a} - \mathbf{b}$.

Step 1, with geometry. The law of cosines says that for any triangle with sides of length $|\mathbf{a}|$ and $|\mathbf{b}|$ and angle $\theta$ between them, the third side satisfies $|\mathbf{a} - \mathbf{b}|^2 = |\mathbf{a}|^2 + |\mathbf{b}|^2 - 2\,|\mathbf{a}|\,|\mathbf{b}|\cos\theta.$

Step 2, with algebra. The squared length of any vector is its dot product with itself: $|\mathbf{v}|^2 = v_1^2 + v_2^2 = \mathbf{v}\cdot\mathbf{v}$. So expand using linearity: $|\mathbf{a} - \mathbf{b}|^2 = (\mathbf{a} - \mathbf{b})\cdot(\mathbf{a} - \mathbf{b}) = \mathbf{a}\cdot\mathbf{a} - 2\,\mathbf{a}\cdot\mathbf{b} + \mathbf{b}\cdot\mathbf{b} = |\mathbf{a}|^2 + |\mathbf{b}|^2 - 2\,\mathbf{a}\cdot\mathbf{b}.$

Step 3, compare. Both lines describe the same length. Cancel $|\mathbf{a}|^2 + |\mathbf{b}|^2$ from each and divide by $-2$: $\mathbf{a}\cdot\mathbf{b} = |\mathbf{a}|\,|\mathbf{b}|\cos\theta.$

Nothing in this argument used two dimensions. It works for vectors with 3 entries or 4,096 entries, which is how we can talk about "the angle" between two word embeddings that live in spaces we cannot picture.

#The shadow: projection

The widget's green shadow has a precise meaning. Shine a light straight down onto the line through $\mathbf{b}$. The shadow that $\mathbf{a}$ casts on that line has signed length

$$\text{shadow length} = |\mathbf{a}|\cos\theta = \frac{\mathbf{a}\cdot\mathbf{b}}{|\mathbf{b}|}.$$

The second form comes from dividing the key formula by $|\mathbf{b}|$. So another way to say what the dot product is: the length of $\mathbf{a}$'s shadow on $\mathbf{b}$, times the length of $\mathbf{b}$. If $\mathbf{b}$ has length 1, the dot product is the shadow length. That is why we so often normalize vectors to length 1 before comparing them, which is the subject of the next lesson.

To get the shadow as a vector, rather than just its length, walk that far along the unit direction $\mathbf{b}/|\mathbf{b}|$:

$$\operatorname{proj}_{\mathbf{b}}\mathbf{a} = \frac{\mathbf{a}\cdot\mathbf{b}}{|\mathbf{b}|}\,\frac{\mathbf{b}}{|\mathbf{b}|} = \frac{\mathbf{a}\cdot\mathbf{b}}{\mathbf{b}\cdot\mathbf{b}}\,\mathbf{b}.$$

The last form needs no square roots, which is how you should compute it by hand.

#Where this shows up

You now know the operation that most of a neural network's time is spent on.

The numbers in these systems have hundreds or thousands of entries, but the operation is exactly the one you did on paper.

Now write it yourself. Python lists and a loop first, then numpy.