ml.lab
Python sleeps until you run code
02 Matrices are machines

Lesson 1 of 4

Matrices as transformations

A matrix is a function you can write down. Its columns record where the basis vectors land, and that is enough to know where every vector goes. Read row by row, the same numbers are a layer of neurons.

About 85 minutes
By the end you can
  • Test whether a map is linear by checking that it respects addition and scaling, the two operations from module 1.

  • Write the matrix of a transformation by finding where the basis vectors land.

  • Compute Ax\mathbf{A}\mathbf{x} two ways, as a combination of columns and as dot products with rows.

  • Derive the rotation matrix and recognize scales, stretches, shears, reflections, projections and singular matrices.

  • Read a layer of neurons as a matrix, one row of weights per neuron, and compute its outputs.

  • Implement matrix-vector multiplication both ways in Python.

Inside a language model, every token is carried as a vector, a list of a few thousand numbers like the ones in module 1, and each layer turns that vector into another vector. The machine that does most of that work is a matrix, a rectangular grid of numbers. This lesson is about what that machine does, and it starts with a small surprise.

Here is a rule that moves the points of the plane: send (x,y)(x,\allowbreak y) to (2x−y, x+y)(2x - y,\allowbreak \ x + y). Try it on the two simplest points. The point (1,0)(1,\allowbreak 0) goes to (2⋅1−0, 1+0)=(2,1)(2 \cdot 1 - 0,\allowbreak \ 1 + 0) = (2,\allowbreak 1), and the point (0,1)(0,\allowbreak 1) goes to (2⋅0−1, 0+1)=(−1,1)(2 \cdot 0 - 1,\allowbreak \ 0 + 1) = (-1,\allowbreak 1). Now, where does (3,2)(3,\allowbreak 2) go? You could substitute: (2⋅3−2, 3+2)=(4,5)(2 \cdot 3 - 2,\allowbreak \ 3 + 2) = (4,\allowbreak 5). Or you could notice that (3,2)(3,\allowbreak 2) is 3 steps along (1,0)(1,\allowbreak 0) plus 2 steps along (0,1)(0,\allowbreak 1), and guess that it lands 3 steps along (2,1)(2,\allowbreak 1) plus 2 steps along (−1,1)(-1,\allowbreak 1):

3 (2,1)+2 (−1,1)=(6,3)+(−2,2)=(4,5).3\,(2, 1) + 2\,(-1, 1) = (6, 3) + (-2, 2) = (4, 5).

Same answer. That is not luck. For this kind of rule, the two answers for (1,0)(1,\allowbreak 0) and (0,1)(0,\allowbreak 1) decide where every other point goes. Write those two answers side by side as columns and you have the rule's matrix.

The lesson goes in four steps. First, what "this kind of rule" means: the rule has to be linear. Second, why two columns then decide everything. Third, a gallery of the common matrices, each read off from its columns. Fourth, a second way to read the same grid of numbers, row by row, which is exactly how a layer of neurons uses it.

Linear means the grid stays a grid

The shortcut in the opening does not work for every rule. This section says which rules it works for, and what those rules look like.

In Arrows and lists you did two things with vectors: added them and scaled them. Write TT (for transformation) for a function that turns vectors into vectors, and T(u)T(\mathbf{u}) for the vector it produces from u\mathbf{u}. The function is linear when it respects both operations:

T(u+v)=T(u)+T(v),T(c u)=c T(u)T(\mathbf{u} + \mathbf{v}) = T(\mathbf{u}) + T(\mathbf{v}), \qquad T(c\,\mathbf{u}) = c\,T(\mathbf{u})

for all vectors u\mathbf{u}, v\mathbf{v} and every number cc. In words: adding two inputs and then transforming gives the same result as transforming each and then adding; scaling before or after makes no difference either.

Check both rules on the opening rule, T(x,y)=(2x−y, x+y)T(x,\allowbreak y) = (2x - y,\allowbreak \ x + y), with u=(1,0)\mathbf{u} = (1,\allowbreak 0), v=(0,1)\mathbf{v} = (0,\allowbreak 1) and c=3c = 3:

  • Addition. On the left, u+v=(1,1)\mathbf{u} + \mathbf{v} = (1,\allowbreak 1) and T(1,1)=(2−1, 1+1)=(1,2)T(1,\allowbreak 1) = (2 - 1,\allowbreak \ 1 + 1) = (1,\allowbreak 2). On the right, T(u)+T(v)=(2,1)+(−1,1)=(1,2)T(\mathbf{u}) + T(\mathbf{v}) = (2,\allowbreak 1) + (-1,\allowbreak 1) = (1,\allowbreak 2). Equal.
  • Scaling. On the left, 3u=(3,0)3\mathbf{u} = (3,\allowbreak 0) and T(3,0)=(6−0, 3+0)=(6,3)T(3,\allowbreak 0) = (6 - 0,\allowbreak \ 3 + 0) = (6,\allowbreak 3). On the right, 3 T(u)=3 (2,1)=(6,3)3\,T(\mathbf{u}) = 3\,(2,\allowbreak 1) = (6,\allowbreak 3). Equal.

One check with particular numbers is not a proof, since the rules must hold for every input. The slow box does it for every input at once.

Go slower: Why the opening rule passes for every input

Write u=(u1,u2)\mathbf{u} = (u_1,\allowbreak u_2) and v=(v1,v2)\mathbf{v} = (v_1,\allowbreak v_2).

Step 1, add first. u+v=(u1+v1, u2+v2)\mathbf{u} + \mathbf{v} = (u_1 + v_1,\allowbreak \ u_2 + v_2).

Step 2, apply the rule to that sum: T(u+v)=(2(u1+v1)−(u2+v2), (u1+v1)+(u2+v2)).T(\mathbf{u} + \mathbf{v}) = \big(2(u_1 + v_1) - (u_2 + v_2),\allowbreak \ (u_1 + v_1) + (u_2 + v_2)\big).

Step 3, expand the brackets in the first entry: 2u1+2v1−u2−v22u_1 + 2v_1 - u_2 - v_2.

Step 4, sort the terms by where they came from, the uu terms together and the vv terms together: the first entry is (2u1−u2)+(2v1−v2)(2u_1 - u_2) + (2v_1 - v_2) and the second is (u1+u2)+(v1+v2)(u_1 + u_2) + (v_1 + v_2).

Step 5, read the groups. (2u1−u2, u1+u2)(2u_1 - u_2,\allowbreak \ u_1 + u_2) is T(u)T(\mathbf{u}) and (2v1−v2, v1+v2)(2v_1 - v_2,\allowbreak \ v_1 + v_2) is T(v)T(\mathbf{v}), so T(u+v)=T(u)+T(v)T(\mathbf{u} + \mathbf{v}) = T(\mathbf{u}) + T(\mathbf{v}).

Step 6, scale first. c u=(cu1, cu2)c\,\mathbf{u} = (cu_1,\allowbreak \ cu_2).

Step 7, apply the rule to that: T(c u)=(2cu1−cu2, cu1+cu2)T(c\,\mathbf{u}) = (2cu_1 - cu_2,\allowbreak \ cu_1 + cu_2).

Step 8, pull out cc. Every term has a factor cc, so take it out of each entry: (2cu1−cu2, cu1+cu2)=c (2u1−u2, u1+u2)=c T(u)(2cu_1 - cu_2,\allowbreak \ cu_1 + cu_2) = c\,(2u_1 - u_2,\allowbreak \ u_1 + u_2) = c\,T(\mathbf{u}).

The same argument works whenever every output entry is a fixed weighted sum of the inputs, with no constant term and no products of inputs. That gives a quick test for linearity.

Put the two rules together and you get the version we will use all the time. For any numbers c1,c2c_1,\allowbreak c_2 and vectors v1,v2\mathbf{v}_1,\allowbreak \mathbf{v}_2, the addition rule splits the sum (first equals sign), and then the scaling rule pulls out each number (second equals sign):

T(c1v1+c2v2)=T(c1v1)+T(c2v2)=c1 T(v1)+c2 T(v2).T(c_1\mathbf{v}_1 + c_2\mathbf{v}_2) = T(c_1\mathbf{v}_1) + T(c_2\mathbf{v}_2) = c_1\,T(\mathbf{v}_1) + c_2\,T(\mathbf{v}_2).

A linear function carries linear combinations through unchanged. The recipe "this much of v1\mathbf{v}_1, that much of v2\mathbf{v}_2" survives the trip, with each ingredient replaced by its image. The opening used exactly this: (3,2)=3 (1,0)+2 (0,1)(3,\allowbreak 2) = 3\,(1,\allowbreak 0) + 2\,(0,\allowbreak 1), so T(3,2)=3 T(1,0)+2 T(0,1)T(3,\allowbreak 2) = 3\,T(1,\allowbreak 0) + 2\,T(0,\allowbreak 1). That sentence is behind everything in this lesson.

What does linearity look like? Three things follow from the two rules:

  • The origin stays put. Take c=0c = 0 in the scaling rule: T(0)=T(0 u)=0 T(u)=0T(\mathbf{0}) = T(0\,\mathbf{u}) = 0\,T(\mathbf{u}) = \mathbf{0}.
  • Straight lines stay straight, or collapse to a single point.
  • Parallel, evenly spaced grid lines stay parallel and evenly spaced. The square grid of the plane becomes a grid of identical parallelograms. It can be stretched, turned, flipped or flattened, but never bent.
Go slower: Why lines stay lines and spacing stays even

A line through the point p\mathbf{p} with direction d\mathbf{d} is the set of all points p+t d\mathbf{p} + t\,\mathbf{d}, where tt ranges over every number.

Step 1. Apply TT and use the addition rule, then the scaling rule: T(p+t d)=T(p)+T(t d)=T(p)+t T(d).T(\mathbf{p} + t\,\mathbf{d}) = T(\mathbf{p}) + T(t\,\mathbf{d}) = T(\mathbf{p}) + t\,T(\mathbf{d}).

Step 2. Read the result. As tt varies, T(p)+t T(d)T(\mathbf{p}) + t\,T(\mathbf{d}) traces the line through the point T(p)T(\mathbf{p}) with direction T(d)T(\mathbf{d}). If T(d)=0T(\mathbf{d}) = \mathbf{0}, every point of the line lands on the single point T(p)T(\mathbf{p}).

Step 3, parallel lines. Two parallel lines share the direction d\mathbf{d}, so their images share the direction T(d)T(\mathbf{d}): still parallel.

Step 4, even spacing. The points at t=0,1,2,3t = 0,\allowbreak 1,\allowbreak 2,\allowbreak 3 are evenly spaced along the original line. Their images T(p), T(p)+T(d), T(p)+2 T(d), …T(\mathbf{p}),\allowbreak \ T(\mathbf{p}) + T(\mathbf{d}),\allowbreak \ T(\mathbf{p}) + 2\,T(\mathbf{d}),\allowbreak \ \ldots are evenly spaced too, one step of T(d)T(\mathbf{d}) apart.

A grid is made of such lines, so a grid of squares becomes a grid of parallelograms.

Knowing what is not linear is just as useful. (Read the arrow ↦\mapsto as "goes to".) Each of these fails at least one rule, and one set of numbers is enough to show it:

  • Shifting, (x,y)↦(x+1,y)(x,\allowbreak y) \mapsto (x + 1,\allowbreak y). It sends (0,0)(0,\allowbreak 0) to (1,0)(1,\allowbreak 0), but a linear map must keep the origin fixed.
  • Products and powers of the inputs, such as (x,y)↦(xy, y)(x,\allowbreak y) \mapsto (xy,\allowbreak \ y). It sends (1,1)(1,\allowbreak 1) to (1,1)(1,\allowbreak 1) and (2,2)(2,\allowbreak 2) to (4,2)(4,\allowbreak 2). Doubling the input quadrupled the first output, where the scaling rule demands 2 (1,1)=(2,2)2\,(1,\allowbreak 1) = (2,\allowbreak 2).
  • Bending: sin⁡\sin, tanh⁡\tanh, the ReLU, applied to single numbers (vectors with one entry). The ReLU keeps positive numbers and turns negative ones into 0. It fails the addition rule: ReLU(−1+1)=ReLU(0)=0\text{ReLU}(-1 + 1) = \text{ReLU}(0) = 0, but ReLU(−1)+ReLU(1)=0+1=1\text{ReLU}(-1) + \text{ReLU}(1) = 0 + 1 = 1. The nonlinearities in a neural network are exactly the parts that break these rules, on purpose (Neurons and layers shows why they have to).

Three panels on a dark background, each with a faint square grid and a white dot at its center, the origin. In the left panel a bright grid of identical parallelograms, one family of lines blue and the other coral, is drawn over the square grid; its lines are straight, parallel and evenly spaced, and one crossing sits exactly on the white dot. In the middle panel a bright square grid is moved half a cell right and half a cell up, so its lines run midway between the faint ones and none of its crossings lands on the white dot. In the right panel the bright grid lines are smooth wavy curves that still cross at the white dot.Left: a linear map. Grid lines stay straight, parallel and evenly spaced, and the origin stays put. Middle: a shift moves the origin, so it is not linear. Right: a map that bends the grid lines is not linear either, even though it keeps the origin in place.

Now test the two rules yourself on five maps. The quick test from the slow box (fixed weighted sums, no constants, no products) will do most of the work.

Quick checkWhich maps are linear?

Which of these maps of the plane are linear? Test each one against the two rules T(u+v)=T(u)+T(v)T(\mathbf{u} + \mathbf{v}) = T(\mathbf{u}) + T(\mathbf{v}) and T(c u)=c T(u)T(c\,\mathbf{u}) = c\,T(\mathbf{u}). Select every linear one; more than one is.

Choose every option that applies, then check.

The columns are where the basis vectors land

The opening guessed that two answers decide where every point goes. Linearity turns that guess into a fact, and the fact hands us the matrix.

In the plane, the standard basis vectors from Arrows and lists, e1=(1,0)\mathbf{e}_1 = (1,\allowbreak 0) and e2=(0,1)\mathbf{e}_2 = (0,\allowbreak 1), are also written ı^\hat{\imath} and ȷ^\hat{\jmath}, read "i-hat" and "j-hat". Every vector is a combination of them: (x1,x2)=x1 ı^+x2 ȷ^(x_1,\allowbreak x_2) = x_1\,\hat{\imath} + x_2\,\hat{\jmath}. The coordinates are the coefficients. Now apply a linear TT and let it carry the combination through, as the last section showed it must:

T(x)=T(x1 ı^+x2 ȷ^)=x1 T(ı^)+x2 T(ȷ^).T(\mathbf{x}) = T(x_1\,\hat{\imath} + x_2\,\hat{\jmath}) = x_1\,T(\hat{\imath}) + x_2\,T(\hat{\jmath}).

So once you know the two vectors T(ı^)T(\hat{\imath}) and T(ȷ^)T(\hat{\jmath}), you know TT everywhere. Write them down as the two columns of a grid of numbers. That grid is the matrix of TT.

For the rule in the opening, T(ı^)=(2,1)T(\hat{\imath}) = (2,\allowbreak 1) and T(ȷ^)=(−1,1)T(\hat{\jmath}) = (-1,\allowbreak 1), so

A=[2−111],\mathbf{A} = \begin{bmatrix} 2 & -1 \\ 1 & 1 \end{bmatrix},

and multiplying A\mathbf{A} by a vector is defined to mean that combination of its columns:

Ax=x1[21]+x2[−11]=[2x1−x2x1+x2].\mathbf{A}\mathbf{x} = x_1 \begin{bmatrix} 2 \\ 1 \end{bmatrix} + x_2 \begin{bmatrix} -1 \\ 1 \end{bmatrix} = \begin{bmatrix} 2x_1 - x_2 \\ x_1 + x_2 \end{bmatrix}.

With x=(3,2)\mathbf{x} = (3,\allowbreak 2), the columns give 3 (2,1)+2 (−1,1)=(6−2, 3+2)=(4,5)3\,(2,\allowbreak 1) + 2\,(-1,\allowbreak 1) = (6 - 2,\allowbreak \ 3 + 2) = (4,\allowbreak 5), as before. (Inside sentences we write vectors in a row, like (3,2)(3,\allowbreak 2), to save space. In the math they are columns, which is why the columns of A\mathbf{A} stand upright in the grid.)

The widget below draws this matrix. It answers one question: if you know only where the two basis vectors land, can you find where any vector lands? On screen is the plane with its faint square grid. The blue arrow Aı^\mathbf{A}\hat{\imath} ends at (2,1)(2,\allowbreak 1), the first column; the coral arrow Aȷ^\mathbf{A}\hat{\jmath} ends at (−1,1)(-1,\allowbreak 1), the second column. The blue and coral lines are the whole grid after A\mathbf{A} has moved it. The dashed white arrow is the input v=(3,2)\mathbf{v} = (3,\allowbreak 2) and the solid white arrow is its image Av\mathbf{A}\mathbf{v}.

Here is the general statement. A matrix with mm rows and nn columns is called m×nm \times n ("mm by nn"). Its entry in row ii and column jj is written AijA_{ij}: row first, then column. Call its columns a1,…,an\mathbf{a}_1,\allowbreak \ldots,\allowbreak \mathbf{a}_n. For a vector x\mathbf{x} with nn entries,

Ax=x1 a1+x2 a2+⋯+xn an.\mathbf{A}\mathbf{x} = x_1\,\mathbf{a}_1 + x_2\,\mathbf{a}_2 + \cdots + x_n\,\mathbf{a}_n .

There is one coefficient per column, so the input needs nn entries. Each column has mm entries, so the output has mm. Writing Rn\mathbb{R}^n for the set of all vectors with nn real entries, an m×nm \times n matrix is a function from Rn\mathbb{R}^n to Rm\mathbb{R}^m. Notice that the order flips: the number of columns (second in the shape) is the input size, and the number of rows (first) is the output size. A 3×23 \times 2 matrix, for example, takes points of the plane to points in 3D. Its two columns are two arrows in 3D, and every output is a combination of them, so the whole plane lands on a flat sheet through the origin in 3D space (or on a line, if the two columns are parallel).

The logic also runs the other way: every matrix gives a linear function. Check the addition rule, one move per equals sign. As in module 1, ∑j\sum_j means "add up over every jj", here over the columns j=1,…,nj = 1,\allowbreak \ldots,\allowbreak n, so Ax=∑jxj aj\mathbf{A}\mathbf{x} = \sum_j x_j\,\mathbf{a}_j:

A(u+v)=∑j(uj+vj) aj=∑j(uj aj+vj aj)=∑juj aj+∑jvj aj=Au+Av.\mathbf{A}(\mathbf{u} + \mathbf{v}) = \sum_j (u_j + v_j)\,\mathbf{a}_j = \sum_j \big(u_j\,\mathbf{a}_j + v_j\,\mathbf{a}_j\big) = \sum_j u_j\,\mathbf{a}_j + \sum_j v_j\,\mathbf{a}_j = \mathbf{A}\mathbf{u} + \mathbf{A}\mathbf{v}.

The first equals sign is the definition, since the entries of u+v\mathbf{u} + \mathbf{v} are uj+vju_j + v_j. The second multiplies out each term. The third splits one sum into two. The last reads each sum as a matrix times a vector. Scaling is shorter: A(c u)=∑j(c uj) aj=c∑juj aj=c Au\mathbf{A}(c\,\mathbf{u}) = \sum_j (c\,u_j)\,\mathbf{a}_j = c\sum_j u_j\,\mathbf{a}_j = c\,\mathbf{A}\mathbf{u}, pulling the common factor cc out of every term. So "linear function" and "matrix" are two names for one thing, once you fix the basis. That is why this module is called Matrices are machines: a matrix is a function you can write down.

To recap: a linear map is decided by where the basis vectors land; those landing points are the columns of its matrix; and Ax\mathbf{A}\mathbf{x} takes x1x_1 of the first column, x2x_2 of the second, and so on.

The first exercise goes from a description in words to a matrix and back to a point, the way the opening did.

On paperWrite the matrix, then apply it

For each transformation, find where ı^=(1,0)\hat{\imath} = (1,\allowbreak 0) and ȷ^=(0,1)\hat{\jmath} = (0,\allowbreak 1) land, write those two vectors as the columns of a 2×22 \times 2 matrix, then apply the matrix to the given vector as a combination of its columns: (first entry) times column 1 plus (second entry) times column 2.

  1. Rotate a quarter turn counterclockwise. Apply it to (2,1)(2,\allowbreak 1).
  2. Reflect across the line y=xy = x. Apply it to (3,−1)(3,\allowbreak -1).
  3. Stretch horizontally by 3 and turn the plane upside down (every yy becomes −y-y). Apply it to (1,2)(1,\allowbreak 2).
  4. The shear that keeps ı^\hat{\imath} where it is and slides ȷ^\hat{\jmath} to (2,1)(2,\allowbreak 1). Apply it to (1,1)(1,\allowbreak 1).

To check, enter the four images in order, one per row, as xx then yy.

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

The four images in order, one per row: x, then y

One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

The next one gives you clues about two vectors that are not both basis vectors, so you have to use linearity to find the missing column.

Work it outA matrix from two clues

A linear map sends (1,0)(1,\allowbreak 0) to (2,1)(2,\allowbreak 1) and sends (1,1)(1,\allowbreak 1) to (1,3)(1,\allowbreak 3). Find its matrix. You are not told directly where (0,1)(0,\allowbreak 1) goes. Enter the 2×22 \times 2 matrix row by row.

A

One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Most matrices you will meet are built from a handful of basic moves: scaling, stretching, shearing, rotating, reflecting and flattening. Each one is easiest to recognize by where it sends ı^\hat{\imath} and ȷ^\hat{\jmath}, so it is worth seeing each one move the grid once.

The widget below starts at the identity, the matrix that moves nothing, and has one button per move. On screen are the faint original grid, the moved grid, the blue and coral arrows (the two columns, which you can also read in the matrix cells: blue cells are column 1, coral cells column 2), and the unit square, the square with corners (0,0)(0,\allowbreak 0), (1,0)(1,\allowbreak 0), (1,1)(1,\allowbreak 1) and (0,1)(0,\allowbreak 1). Where it lands is shaded, and the area readout gives the shaded area. The sentence under the t slider names the move.

Every entry in this table comes from the same question, where do ı^\hat{\imath} and ȷ^\hat{\jmath} go. The two middle columns of the table are the two columns of the matrix. The letter kk stands for any amount you choose.

Transformationı^\hat{\imath} lands atȷ^\hat{\jmath} lands atMatrix
Scale everything by kk(k,0)(k,\allowbreak 0)(0,k)(0,\allowbreak k)[k00k]\begin{bmatrix} k & 0 \\ 0 & k \end{bmatrix}
Stretch horizontally by kk(k,0)(k,\allowbreak 0)(0,1)(0,\allowbreak 1)[k001]\begin{bmatrix} k & 0 \\ 0 & 1 \end{bmatrix}
Shear sideways by kk(1,0)(1,\allowbreak 0)(k,1)(k,\allowbreak 1)[1k01]\begin{bmatrix} 1 & k \\ 0 & 1 \end{bmatrix}
Rotate by θ\theta(cos⁡θ,sin⁡θ)(\cos\theta,\allowbreak \sin\theta)(−sin⁡θ,cos⁡θ)(-\sin\theta,\allowbreak \cos\theta)[cos⁡θ−sin⁡θsin⁡θcos⁡θ]\begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}
Reflect across the xx-axis(1,0)(1,\allowbreak 0)(0,−1)(0,\allowbreak -1)[100−1]\begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix}
Reflect across y=xy = x(0,1)(0,\allowbreak 1)(1,0)(1,\allowbreak 0)[0110]\begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}
Project onto the xx-axis(1,0)(1,\allowbreak 0)(0,0)(0,\allowbreak 0)[1000]\begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix}
Squash onto a line (singular)(1,2)(1,\allowbreak 2)(2,4)(2,\allowbreak 4)[1224]\begin{bmatrix} 1 & 2 \\ 2 & 4 \end{bmatrix}

Three of these deserve a closer look.

Rotation. The unit circle is the circle of radius 1 around the origin, and the point on it at angle α\alpha (measured counterclockwise from the positive xx-axis) is (cos⁡α,sin⁡α)(\cos\alpha,\allowbreak \sin\alpha): that is the definition of cosine and sine. Rotating counterclockwise by θ\theta carries ı^\hat{\imath}, which sits on the unit circle at angle 00, to the point at angle θ\theta, which is (cos⁡θ,sin⁡θ)(\cos\theta,\allowbreak \sin\theta). The vector ȷ^\hat{\jmath} starts a quarter turn ahead, at angle 90∘90^\circ, so it lands at angle θ+90∘\theta + 90^\circ, which is the point (−sin⁡θ,cos⁡θ)(-\sin\theta,\allowbreak \cos\theta). Those are the two columns.

Go slower: Where ȷ^\hat{\jmath} lands under a rotation

The point on the unit circle at angle α\alpha is (cos⁡α,sin⁡α)(\cos\alpha,\allowbreak \sin\alpha). For ȷ^\hat{\jmath} after the rotation, α=θ+90∘\alpha = \theta + 90^\circ. Use the angle-addition formulas, then cos⁡90∘=0\cos 90^\circ = 0 and sin⁡90∘=1\sin 90^\circ = 1: cos⁡(θ+90∘)=cos⁡θcos⁡90∘−sin⁡θsin⁡90∘=cos⁡θ⋅0−sin⁡θ⋅1=−sin⁡θ,\cos(\theta + 90^\circ) = \cos\theta\cos 90^\circ - \sin\theta\sin 90^\circ = \cos\theta \cdot 0 - \sin\theta \cdot 1 = -\sin\theta, sin⁡(θ+90∘)=sin⁡θcos⁡90∘+cos⁡θsin⁡90∘=sin⁡θ⋅0+cos⁡θ⋅1=cos⁡θ.\sin(\theta + 90^\circ) = \sin\theta\cos 90^\circ + \cos\theta\sin 90^\circ = \sin\theta \cdot 0 + \cos\theta \cdot 1 = \cos\theta. So ȷ^\hat{\jmath} lands at (−sin⁡θ,cos⁡θ)(-\sin\theta,\allowbreak \cos\theta). A shortcut that gives the same answer: turning any vector (a,b)(a,\allowbreak b) a quarter turn counterclockwise gives (−b,a)(-b,\allowbreak a), and the image of ȷ^\hat{\jmath} is the image of ı^\hat{\imath}, (cos⁡θ,sin⁡θ)(\cos\theta,\allowbreak \sin\theta), turned a further quarter turn.

Two angles as examples. For θ=90∘\theta = 90^\circ, cos⁡θ=0\cos\theta = 0 and sin⁡θ=1\sin\theta = 1, so the columns are (0,1)(0,\allowbreak 1) and (−1,0)(-1,\allowbreak 0): the matrix [0−110]\begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix} you saw in the widget. For θ=30∘\theta = 30^\circ, cos⁡θ=3/2≈0.866\cos\theta = \sqrt{3}/2 \approx 0.866 and sin⁡θ=1/2\sin\theta = 1/2, so the matrix is [3/2−1/21/23/2]≈[0.866−0.50.50.866]\begin{bmatrix} \sqrt{3}/2 & -1/2 \\ 1/2 & \sqrt{3}/2 \end{bmatrix} \approx \begin{bmatrix} 0.866 & -0.5 \\ 0.5 & 0.866 \end{bmatrix}. It sends (2,0)(2,\allowbreak 0) to 2 times the first column, 2 (3/2, 1/2)=(3,1)≈(1.732,1)2\,(\sqrt{3}/2,\allowbreak \ 1/2) = (\sqrt{3},\allowbreak 1) \approx (1.732,\allowbreak 1). Its length is (3)2+12=3+1=2\sqrt{(\sqrt{3})^2 + 1^2} = \sqrt{3 + 1} = 2, the same as before, as a rotation should leave it.

Projection. The projection onto the xx-axis keeps the xx-part of every vector and throws away the yy-part, so ȷ^\hat{\jmath} lands on 0\mathbf{0}. Projections onto other lines are matrices too. In The dot product you found that the projection of a\mathbf{a} onto the line through b\mathbf{b} is a⋅bb⋅b b\frac{\mathbf{a}\cdot\mathbf{b}}{\mathbf{b}\cdot\mathbf{b}}\,\mathbf{b}. Take b=(1,1)\mathbf{b} = (1,\allowbreak 1). Then b⋅b=1+1=2\mathbf{b}\cdot\mathbf{b} = 1 + 1 = 2 and a⋅b=a1+a2\mathbf{a}\cdot\mathbf{b} = a_1 + a_2, so the projection is a1+a22 (1,1)\frac{a_1 + a_2}{2}\,(1,\allowbreak 1). For ı^=(1,0)\hat{\imath} = (1,\allowbreak 0) that is 1+02 (1,1)=(12,12)\frac{1 + 0}{2}\,(1,\allowbreak 1) = (\tfrac{1}{2},\allowbreak \tfrac{1}{2}), and for ȷ^=(0,1)\hat{\jmath} = (0,\allowbreak 1) it is 0+12 (1,1)=(12,12)\frac{0 + 1}{2}\,(1,\allowbreak 1) = (\tfrac{1}{2},\allowbreak \tfrac{1}{2}) as well. So the matrix is [1/21/21/21/2]\begin{bmatrix} 1/2 & 1/2 \\ 1/2 & 1/2 \end{bmatrix}. Check it on a=(4,2)\mathbf{a} = (4,\allowbreak 2) from that lesson's exercise: 4 (12,12)+2 (12,12)=(2,2)+(1,1)=(3,3)4\,(\tfrac12,\allowbreak \tfrac12) + 2\,(\tfrac12,\allowbreak \tfrac12) = (2,\allowbreak 2) + (1,\allowbreak 1) = (3,\allowbreak 3), the same projection you computed there. Applying the matrix a second time changes nothing: 3 (12,12)+3 (12,12)=(3,3)3\,(\tfrac12,\allowbreak \tfrac12) + 3\,(\tfrac12,\allowbreak \tfrac12) = (3,\allowbreak 3). The point is already on the line, so its shadow is itself.

Singular matrices. In [1224]\begin{bmatrix} 1 & 2 \\ 2 & 4 \end{bmatrix} the second column is twice the first. (The widget's Singular preset uses a different pair with the same property, columns (1,0.5)(1,\allowbreak 0.5) and (2,1)(2,\allowbreak 1).) Every output is x1(1,2)+x2(2,4)x_1(1,\allowbreak 2) + x_2(2,\allowbreak 4). Replace (2,4)(2,\allowbreak 4) by 2 (1,2)2\,(1,\allowbreak 2) and collect the two multiples of (1,2)(1,\allowbreak 2):

x1(1,2)+x2⋅2 (1,2)=(x1+2x2) (1,2).x_1(1, 2) + x_2 \cdot 2\,(1, 2) = (x_1 + 2x_2)\,(1, 2).

So every output is a multiple of (1,2)(1,\allowbreak 2), and the whole plane is squashed onto the line through (1,2)(1,\allowbreak 2). Different inputs collide. For (1,1)(1,\allowbreak 1) the multiple is 1+2⋅1=31 + 2 \cdot 1 = 3, and for (3,0)(3,\allowbreak 0) it is 3+2⋅0=33 + 2 \cdot 0 = 3, so both land on 3 (1,2)=(3,6)3\,(1,\allowbreak 2) = (3,\allowbreak 6). For (2,−1)(2,\allowbreak -1) it is 2+2⋅(−1)=02 + 2 \cdot (-1) = 0, so it lands on the origin. Once two inputs share an output, nothing can tell you which one you started from, so no matrix can undo this one. Module 3 measures this collapse with the determinant and counts the surviving dimensions with rank.

To recap the gallery: every matrix in it came from the one question, where do ı^\hat{\imath} and ȷ^\hat{\jmath} land. Rotations and shears keep areas, reflections flip the plane over, and a matrix whose columns lie on one line flattens the plane and cannot be undone.

Now read a matrix straight off a picture, the way you read the presets.

Quick checkRead the matrix off the picture

In the transformation widget you see this picture: the blue arrow, the image of ı^\hat{\imath}, ends at (1,1)(1,\allowbreak 1). The coral arrow, the image of ȷ^\hat{\jmath}, ends at (−1,1)(-1,\allowbreak 1). The unit square has become a larger square, turned 45∘45^\circ counterclockwise. Which matrix is on the screen?

Choose one answer, then check.

The row view: every output is a dot product

The column view says what a matrix does to the plane. There is a second way to read the same arithmetic, and it is the one a neural network uses, so it is worth deriving carefully. Write Ax\mathbf{A}\mathbf{x} out entry by entry for a 2×22 \times 2 matrix. The first equals sign is the column view; the second multiplies each column by its number and adds the two columns entry by entry:

Ax=x1[A11A21]+x2[A12A22]=[A11x1+A12x2A21x1+A22x2].\mathbf{A}\mathbf{x} = x_1 \begin{bmatrix} A_{11} \\ A_{21} \end{bmatrix} + x_2 \begin{bmatrix} A_{12} \\ A_{22} \end{bmatrix} = \begin{bmatrix} A_{11}x_1 + A_{12}x_2 \\ A_{21}x_1 + A_{22}x_2 \end{bmatrix}.

Now look at the top entry, A11x1+A12x2A_{11}x_1 + A_{12}x_2. It uses only the first row of A\mathbf{A}, (A11,A12)(A_{11},\allowbreak A_{12}), and it multiplies matching entries and adds: it is (A11,A12)⋅(x1,x2)(A_{11},\allowbreak A_{12})\cdot(x_1,\allowbreak x_2), row 1 dotted with x\mathbf{x}. The bottom entry is row 2 dotted with x\mathbf{x}. In general, writing (Ax)i(\mathbf{A}\mathbf{x})_i for entry ii of the vector Ax\mathbf{A}\mathbf{x}, and with ∑j=1n\sum_{j=1}^{n} meaning "add up for j=1,2,…,nj = 1,\allowbreak 2,\allowbreak \ldots,\allowbreak n",

(Ax)i=∑j=1nAij xj=(row i of A)⋅x.(\mathbf{A}\mathbf{x})_i = \sum_{j=1}^{n} A_{ij}\,x_j = (\text{row } i \text{ of } \mathbf{A})\cdot\mathbf{x}.

Both views compute the same terms AijxjA_{ij}x_j; they only group them differently. The row view fixes the output entry ii and adds along the row. The column view collects every term that shares the same xjx_j, which gives xjx_j times column jj.

Check it on the opening example. The rows of A=[2−111]\mathbf{A} = \begin{bmatrix} 2 & -1 \\ 1 & 1 \end{bmatrix} are (2,−1)(2,\allowbreak -1) and (1,1)(1,\allowbreak 1). With x=(3,2)\mathbf{x} = (3,\allowbreak 2), the first entry is (2)(3)+(−1)(2)=6−2=4(2)(3) + (-1)(2) = 6 - 2 = 4 and the second is (1)(3)+(1)(2)=3+2=5(1)(3) + (1)(2) = 3 + 2 = 5. The same (4,5)(4,\allowbreak 5) we built from the columns.

Each view earns its keep:

  • The row view is how you compute by hand, and how you think about each output separately. Every row is a pattern, and output ii measures how much x\mathbf{x} lines up with pattern ii: large and positive when they agree, zero when they are perpendicular, negative when they disagree. That is the dot product's meaning, from The dot product.
  • The column view is how you see geometry. Every output is a combination of the columns, so the outputs can only ever reach the span of the columns (module 1). In module 3 that set gets a name, the column space.

A layer of neurons is a matrix

"Every row is a pattern" is not a figure of speech. It is exactly what a layer of a neural network is, and this is the first place in the course where a matrix meets one.

A neuron takes a list of numbers x\mathbf{x}, multiplies each by its own weight, adds the products, and adds one more number, its bias: it computes w⋅x+b\mathbf{w}\cdot\mathbf{x} + b, where w\mathbf{w} is its weight vector and bb its bias. A layer is several neurons reading the same input, each with its own weights and bias. Stack the weight vectors as the rows of a matrix W\mathbf{W} and the biases into a vector b\mathbf{b}, and the row view says that Wx+b\mathbf{W}\mathbf{x} + \mathbf{b} computes every neuron at once. (In a network a nonlinearity such as the ReLU then acts on each output. This lesson leaves it aside; Layers and parameter counts adds it.)

The diagram below answers the question: how does a grid of numbers become a layer of neurons? It has two parts, side by side on a wide screen and one above the other on a phone. The network drawing shows three inputs x1,x2,x3x_1,\allowbreak x_2,\allowbreak x_3 (blue circles holding 2, 1 and −1-1) joined by lines to two neurons (white circles), one weight per line, with the biases b1=1b_1 = 1 and b2=2b_2 = 2 written beside the neurons. The equation shows the same layer as Wx+b=z\mathbf{W}\mathbf{x} + \mathbf{b} = \mathbf{z}, with each shape (rows × columns) written under its bracket. Move through it with Next and Back, or the numbered dots.

In matrix language, a layer of mm neurons that each read nn inputs is an m×nm \times n weight matrix: one row per neuron, one column per input. Now compute one yourself, with three neurons this time and two inputs.

Work it outThree neurons, three dot products

A layer of three neurons has weight rows (1,2)(1,\allowbreak 2), (0,−1)(0,\allowbreak -1) and (3,1)(3,\allowbreak 1), stacked into W=[120−131]\mathbf{W} = \begin{bmatrix} 1 & 2 \\ 0 & -1 \\ 3 & 1 \end{bmatrix}. There is no bias. Compute Wx\mathbf{W}\mathbf{x} for x=(2,−1)\mathbf{x} = (2,\allowbreak -1) with the row view: one dot product per neuron. Enter the three outputs as a column, neuron 1 at the top.

Wx

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Both views in code

Writing each view as a loop forces you to say which index moves, and that is exactly the difference between them: the row view moves along a row for each output entry, and the column view adds one whole column per input entry.

Here are both views in numpy, and the 30∘30^\circ rotation from the gallery. Note that numpy counts from 0: A[0] is the first row and A[:, 0] is the first column. The first three lines should all print [4. 5.], and the rotation should send (2,0)(2,\allowbreak 0) to about (1.732,1)(1.732,\allowbreak 1), of length 2.

⌘+Enter runs · edit freelyPython sleeps until you run code

Now write both views yourself, without numpy's @, and build a rotation matrix from where the basis vectors go.

Code itMatrix times vector, both ways

Write matrix-vector multiplication twice, without @ or np.dot, so that both views from this lesson become code.

  • matvec_rows(A, x): entry ii is row ii of A dotted with x.
  • matvec_cols(A, x): x[0] times column 0, plus x[1] times column 1, and so on.
  • Both raise ValueError when x does not have one entry per column of A, and both work for non-square matrices.
  • rotation(theta) returns the 2×22 \times 2 matrix that rotates counterclockwise by theta radians, built from where ı^\hat{\imath} and ȷ^\hat{\jmath} land.

In numpy, A[i] is row i, A[:, j] is column j, and indices start at 0 (the math starts at 1).

⌘+Enter runsPython sleeps until you run code
Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.

A few quick reps by hand, with new numbers each time, make the row view automatic.

Practice setMatrix-vector reps
3 correct in a row completes the set. A miss starts the count again.

Compute Ax\mathbf{A}\mathbf{x} for A=[−21−3−11−3]\mathbf{A} = \begin{bmatrix}-2 & 1 & -3 \\ -1 & 1 & -3\end{bmatrix} and x=[21−2]\mathbf{x} = \begin{bmatrix}2 \\ 1 \\ -2\end{bmatrix}.

Ax

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

To recap the lesson: a matrix is a linear map; its columns are where the basis vectors land, so Ax\mathbf{A}\mathbf{x} is a combination of the columns; and read row by row, the same product is a stack of dot products, which is a layer of neurons. To finish, put the whole idea into your own words.

In your own wordsMore than a table of numbers

A colleague says a matrix is just a table of numbers. Explain what the columns of a 2×22 \times 2 matrix mean geometrically, why those two columns are enough to know where every vector goes, and what the row view adds.

A few sentences first: 0 of 60 characters.

Saved in this browser as you type.

Next: Matrix multiplication