ml.lab
Python sleeps until you run code
06 Neural networks from scratch

Lesson 1 of 6

Neurons and layers

A neuron is a dot product, a bias and a bend. A layer is many neurons reading the same input, which is one matrix multiply, and a network is layers applied one after another. Without the bend, a stack of layers of any depth is still a single layer.

About 45 minutes
By the end you can
  • Compute a neuron's output and read its decision boundary, the line where z=0z = 0, perpendicular to w\mathbf{w}, with every input at signed distance z/∣w∣z/|\mathbf{w}| from it.

  • Compute a layer and a small network by hand, in both the column convention and the row-batch convention used in code.

  • Count the parameters of a network from its layer sizes.

  • Prove that layers without a nonlinearity collapse into a single affine map.

In the dot product lesson, a movie's score was your taste vector dotted with the movie's features. Add a threshold ("recommend it if the score beats 1"), which is the same as subtracting 1 from the score and asking whether the result is positive. Pass that result through a function that turns it into a number between 0 and 1, and you have built a neuron.

Put a few neurons side by side, all reading the same input, and you have a layer. Feed one layer's outputs into another layer and you have a neural network. Modules 1 to 5 built every piece of mathematics this needs. This lesson assembles the pieces: one neuron and the line it draws, a layer as one matrix multiply, a network as layers in sequence, and a short proof of why the bend inside each neuron cannot be left out. The next lesson shows what that bend buys.

One neuron: a dot product, a bias and a bend

Everything in this module is built from one small unit, so it pays to know it exactly. A neuron takes an input vector x=(x1,…,xn)\mathbf{x} = (x_1,\allowbreak \ldots,\allowbreak x_n) and computes two numbers:

z=w⋅x+b,a=ϕ(z).z = \mathbf{w}\cdot\mathbf{x} + b, \qquad a = \phi(z).

Here is what each symbol means.

  • w=(w1,…,wn)\mathbf{w} = (w_1,\allowbreak \ldots,\allowbreak w_n) is the neuron's weight vector, one weight per input.
  • bb is its bias, a single number added at the end. In the movie example, b=−1b = -1 plays the threshold.
  • zz is the pre-activation: the weighted sum plus the bias.
  • ϕ\phi (the Greek letter phi) is the activation function, a fixed nonlinear function applied to zz.
  • aa is the neuron's output, its activation.

The weights and the bias are learned from data; ϕ\phi is chosen in advance. Two common choices of ϕ\phi are the ReLU, ReLU⁡(z)=max⁡(0,z)\operatorname{ReLU}(z) = \max(0,\allowbreak z), which passes positive values and replaces negative ones by 0, and the sigmoid σ(z)=1/(1+e−z)\sigma(z) = 1/(1 + e^{-z}) from the derivatives lesson, which squashes any number into the interval (0,1)(0,\allowbreak 1).

A worked example. Take w=(2,−1)\mathbf{w} = (2,\allowbreak -1) and b=0.5b = 0.5.

  • For x=(2,1)\mathbf{x} = (2,\allowbreak 1): the products are (2)(2)=4(2)(2) = 4 and (−1)(1)=−1(-1)(1) = -1, their sum is 33, and adding the bias gives z=3.5z = 3.5. The ReLU gives a=max⁡(0,3.5)=3.5a = \max(0,\allowbreak 3.5) = 3.5. The sigmoid gives a=1/(1+e−3.5)=1/(1+0.0302)≈0.971a = 1/(1 + e^{-3.5}) = 1/(1 + 0.0302) \approx 0.971.
  • For x=(1,3)\mathbf{x} = (1,\allowbreak 3): the products are (2)(1)=2(2)(1) = 2 and (−1)(3)=−3(-1)(3) = -3, their sum is −1-1, and z=−1+0.5=−0.5z = -1 + 0.5 = -0.5. The ReLU gives a=max⁡(0,−0.5)=0a = \max(0,\allowbreak -0.5) = 0: the neuron is off for this input. The sigmoid gives 1/(1+e0.5)=1/(1+1.6487)≈0.3781/(1 + e^{0.5}) = 1/(1 + 1.6487) \approx 0.378.

The line where the neuron switches

The inputs where z=0z = 0 separate the inputs where the neuron says "more yes" (z>0z > 0) from those where it says "more no" (z<0z < 0). That set is the neuron's decision boundary, and the dot product tells us exactly what it looks like. Five short steps:

  1. Shadows. The shadow of an input x\mathbf{x} on the line through w\mathbf{w} has signed length w⋅x/∣w∣\mathbf{w}\cdot\mathbf{x}/|\mathbf{w}| (the dot product lesson).
  2. Every boundary point casts the same shadow. On the boundary, z=0z = 0, so w⋅x=−b\mathbf{w}\cdot\mathbf{x} = -b. Dividing by ∣w∣|\mathbf{w}|, every boundary point has shadow −b/∣w∣-b/|\mathbf{w}|.
  3. Equal shadows make a line. Sliding a point at right angles to w\mathbf{w} leaves its shadow unchanged, while sliding it along w\mathbf{w} changes it. So the points that share one shadow form a straight line perpendicular to w\mathbf{w} (a plane in three dimensions, a hyperplane in more), and the boundary is that line.
  4. Which side is positive. Moving x\mathbf{x} in the direction of w\mathbf{w} increases w⋅x\mathbf{w}\cdot\mathbf{x}, so z>0z > 0 on the side that w\mathbf{w} points toward.
  5. How far. An input's distance past the boundary, measured along w\mathbf{w}, is its shadow minus the boundary's shadow: w⋅x∣w∣−−b∣w∣=w⋅x+b∣w∣=z∣w∣\frac{\mathbf{w}\cdot\mathbf{x}}{|\mathbf{w}|} - \frac{-b}{|\mathbf{w}|} = \frac{\mathbf{w}\cdot\mathbf{x} + b}{|\mathbf{w}|} = \frac{z}{|\mathbf{w}|}. So zz is the signed distance from the boundary, scaled by ∣w∣|\mathbf{w}|.

In the example, ∣w∣=22+(−1)2=5≈2.236|\mathbf{w}| = \sqrt{2^2 + (-1)^2} = \sqrt{5} \approx 2.236. The input (2,1)(2,\allowbreak 1) has z=3.5z = 3.5, so it sits 3.5/2.236≈1.573.5/2.236 \approx 1.57 units on the positive side. The input (1,3)(1,\allowbreak 3) has z=−0.5z = -0.5, so it sits 0.5/2.236≈0.220.5/2.236 \approx 0.22 units on the negative side.

A sigmoid neuron therefore says: "the farther you are on the positive side of my line, the closer my output is to 1". Read the output as the probability of class 1 and you have logistic regression, the classic linear classifier.

The widget below is that classifier. It answers one question: how do w1w_1, w2w_2 and bb place the line? The big square is the input plane, x1x_1 across and x2x_2 up, with 200 points: blue ones labelled class 0 and coral ones labelled class 1. The background shows the neuron's output p=σ(w1x1+w2x2+b)p = \sigma(w_1x_1 + w_2x_2 + b) at every spot, coral where p>0.5p > 0.5 and blue where p<0.5p < 0.5, and the white line is the boundary p=0.5p = 0.5, which is where z=0z = 0. The arrow w\mathbf{w} starts on the line and points to the side where zz grows. Under the plot, a live note says what your last action did.

Step 4 of the walkthrough is worth checking with numbers of your own before moving on.

Quick checkScale a neuron's weights

A sigmoid neuron has w=(1,−1)\mathbf{w} = (1,\allowbreak -1) and b=0.5b = 0.5, so it computes σ(x1−x2+0.5)\sigma(x_1 - x_2 + 0.5). You multiply w\mathbf{w} and bb by 10. What happens?

Choose one answer, then check.

To recap: a neuron is a dot product plus a bias, passed through a fixed bend. The boundary where it switches is a line perpendicular to w\mathbf{w}, and z/∣w∣z/|\mathbf{w}| says how far an input is from it. One neuron can only ever draw one straight line.

A layer is a matrix

One line is rarely enough, so networks use many neurons at once. Writing them as a matrix turns "run every neuron" into one matrix multiply, which is how every framework computes them.

Put mm neurons side by side, all reading the same input x\mathbf{x} with nn entries. Neuron jj has its own weights wj\mathbf{w}_j and bias bjb_j. Stack the weight vectors as the rows of an m×nm \times n matrix W\mathbf{W}, and the biases into a vector b\mathbf{b} with mm entries. The row view of a matrix times a vector from module 2 says that entry jj of Wx\mathbf{W}\mathbf{x} is row jj dotted with x\mathbf{x}, which is exactly neuron jj's weighted sum. So the whole layer is

z=Wx+b,h=ϕ(z),\mathbf{z} = \mathbf{W}\mathbf{x} + \mathbf{b}, \qquad \mathbf{h} = \phi(\mathbf{z}),

where ϕ\phi is applied to each entry separately. A layer is the transformation of module 2, a matrix acting on a vector, followed by a bend applied entry by entry.

The diagram below builds one layer of two neurons reading three inputs, one neuron at a time. On the left is the network drawn as circles and lines, with one weight per line; on the right are the same numbers as a matrix, a vector and a bias. Press Next to move through the five steps; the lime marks what each step computes.

That last step is the bridge to the code you will write, and the next section uses it.

A network is layers in sequence

A single layer still draws only straight lines, one per neuron. A network feeds one layer's outputs into the next layer, so later layers can combine what earlier ones computed. A network with one hidden layer computes

h=ϕ(W1x+b1),y^=W2h+b2.\mathbf{h} = \phi(\mathbf{W}_1\mathbf{x} + \mathbf{b}_1), \qquad \hat{\mathbf{y}} = \mathbf{W}_2\mathbf{h} + \mathbf{b}_2 .

The subscript says which layer a matrix belongs to, and y^\hat{\mathbf{y}} ("y hat") is the network's output, its prediction. The layer in the middle is called hidden because nobody tells it what its outputs should be; training decides. The last layer usually has no activation, or a special one chosen for the task (softmax and cross-entropy). We describe a network by its layer sizes: "2-2-1" means 2 inputs, 2 hidden units and 1 output. As a function, the network is the composition f2∘f1f_2 \circ f_1 of its layers: first f1f_1, then f2f_2.

A worked 2-2-1 example with ReLU. Let x=(2,1)\mathbf{x} = (2,\allowbreak 1),

W1=(1−1−21),b1=(01),w2=(2,−3),b2=1.\mathbf{W}_1 = \begin{pmatrix} 1 & -1 \\ -2 & 1 \end{pmatrix}, \quad \mathbf{b}_1 = \begin{pmatrix} 0 \\ 1 \end{pmatrix}, \quad \mathbf{w}_2 = (2, -3), \quad b_2 = 1 .

With a single output, W2\mathbf{W}_2 has one row, written here as the vector w2\mathbf{w}_2.

  1. Hidden pre-activations, one row of W1\mathbf{W}_1 at a time: z1=(1)(2)+(−1)(1)+0=2−1+0=1z_1 = (1)(2) + (-1)(1) + 0 = 2 - 1 + 0 = 1 and z2=(−2)(2)+(1)(1)+1=−4+1+1=−2z_2 = (-2)(2) + (1)(1) + 1 = -4 + 1 + 1 = -2.
  2. ReLU, entry by entry: h=(max⁡(0,1),max⁡(0,−2))=(1,0)\mathbf{h} = (\max(0,\allowbreak 1),\allowbreak \max(0,\allowbreak -2)) = (1,\allowbreak 0). The second hidden unit is off.
  3. Output: y^=w2⋅h+b2=(2)(1)+(−3)(0)+1=3\hat{y} = \mathbf{w}_2\cdot\mathbf{h} + b_2 = (2)(1) + (-3)(0) + 1 = 3.

In code, a batch is a matrix with one example per row. As shapes and batches explained and step 5 of the diagram showed, code stacks NN inputs as the rows of X\mathbf{X}, shape (N,n)(N,\allowbreak n), and stores each layer's weights with shape (inputs, outputs), which is the transpose of the column-convention W\mathbf{W}. The layer becomes

H=ϕ(XW+b),\mathbf{H} = \phi(\mathbf{X}\mathbf{W} + \mathbf{b}),

with the bias row added to every row of XW\mathbf{X}\mathbf{W} by broadcasting. The numbers do not change. In the example above, the stored matrix is the transpose (1−2−11)\begin{pmatrix} 1 & -2 \\ -1 & 1 \end{pmatrix}. The row (2,1)(2,\allowbreak 1) times it is ((2)(1)+(1)(−1), (2)(−2)+(1)(1))=(1,−3)\big((2)(1) + (1)(-1),\allowbreak \ (2)(-2) + (1)(1)\big) = (1,\allowbreak -3), and adding the bias row (0,1)(0,\allowbreak 1) gives (1,−2)(1,\allowbreak -2): the same pre-activations as step 1. From here on, formulas written with X\mathbf{X} use the row convention and formulas written with x\mathbf{x} use the column convention.

Counting parameters. A layer from nn inputs to mm outputs has nmnm weights (one per pair of an input and an output, whichever way round the matrix is stored) and mm biases, so nm+mnm + m parameters. The 2-2-1 network has (2⋅2+2)+(2⋅1+1)=6+3=9(2 \cdot 2 + 2) + (2 \cdot 1 + 1) = 6 + 3 = 9. A classic digit classifier with sizes 784-128-10 has (784⋅128+128)+(128⋅10+10)=100,480+1,290=101,770(784 \cdot 128 + 128) + (128 \cdot 10 + 10) = 100{,}480 + 1{,}290 = 101{,}770.

Now run a 2-2-1 network yourself, on paper, in both conventions.

On paperA 2-2-1 network by hand

A 2-2-1 network with ReLU hidden units computes h=ReLU⁡(W1x+b1)\mathbf{h} = \operatorname{ReLU}(\mathbf{W}_1\mathbf{x} + \mathbf{b}_1) and y^=w2⋅h+b2\hat{y} = \mathbf{w}_2\cdot\mathbf{h} + b_2, with

W1=(1−221),b1=(1−1),w2=(4,−1),b2=2.\mathbf{W}_1 = \begin{pmatrix} 1 & -2 \\ 2 & 1 \end{pmatrix}, \quad \mathbf{b}_1 = \begin{pmatrix} 1 \\ -1 \end{pmatrix}, \quad \mathbf{w}_2 = (4, -1), \quad b_2 = 2 .
  1. For x=(1,2)\mathbf{x} = (1,\allowbreak 2), compute the hidden pre-activations z\mathbf{z} and the hidden activations h\mathbf{h}.
  2. Compute y^\hat{y}, and the probability σ(y^)\sigma(\hat{y}) that a sigmoid output unit would report.
  3. Redo the hidden layer in the row convention of code: write the stored weight matrix, of shape (inputs, units), multiply the row (1,2)(1,\allowbreak 2) by it, and add the bias row. Check that you get the same z\mathbf{z}.
  4. Which weights could you change a little without changing y^\hat{y} at all for this input? Why?

To check parts 1 and 2 before reading the steps, enter h1h_1, h2h_2 and y^\hat{y} as a column, top to bottom.

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

h1, h2 and ŷ

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

The same row-by-row recipe works for a layer of any width. This one has three units and three inputs.

Work it outOne layer of three units

A layer of three ReLU units reads x=(1,−2,3)\mathbf{x} = (1,\allowbreak -2,\allowbreak 3) with

W=(10121−10−11),b=(−102).\mathbf{W} = \begin{pmatrix} 1 & 0 & 1 \\ 2 & 1 & -1 \\ 0 & -1 & 1 \end{pmatrix}, \qquad \mathbf{b} = \begin{pmatrix} -1 \\ 0 \\ 2 \end{pmatrix}.

Compute h=ReLU⁡(Wx+b)\mathbf{h} = \operatorname{ReLU}(\mathbf{W}\mathbf{x} + \mathbf{b}). Enter h\mathbf{h} as a column of three whole numbers, h1h_1 at the top.

h

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

A few more forward passes, with fresh numbers each time, until the recipe is automatic.

Practice setForward pass reps
3 correct in a row completes the set. A miss starts the count again.

A network has input x=[−2−2]\mathbf{x} = \begin{bmatrix}-2 \\ -2\end{bmatrix} and a hidden layer h=ReLU⁡(W1x+b1)\mathbf{h} = \operatorname{ReLU}(\mathbf{W}_1\mathbf{x} + \mathbf{b}_1) with W1=[0201−20]\mathbf{W}_1 = \begin{bmatrix}0 & 2 \\ 0 & 1 \\ -2 & 0\end{bmatrix} and b1=[−12−1]\mathbf{b}_1 = \begin{bmatrix}-1 \\ 2 \\ -1\end{bmatrix}. The output is a probability: p^=σ(w2⋅h+b2)\hat{p} = \sigma(\mathbf{w}_2\cdot\mathbf{h} + b_2) with w2=(−1,−1,−1)\mathbf{w}_2 = (-1,\allowbreak -1,\allowbreak -1) and b2=1b_2 = 1, where σ(z)=1/(1+e−z)\sigma(z) = 1/(1 + e^{-z}). Compute p^\hat{p} to 3 decimal places.

p̂

Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.

To recap: a layer is a matrix multiply, a bias and an entrywise bend; a network is layers in sequence; and code runs a whole batch at once as XW+b\mathbf{X}\mathbf{W} + \mathbf{b}, with the same numbers as the column form.

Stacking linear layers gains nothing

Why does every neuron carry the bend ϕ\phi at all? Leave it out and see what happens to depth.

Start with numbers. A first "layer" computes h=2x+1h = 2x + 1 and a second computes y=3h−4y = 3h - 4, with no activation in between. Substitute the first into the second:

y=3(2x+1)−4=6x+3−4=6x−1.y = 3(2x + 1) - 4 = 6x + 3 - 4 = 6x - 1 .

Two layers, and the result is one layer with weight 6 and bias −1-1. The same happens with matrices, one move per line:

W2(W1x+b1)+b2=W2(W1x)+W2b1+b2distribute W2 over the sum=(W2W1) x+W2b1+b2associativity=(W2W1) x+(W2b1+b2)group the terms without x\begin{aligned} \mathbf{W}_2(\mathbf{W}_1\mathbf{x} + \mathbf{b}_1) + \mathbf{b}_2 &= \mathbf{W}_2(\mathbf{W}_1\mathbf{x}) + \mathbf{W}_2\mathbf{b}_1 + \mathbf{b}_2 && \text{distribute } \mathbf{W}_2 \text{ over the sum} \\ &= (\mathbf{W}_2\mathbf{W}_1)\,\mathbf{x} + \mathbf{W}_2\mathbf{b}_1 + \mathbf{b}_2 && \text{associativity} \\ &= (\mathbf{W}_2\mathbf{W}_1)\,\mathbf{x} + (\mathbf{W}_2\mathbf{b}_1 + \mathbf{b}_2) && \text{group the terms without } \mathbf{x} \end{aligned}

The distributive law and associativity both come from matrix multiplication. The right side is a single layer with weight matrix A=W2W1\mathbf{A} = \mathbf{W}_2\mathbf{W}_1 and bias c=W2b1+b2\mathbf{c} = \mathbf{W}_2\mathbf{b}_1 + \mathbf{b}_2. Apply the same step to that layer and the next one, and again, and a stack of any depth collapses into one affine map Ax+c\mathbf{A}\mathbf{x} + \mathbf{c}: a matrix product plus a constant vector. Its decision boundaries are still straight lines, so a thousand linear layers cannot split the XOR data from step 6 of the walkthrough either.

Stacking linear layers can even lose power. A linear 100-5-100 network computes W2W1\mathbf{W}_2\mathbf{W}_1, a 100×100100 \times 100 matrix of rank at most 5, because every output has passed through only 5 numbers. It cannot represent most linear maps from 100 numbers to 100.

Test the collapse on a network that looks impressive.

Quick checkDeep but linear

To keep things simple, a colleague builds a network with layer sizes 100-512-512-512-10 and no activation functions anywhere. What can it compute?

Choose one answer, then check.

So the bend ϕ\phi is what makes depth worth having. The next lesson looks at the bends in common use, watches a hidden layer use one to solve the XOR puzzle, and works out how to choose the starting weights.