A neuron is a dot product, a bias and a bend. A layer is many neurons reading the same input, which is one matrix multiply, and a network is layers applied one after another. Without the bend, a stack of layers of any depth is still a single layer.
About 45 minutes
By the end you can
Compute a neuron's output and read its decision boundary, the line where z=0, perpendicular to w, with every input at signed distance z/∣w∣ from it.
Compute a layer and a small network by hand, in both the column convention and the row-batch convention used in code.
Count the parameters of a network from its layer sizes.
Prove that layers without a nonlinearity collapse into a single affine map.
In the dot product lesson, a movie's score was your taste vector dotted with the movie's features. Add a threshold ("recommend it if the score beats 1"), which is the same as subtracting 1 from the score and asking whether the result is positive. Pass that result through a function that turns it into a number between 0 and 1, and you have built a neuron.
Put a few neurons side by side, all reading the same input, and you have a layer. Feed one layer's outputs into another layer and you have a neural network. Modules 1 to 5 built every piece of mathematics this needs. This lesson assembles the pieces: one neuron and the line it draws, a layer as one matrix multiply, a network as layers in sequence, and a short proof of why the bend inside each neuron cannot be left out. The next lesson shows what that bend buys.
Everything in this module is built from one small unit, so it pays to know it exactly. A neuron takes an input vector x=(x1,…,xn) and computes two numbers:
z=w⋅x+b,a=ϕ(z).
Here is what each symbol means.
w=(w1,…,wn) is the neuron's weight vector, one weight per input.
b is its bias, a single number added at the end. In the movie example, b=−1 plays the threshold.
z is the pre-activation: the weighted sum plus the bias.
ϕ (the Greek letter phi) is the activation function, a fixed nonlinear function applied to z.
a is the neuron's output, its activation.
The weights and the bias are learned from data; ϕ is chosen in advance. Two common choices of ϕ are the ReLU, ReLU(z)=max(0,z), which passes positive values and replaces negative ones by 0, and the sigmoid σ(z)=1/(1+e−z) from the derivatives lesson, which squashes any number into the interval (0,1).
A worked example. Take w=(2,−1) and b=0.5.
For x=(2,1): the products are (2)(2)=4 and (−1)(1)=−1, their sum is 3, and adding the bias gives z=3.5. The ReLU gives a=max(0,3.5)=3.5. The sigmoid gives a=1/(1+e−3.5)=1/(1+0.0302)≈0.971.
For x=(1,3): the products are (2)(1)=2 and (−1)(3)=−3, their sum is −1, and z=−1+0.5=−0.5. The ReLU gives a=max(0,−0.5)=0: the neuron is off for this input. The sigmoid gives 1/(1+e0.5)=1/(1+1.6487)≈0.378.
The line where the neuron switches
The inputs where z=0 separate the inputs where the neuron says "more yes" (z>0) from those where it says "more no" (z<0). That set is the neuron's decision boundary, and the dot product tells us exactly what it looks like. Five short steps:
Shadows. The shadow of an input x on the line through w has signed length w⋅x/∣w∣ (the dot product lesson).
Every boundary point casts the same shadow. On the boundary, z=0, so w⋅x=−b. Dividing by ∣w∣, every boundary point has shadow −b/∣w∣.
Equal shadows make a line. Sliding a point at right angles to w leaves its shadow unchanged, while sliding it along w changes it. So the points that share one shadow form a straight line perpendicular to w (a plane in three dimensions, a hyperplane in more), and the boundary is that line.
Which side is positive. Moving x in the direction of w increases w⋅x, so z>0 on the side that w points toward.
How far. An input's distance past the boundary, measured along w, is its shadow minus the boundary's shadow: ∣w∣w⋅x−∣w∣−b=∣w∣w⋅x+b=∣w∣z. So z is the signed distance from the boundary, scaled by ∣w∣.
In the example, ∣w∣=22+(−1)2=5≈2.236. The input (2,1) has z=3.5, so it sits 3.5/2.236≈1.57 units on the positive side. The input (1,3) has z=−0.5, so it sits 0.5/2.236≈0.22 units on the negative side.
A sigmoid neuron therefore says: "the farther you are on the positive side of my line, the closer my output is to 1". Read the output as the probability of class 1 and you have logistic regression, the classic linear classifier.
The widget below is that classifier. It answers one question: how do w1, w2 and b place the line? The big square is the input plane, x1 across and x2 up, with 200 points: blue ones labelled class 0 and coral ones labelled class 1. The background shows the neuron's output p=σ(w1x1+w2x2+b) at every spot, coral where p>0.5 and blue where p<0.5, and the white line is the boundary p=0.5, which is where z=0. The arrow w starts on the line and points to the side where z grows. Under the plot, a live note says what your last action did.
Step 4 of the walkthrough is worth checking with numbers of your own before moving on.
Quick checkScale a neuron's weights
A sigmoid neuron has w=(1,−1) and b=0.5, so it computes σ(x1−x2+0.5). You multiply w and b by 10. What happens?
Choose one answer, then check.
To recap: a neuron is a dot product plus a bias, passed through a fixed bend. The boundary where it switches is a line perpendicular to w, and z/∣w∣ says how far an input is from it. One neuron can only ever draw one straight line.
One line is rarely enough, so networks use many neurons at once. Writing them as a matrix turns "run every neuron" into one matrix multiply, which is how every framework computes them.
Put m neurons side by side, all reading the same input x with n entries. Neuron j has its own weights wj and bias bj. Stack the weight vectors as the rows of an m×n matrix W, and the biases into a vector b with m entries. The row view of a matrix times a vector from module 2 says that entry j of Wx is row j dotted with x, which is exactly neuron j's weighted sum. So the whole layer is
z=Wx+b,h=ϕ(z),
where ϕ is applied to each entry separately. A layer is the transformation of module 2, a matrix acting on a vector, followed by a bend applied entry by entry.
The diagram below builds one layer of two neurons reading three inputs, one neuron at a time. On the left is the network drawn as circles and lines, with one weight per line; on the right are the same numbers as a matrix, a vector and a bias. Press Next to move through the five steps; the lime marks what each step computes.
That last step is the bridge to the code you will write, and the next section uses it.
A single layer still draws only straight lines, one per neuron. A network feeds one layer's outputs into the next layer, so later layers can combine what earlier ones computed. A network with one hidden layer computes
h=ϕ(W1x+b1),y^=W2h+b2.
The subscript says which layer a matrix belongs to, and y^ ("y hat") is the network's output, its prediction. The layer in the middle is called hidden because nobody tells it what its outputs should be; training decides. The last layer usually has no activation, or a special one chosen for the task (softmax and cross-entropy). We describe a network by its layer sizes: "2-2-1" means 2 inputs, 2 hidden units and 1 output. As a function, the network is the composition f2∘f1 of its layers: first f1, then f2.
A worked 2-2-1 example with ReLU. Let x=(2,1),
W1=(1−2−11),b1=(01),w2=(2,−3),b2=1.
With a single output, W2 has one row, written here as the vector w2.
Hidden pre-activations, one row of W1 at a time: z1=(1)(2)+(−1)(1)+0=2−1+0=1 and z2=(−2)(2)+(1)(1)+1=−4+1+1=−2.
ReLU, entry by entry: h=(max(0,1),max(0,−2))=(1,0). The second hidden unit is off.
Output: y^=w2⋅h+b2=(2)(1)+(−3)(0)+1=3.
In code, a batch is a matrix with one example per row. As shapes and batches explained and step 5 of the diagram showed, code stacks N inputs as the rows of X, shape (N,n), and stores each layer's weights with shape (inputs, outputs), which is the transpose of the column-convention W. The layer becomes
H=ϕ(XW+b),
with the bias row added to every row of XW by broadcasting. The numbers do not change. In the example above, the stored matrix is the transpose (1−1−21). The row (2,1) times it is ((2)(1)+(1)(−1),(2)(−2)+(1)(1))=(1,−3), and adding the bias row (0,1) gives (1,−2): the same pre-activations as step 1. From here on, formulas written with X use the row convention and formulas written with x use the column convention.
Counting parameters. A layer from n inputs to m outputs has nm weights (one per pair of an input and an output, whichever way round the matrix is stored) and m biases, so nm+m parameters. The 2-2-1 network has (2⋅2+2)+(2⋅1+1)=6+3=9. A classic digit classifier with sizes 784-128-10 has (784⋅128+128)+(128⋅10+10)=100,480+1,290=101,770.
Now run a 2-2-1 network yourself, on paper, in both conventions.
On paperA 2-2-1 network by hand
A 2-2-1 network with ReLU hidden units computes h=ReLU(W1x+b1) and y^=w2⋅h+b2, with
W1=(12−21),b1=(1−1),w2=(4,−1),b2=2.
For x=(1,2), compute the hidden pre-activations z and the hidden activations h.
Compute y^, and the probability σ(y^) that a sigmoid output unit would report.
Redo the hidden layer in the row convention of code: write the stored weight matrix, of shape (inputs, units), multiply the row (1,2) by it, and add the bias row. Check that you get the same z.
Which weights could you change a little without changing y^ at all for this input? Why?
To check parts 1 and 2 before reading the steps, enter h1, h2 and y^ as a column, top to bottom.
Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.
h1, h2 and ŷ
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
The same row-by-row recipe works for a layer of any width. This one has three units and three inputs.
Work it outOne layer of three units
A layer of three ReLU units reads x=(1,−2,3) with
W=12001−11−11,b=−102.
Compute h=ReLU(Wx+b). Enter h as a column of three whole numbers, h1 at the top.
h
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
A few more forward passes, with fresh numbers each time, until the recipe is automatic.
Practice setForward pass reps
3 correct in a row completes the set. A miss starts the count again.
A network has input x=[−2−2] and a hidden layer h=ReLU(W1x+b1) with W1=00−2210 and b1=−12−1. The output is a probability: p^=σ(w2⋅h+b2) with w2=(−1,−1,−1) and b2=1, where σ(z)=1/(1+e−z). Compute p^ to 3 decimal places.
p̂
Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.
To recap: a layer is a matrix multiply, a bias and an entrywise bend; a network is layers in sequence; and code runs a whole batch at once as XW+b, with the same numbers as the column form.
Why does every neuron carry the bend ϕ at all? Leave it out and see what happens to depth.
Start with numbers. A first "layer" computes h=2x+1 and a second computes y=3h−4, with no activation in between. Substitute the first into the second:
y=3(2x+1)−4=6x+3−4=6x−1.
Two layers, and the result is one layer with weight 6 and bias −1. The same happens with matrices, one move per line:
W2(W1x+b1)+b2=W2(W1x)+W2b1+b2=(W2W1)x+W2b1+b2=(W2W1)x+(W2b1+b2)distribute W2 over the sumassociativitygroup the terms without x
The distributive law and associativity both come from matrix multiplication. The right side is a single layer with weight matrix A=W2W1 and bias c=W2b1+b2. Apply the same step to that layer and the next one, and again, and a stack of any depth collapses into one affine mapAx+c: a matrix product plus a constant vector. Its decision boundaries are still straight lines, so a thousand linear layers cannot split the XOR data from step 6 of the walkthrough either.
Stacking linear layers can even lose power. A linear 100-5-100 network computes W2W1, a 100×100 matrix of rank at most 5, because every output has passed through only 5 numbers. It cannot represent most linear maps from 100 numbers to 100.
Test the collapse on a network that looks impressive.
Quick checkDeep but linear
To keep things simple, a colleague builds a network with layer sizes 100-512-512-512-10 and no activation functions anywhere. What can it compute?
Choose one answer, then check.
So the bend ϕ is what makes depth worth having. The next lesson looks at the bends in common use, watches a hidden layer use one to solve the XOR puzzle, and works out how to choose the starting weights.