ml.lab
Python sleeps until you run code

Module 6 · 6 lessons · about 355 minutes

Neural networks from scratch

Neurons and layers as matrix operations, softmax and cross-entropy, backpropagation derived and implemented, and a full training loop you write yourself.

Lessons
0/6
Exercises
0/26
Code
0/5
Checkpoint
Not taken

Already comfortable with this? Take the checkpoint cold. If you pass, move on and come back only for what you missed.

  1. 1Neurons and layersA neuron is a dot product, a bias and a bend. A layer is many neurons reading the same input, which is one matrix multiply, and a network is layers applied one after another. Without the bend, a stack of layers of any depth is still a single layer.45 min
  2. 2Activations and initializationA hidden layer with a nonlinear activation bends the input space until a straight line can separate what no line could before. This lesson looks at the four activations you will meet, watches a small network bend the XOR puzzle apart, and works out how large the starting weights should be.55 min
  3. 3Softmax and cross-entropyA network's last layer produces raw scores. Softmax turns them into probabilities, cross-entropy charges the model for the probability it gave the right answer, and the gradient of the pair is the simplest formula in deep learning, p minus y.75 min
  4. 4BackpropagationBackpropagation computes the gradient of the loss with respect to every parameter in one backward sweep, by passing a single gradient vector from the loss back through the network and multiplying it by each step's local derivatives. This lesson derives it for real layers with batches and builds the layers yourself.85 min
  5. 5Training a networkThe training loop is five moves repeated thousands of times. This lesson follows one cycle of it number by number, watches a small network learn neuron by neuron, has you write the loop and train on data no straight line can split, and teaches you to read what the loss curves say.55 min
  6. 6Momentum and AdamPlain gradient descent has no memory and one step size for every parameter. Momentum adds a memory of recent gradients, which speeds up steady directions and damps oscillations. Adam also gives every parameter its own step size. This lesson derives both, bias correction included, and has you implement them.40 min
  7. Checkpoint12 questions. Pass with 80% to finish the module.

Practice sets for this module

Fresh problems every time. Use them until the mechanics feel automatic.

Practice setForward pass reps
3 correct in a row completes the set. A miss starts the count again.

A network has input x=[−2−2]\mathbf{x} = \begin{bmatrix}-2 \\ -2\end{bmatrix} and a hidden layer h=ReLU⁡(W1x+b1)\mathbf{h} = \operatorname{ReLU}(\mathbf{W}_1\mathbf{x} + \mathbf{b}_1) with W1=[0201−20]\mathbf{W}_1 = \begin{bmatrix}0 & 2 \\ 0 & 1 \\ -2 & 0\end{bmatrix} and b1=[−12−1]\mathbf{b}_1 = \begin{bmatrix}-1 \\ 2 \\ -1\end{bmatrix}. The output is a probability: p^=σ(w2⋅h+b2)\hat{p} = \sigma(\mathbf{w}_2\cdot\mathbf{h} + b_2) with w2=(−1,−1,−1)\mathbf{w}_2 = (-1,\allowbreak -1,\allowbreak -1) and b2=1b_2 = 1, where σ(z)=1/(1+e−z)\sigma(z) = 1/(1 + e^{-z}). Compute p^\hat{p} to 3 decimal places.

p̂

Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.

Practice setSoftmax and cross-entropy reps
3 correct in a row completes the set. A miss starts the count again.

Compute p=softmax⁡(z)\mathbf{p} = \operatorname{softmax}(\mathbf{z}) for the logits z=(3,1,−1)\mathbf{z} = (3,\allowbreak 1,\allowbreak -1). Give each probability to 3 decimal places.

p

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Practice setBackprop reps
3 correct in a row completes the set. A miss starts the count again.

A linear layer computes Y=XW+b\mathbf{Y} = \mathbf{X}\mathbf{W} + \mathbf{b} for a batch, one example per row. Backprop has delivered δ=∂L/∂Y\boldsymbol{\delta} = \partial L/\partial\mathbf{Y}. With W=[−11222−2]\mathbf{W} = \begin{bmatrix}-1 & 1 \\ 2 & 2 \\ 2 & -2\end{bmatrix} and δ=[0−1221−1]\boldsymbol{\delta} = \begin{bmatrix}0 & -1 \\ 2 & 2 \\ 1 & -1\end{bmatrix}, find ∂L/∂X\partial L/\partial\mathbf{X} (the gradient passed back to the previous layer).

∂L/∂X

One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Next moduleRecurrent networks