Module 6 · 6 lessons · about 355 minutes
Neural networks from scratch
Neurons and layers as matrix operations, softmax and cross-entropy, backpropagation derived and implemented, and a full training loop you write yourself.
- Lessons
- 0/6
- Exercises
- 0/26
- Code
- 0/5
- Checkpoint
- Not taken
Already comfortable with this? Take the checkpoint cold. If you pass, move on and come back only for what you missed.

- 1Neurons and layersA neuron is a dot product, a bias and a bend. A layer is many neurons reading the same input, which is one matrix multiply, and a network is layers applied one after another. Without the bend, a stack of layers of any depth is still a single layer.45 min
- 2Activations and initializationA hidden layer with a nonlinear activation bends the input space until a straight line can separate what no line could before. This lesson looks at the four activations you will meet, watches a small network bend the XOR puzzle apart, and works out how large the starting weights should be.55 min
- 3Softmax and cross-entropyA network's last layer produces raw scores. Softmax turns them into probabilities, cross-entropy charges the model for the probability it gave the right answer, and the gradient of the pair is the simplest formula in deep learning, p minus y.75 min
- 4BackpropagationBackpropagation computes the gradient of the loss with respect to every parameter in one backward sweep, by passing a single gradient vector from the loss back through the network and multiplying it by each step's local derivatives. This lesson derives it for real layers with batches and builds the layers yourself.85 min
- 5Training a networkThe training loop is five moves repeated thousands of times. This lesson follows one cycle of it number by number, watches a small network learn neuron by neuron, has you write the loop and train on data no straight line can split, and teaches you to read what the loss curves say.55 min
- 6Momentum and AdamPlain gradient descent has no memory and one step size for every parameter. Momentum adds a memory of recent gradients, which speeds up steady directions and damps oscillations. Adam also gives every parameter its own step size. This lesson derives both, bias correction included, and has you implement them.40 min
- Checkpoint12 questions. Pass with 80% to finish the module.
Practice sets for this module
Fresh problems every time. Use them until the mechanics feel automatic.
A network has input and a hidden layer with and . The output is a probability: with and , where . Compute to 3 decimal places.
Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.
Compute for the logits . Give each probability to 3 decimal places.
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
A linear layer computes for a batch, one example per row. Backprop has delivered . With and , find (the gradient passed back to the previous layer).
One entry per box, row by row (Tab moves along a row). 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.