Lesson 3 of 6
Softmax and cross-entropy
A network's last layer produces raw scores. Softmax turns them into probabilities, cross-entropy charges the model for the probability it gave the right answer, and the gradient of the pair is the simplest formula in deep learning, p minus y.
Compute mean squared error and its gradient, and explain why squared error is the natural loss for predicting numbers.
Compute a softmax by hand, with temperature, and implement it without overflow using the max-subtraction and log-sum-exp tricks.
Compute cross-entropy and perplexity, and use as a sanity check for an untrained classifier.
Derive for softmax followed by cross-entropy, directly and through the softmax Jacobian.

A small language model has read the letters "th" and must guess the next one. Its last layer produces one raw score per candidate letter. Suppose it considers three, "e", "a" and "o", with scores . Those scores are called logits. They can be any real numbers, negative or positive, and they do not add up to anything in particular.
To be useful, the model has to turn the scores into a bet: a probability for each letter. Then, once we see the real next letter, we need a single number that says how good the bet was, so that gradient descent can improve it. This lesson builds both pieces: softmax turns logits into probabilities, and cross-entropy scores the bet. Together they are the loss function of every classifier in this course and of every large language model's pretraining, and their gradient turns out to be almost embarrassingly simple.
First, the easier case, where the answer is a number rather than a class.
Measuring error on numbers: mean squared error
When a network predicts a quantity (a price, a temperature, a coordinate), the usual loss is the mean squared error. For predictions with targets ,
The gradient comes from the chain rule on each square: only term of the sum contains , the derivative of with respect to is , and the mean contributes the factor .
A worked example with : predictions and targets .
- Errors : .
- Squared errors: , which add up to 5.
- Loss: .
- Gradient: times the errors, .
Gradient descent moves each prediction against its gradient: it lowers the first prediction, leaves the second alone and raises the third twice as hard.
Why squares, rather than absolute errors or fourth powers? Three reasons, in increasing depth.
- The gradient is proportional to the error. Big mistakes get big corrections and small ones get gentle nudges, and the loss is smooth everywhere, including at zero error.
- The best constant prediction is the mean. If a model must predict one number for everything, the squared error has derivative , which is zero when is the average of the . More generally, the best any model can do under squared error is to predict, for each input, the average of the targets seen with that input, so a flexible enough model trained this way aims at that average.
- It is a negative log-likelihood. If the data are the model's prediction plus Gaussian noise, minimizing squared error is the same as making the observed data as probable as possible. This last reason is the one that generalizes: the classification loss below is built the same way.
Go slower: Squared error is Gaussian maximum likelihood
Step 1, the noise model. Suppose each target is the prediction plus noise from a normal distribution with mean 0 and a fixed standard deviation : . The probability density of seeing is then
Step 2, take the negative log. . The second term does not depend on the prediction.
Step 3, many independent examples. The probability of all targets is the product of the densities, so its negative log is the sum: .
Step 4, compare. Making the data most probable means making this sum as small as possible, and a positive factor and an added constant do not change where the minimum is. So minimizing squared error is maximizing the likelihood of the data under Gaussian noise.
Try the same four moves on a batch of four.
A regression model predicts for targets , and the loss is . Find the gradient . Enter it as a column of four numbers, in the order of the predictions, as exact decimals.
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
To recap: squared error charges the square of each error, its gradient is times the error, and minimizing it is maximum likelihood under Gaussian noise. Classification needs a different loss, built on the same principle, and the first step is to turn scores into probabilities.
From scores to probabilities: softmax
Back to the three letters. Softmax turns a vector of logits into probabilities:
Here is the number of classes, is the logit of class , and the sum in the denominator runs over every class. Exponentiating makes every score positive, and dividing by the total makes them add up to 1. For :
- Exponentiate: , and .
- Add: .
- Divide each by the total: , and .
The name says what it does: it is a soft version of "argmax". The largest logit gets the largest probability, but the others keep a share. Some properties, each with its reason:
- Order is kept. is increasing, so a larger logit always means a larger probability.
- Only differences matter. Adding the same number to every logit changes nothing, because and the factor cancels between the top and the bottom. Each unit of difference between two logits multiplies the ratio of their probabilities by : here "e" is times as likely as "a", and times as likely as "o".
- Scale matters. Multiplying every logit by 2 does change the answer: the differences double, so the ratios get squared and the distribution gets sharper.
That last property is temperature. Dividing the logits by a temperature before the softmax, , controls how peaked the distribution is. For the logits become ; then , and add up to , and dividing gives the first row of the table.
As shrinks toward 0 the distribution approaches all of its mass on the largest logit, which is plain argmax. As grows, it approaches uniform.
The widget below is this example, live. It answers two questions: how does one logit change every probability, and what does temperature do? The top row has one blue bar per letter, its logit, which you can drag up or down (or focus and move with the arrow keys; Shift moves by 1). Below them the white bars are the probabilities, which always add up to 1, and the dashed lime box marks the correct letter, "e". The readouts give the cross-entropy loss (the next section's subject), the probability of "e", the temperature and the entropy, a measure of how spread out is. Under the bars, a live note says what your last action did.
Now compute a softmax by hand, with the shift that makes it easy.
Compute for . Subtract the largest logit first: it makes the arithmetic easier as well as safer. Enter as a column of four decimals, in the order of the logits, to 3 decimal places.
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
And predict what temperature does before computing it.
A model's logits for the next token are . At temperature 1 the top token has probability about . You sample at temperature instead. What happens?
Choose one answer, then check.
Softmax without overflow
The formula has a practical trap. A 64-bit float can hold numbers up to about , which is . Logits of 1,000 are unusual but not absurd, and overflows to infinity. Then the softmax computes , which is "not a number" (nan), and the nan spreads through every later calculation. In 32-bit floats, common on GPUs, the limit is .
The fix uses the fact that only differences matter. Subtract the largest logit from every logit first:
The answer is identical, but now every exponent is at most 0, so nothing overflows, and the largest term is exactly , so the denominator is at least 1 and never zero. For , subtract 1002 to get , and the answer is .
There is a second trap at the other end. The loss below needs , and a tiny probability can underflow: for , the true is about , far below the smallest positive float (about ), so it rounds to 0 and . The cure is to never form the probability. Take logs of the softmax formula:
where is the log-sum-exp function. It is computed stably by pulling the maximum out:
For this gives to machine precision, so exactly, a perfectly usable number. The function that returns is called log-softmax, and every framework has one.
The naive version fails on the large logits; the stable version survives them; and only log-softmax gets right in the last case, where even the stable softmax has rounded to zero.
To recap: subtract the largest logit before exponentiating, which changes nothing but keeps every exponent at or below 0; and when you need a log-probability, compute directly.
Cross-entropy: the price of a bet
Now score the bet. Say the correct class is , an index from 1 to (code counts classes from 0, as numpy indexes arrays). The cross-entropy loss charges the model the negative log of the probability it gave to that class:
With : if the next letter really is "e", the loss is . If it is "a", the loss is , and if it is "o", . (They differ by exactly 1 because and the logits differ by 1.) The loss depends only on the probability of the correct class, and it has the right shape for a penalty:
- It is 0 when the model gave the right answer probability 1, and it grows without limit as that probability goes to 0.
- Every halving of adds . A model that says "1% chance" for the right answer pays ; one that says "one in a million" pays about . Confident mistakes are expensive.
Why "cross-entropy". In information theory, the cross-entropy between a true distribution and a predicted one is . Our target is the one-hot vector , with a 1 in position and 0 elsewhere, so every term but one vanishes and . (Some training recipes use a softened target instead of a one-hot one; the general formula covers that.)
Why the log: it is maximum likelihood again. For training examples the loss is the average, . A log turns a product into a sum, so this equals : minimizing the mean cross-entropy is the same as making the model assign the highest possible probability to all the training labels together. Squared error came from the same principle with a Gaussian model; cross-entropy comes from it with a categorical one. The big picture made the same observation about the tokens of a text.
Two numbers to keep in your head.
- The uniform guess costs . An untrained classifier with small random weights outputs nearly equal logits, hence and a loss near : , . If the first loss your training loop prints is far from , something is wrong with the initialization or the loss code, before any learning has happened.
- Perplexity is . It converts a loss back into "the model is as uncertain as a uniform choice among this many options". The uniform guess has perplexity . A loss of has perplexity . For an average over many examples, the perplexity is the geometric mean of . Language models are often compared by perplexity, and you will compute it for your own in module 7.
One number to compute before moving on: the loss of a batch of two, one of them a confident mistake.
A classifier sees two examples. It gives the correct class probability on the first and on the second. What is the mean cross-entropy loss over the two, in nats (natural log)? Give it to 3 decimal places.
Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.
The gradient is p minus y
To train, we need the derivative of the loss with respect to each logit. Before deriving it, see the whole computation for our example in one place. The diagram below is a ledger: one column per letter, one row per stage, from the logits at the top to the gradient at the bottom. The row computed in the current step is outlined in lime; earlier rows are dimmed, and later rows wait as empty dashed cells. Press Next to fill in one row at a time.
In general, for any logits and any correct class, the gradient is
where is the one-hot target: predicted probabilities minus true ones. For our letters with target "e", and
The derivation is short from the form . The derivative of with respect to is (the softmax is the gradient of log-sum-exp), and the derivative of with respect to is 1 if and 0 otherwise, which is . Subtract and you have . Both routes, in full:
Go slower: Route 1: directly
Step 1, write the loss with logits only. Write for the denominator, so . The log of a quotient is the log of the top minus the log of the bottom, so . Since , this is .
Step 2, differentiate the first term. if and otherwise. With the one-hot vector, that is .
Step 3, differentiate the second term. By the chain rule, . Only the term of involves , so , and .
Step 4, add. .
Go slower: Route 2: through the softmax Jacobian
Jacobians and the chain rule derived the softmax Jacobian , where is 1 when and 0 otherwise.
Step 1, the loss as a function of the probabilities. , so .
Step 2, chain rule over every path from to . affects every , so
Step 3, cancel the . .
Step 4, simplify each sum. , because only survives. And , because the target is a probability distribution. So .
In the vector-Jacobian language of module 5: the upstream gradient is (entry by entry), and the softmax VJP gives . The division by and the Jacobian cancel exactly, which is why frameworks fuse the two into one "softmax cross-entropy" operation that never divides by a probability.
Read the result as an instruction to gradient descent, which moves each logit against its gradient:
- The correct class's logit has gradient , so it goes up, by more when the model was less sure of it.
- Every other logit has gradient , so it goes down, in proportion to the probability wrongly given to it.
- The entries sum to : the gradient only redistributes belief. Here .
- Every entry lies between and , and the gradient is largest exactly when the model is confidently wrong. It never saturates.
That last point is why classifiers use cross-entropy rather than squared error on probabilities. With a sigmoid output , cross-entropy again gives (the two-class version of the same result). Squared error instead gives . When the model is confidently wrong, say with , cross-entropy's gradient is , while squared error's is , fifty times smaller. The sigmoid's flat tail has swallowed the signal exactly when it was needed most.
For a batch, the loss is a mean over examples, so each example's gradient is divided by : row of is .
The same result falls out of a backward pass through the graph of the computation, one node at a time, with no algebra at all. The widget below draws softmax and cross-entropy for two classes as a graph: the logits and on the left, their exponentials and , the sum , the probability of the correct class, and the loss on the right. White numbers are values; amber numbers are gradients. The step counter and the arrow buttons move through the forward pass and then the backward pass, and the panel beside the graph writes out each step's formula with its numbers.
Now derive yourself, and take one gradient step with it.
A 3-class classifier outputs logits , and the correct class is class 2.
- Write the loss using only the logits.
- Differentiate it with respect to , and , and write the result as .
- Evaluate the gradient for .
- Take one gradient descent step with learning rate . What is the new probability of class 2, and the new loss?
To check part 3 before reading the steps, enter the gradient as a column of three decimals, class 1 at the top.
Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Now implement the whole thing, stably, and check the gradient numerically. This is the loss you will use for every classifier and language model in the rest of Part 1.
Implement the classification loss that the rest of Part 1 uses, stably, and a tool to check any gradient. Z holds logits with one row per example, shape (N, K), and y holds integer class ids.
softmax(Z), row by row, without overflow.log_softmax(Z)as per row, never taking the log of a probability.cross_entropy(Z, y): the mean over rows of .cross_entropy_grad(Z, y): .gradient_check(f, analytic, Z): the largest relative error betweenanalyticand a central-difference gradient offatZ.
raise NotImplementedError, then press Run tests. Each check says what it expects.A few more softmax and cross-entropy computations, with fresh numbers each time.
Compute for the logits . Give each probability to 3 decimal places.
One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.
Finally, explain why classifiers use this loss at all.
A teammate proposes training a 10-class image classifier by applying softmax and then minimizing the squared error between the probabilities and the one-hot label. Explain why cross-entropy is the standard choice instead. Cover what each loss measures, what happens to each gradient when the model is confidently wrong, and one sanity check that cross-entropy gives you for free.
Saved in this browser as you type.