ml.lab
Python sleeps until you run code
06 Neural networks from scratch

Lesson 3 of 6

Softmax and cross-entropy

A network's last layer produces raw scores. Softmax turns them into probabilities, cross-entropy charges the model for the probability it gave the right answer, and the gradient of the pair is the simplest formula in deep learning, p minus y.

About 75 minutes
By the end you can
  • Compute mean squared error and its gradient, and explain why squared error is the natural loss for predicting numbers.

  • Compute a softmax by hand, with temperature, and implement it without overflow using the max-subtraction and log-sum-exp tricks.

  • Compute cross-entropy and perplexity, and use ln⁡K\ln K as a sanity check for an untrained classifier.

  • Derive ∂L/∂z=p−y\partial L/\partial\mathbf{z} = \mathbf{p} - \mathbf{y} for softmax followed by cross-entropy, directly and through the softmax Jacobian.

A small language model has read the letters "th" and must guess the next one. Its last layer produces one raw score per candidate letter. Suppose it considers three, "e", "a" and "o", with scores z=(2,1,0)\mathbf{z} = (2,\allowbreak 1,\allowbreak 0). Those scores are called logits. They can be any real numbers, negative or positive, and they do not add up to anything in particular.

To be useful, the model has to turn the scores into a bet: a probability for each letter. Then, once we see the real next letter, we need a single number that says how good the bet was, so that gradient descent can improve it. This lesson builds both pieces: softmax turns logits into probabilities, and cross-entropy scores the bet. Together they are the loss function of every classifier in this course and of every large language model's pretraining, and their gradient turns out to be almost embarrassingly simple.

First, the easier case, where the answer is a number rather than a class.

Measuring error on numbers: mean squared error

When a network predicts a quantity (a price, a temperature, a coordinate), the usual loss is the mean squared error. For NN predictions y^n\hat{y}_n with targets yny_n,

L=1N∑n=1N(y^n−yn)2,∂L∂y^n=2N(y^n−yn).L = \frac{1}{N}\sum_{n=1}^{N} (\hat{y}_n - y_n)^2, \qquad \frac{\partial L}{\partial \hat{y}_n} = \frac{2}{N}(\hat{y}_n - y_n).

The gradient comes from the chain rule on each square: only term nn of the sum contains y^n\hat{y}_n, the derivative of (y^n−yn)2(\hat{y}_n - y_n)^2 with respect to y^n\hat{y}_n is 2(y^n−yn)2(\hat{y}_n - y_n), and the mean contributes the factor 1N\frac{1}{N}.

A worked example with N=3N = 3: predictions (3,1,4)(3,\allowbreak 1,\allowbreak 4) and targets (2,1,6)(2,\allowbreak 1,\allowbreak 6).

  1. Errors y^n−yn\hat{y}_n - y_n: (3−2, 1−1, 4−6)=(1,0,−2)(3 - 2,\allowbreak \ 1 - 1,\allowbreak \ 4 - 6) = (1,\allowbreak 0,\allowbreak -2).
  2. Squared errors: (1,0,4)(1,\allowbreak 0,\allowbreak 4), which add up to 5.
  3. Loss: L=5/3≈1.667L = 5/3 \approx 1.667.
  4. Gradient: 2N=23\frac{2}{N} = \frac{2}{3} times the errors, 23(1,0,−2)≈(0.667,0,−1.333)\frac{2}{3}(1,\allowbreak 0,\allowbreak -2) \approx (0.667,\allowbreak 0,\allowbreak -1.333).

Gradient descent moves each prediction against its gradient: it lowers the first prediction, leaves the second alone and raises the third twice as hard.

Why squares, rather than absolute errors or fourth powers? Three reasons, in increasing depth.

  • The gradient is proportional to the error. Big mistakes get big corrections and small ones get gentle nudges, and the loss is smooth everywhere, including at zero error.
  • The best constant prediction is the mean. If a model must predict one number cc for everything, the squared error ∑n(c−yn)2\sum_n (c - y_n)^2 has derivative 2∑n(c−yn)2\sum_n (c - y_n), which is zero when cc is the average of the yny_n. More generally, the best any model can do under squared error is to predict, for each input, the average of the targets seen with that input, so a flexible enough model trained this way aims at that average.
  • It is a negative log-likelihood. If the data are the model's prediction plus Gaussian noise, minimizing squared error is the same as making the observed data as probable as possible. This last reason is the one that generalizes: the classification loss below is built the same way.
Go slower: Squared error is Gaussian maximum likelihood

Step 1, the noise model. Suppose each target is the prediction plus noise from a normal distribution with mean 0 and a fixed standard deviation ss: y=y^+ϵy = \hat{y} + \epsilon. The probability density of seeing yy is then p(y∣y^)=12πs2exp⁡(−(y−y^)22s2).p(y \mid \hat{y}) = \frac{1}{\sqrt{2\pi s^2}}\exp\left(-\frac{(y - \hat{y})^2}{2s^2}\right).

Step 2, take the negative log. −ln⁡p(y∣y^)=(y−y^)22s2+12ln⁡(2πs2)-\ln p(y \mid \hat{y}) = \frac{(y - \hat{y})^2}{2s^2} + \frac{1}{2}\ln(2\pi s^2). The second term does not depend on the prediction.

Step 3, many independent examples. The probability of all NN targets is the product of the densities, so its negative log is the sum: 12s2∑n(yn−y^n)2+constant\frac{1}{2s^2}\sum_n (y_n - \hat{y}_n)^2 + \text{constant}.

Step 4, compare. Making the data most probable means making this sum as small as possible, and a positive factor 12s2\frac{1}{2s^2} and an added constant do not change where the minimum is. So minimizing squared error is maximizing the likelihood of the data under Gaussian noise.

Try the same four moves on a batch of four.

Work it outThe gradient of mean squared error

A regression model predicts y^=(2,−1,0,3)\hat{\mathbf{y}} = (2,\allowbreak -1,\allowbreak 0,\allowbreak 3) for targets y=(1,1,0,1)\mathbf{y} = (1,\allowbreak 1,\allowbreak 0,\allowbreak 1), and the loss is L=14∑n=14(y^n−yn)2L = \frac{1}{4}\sum_{n=1}^{4}(\hat{y}_n - y_n)^2. Find the gradient ∂L/∂y^\partial L/\partial\hat{\mathbf{y}}. Enter it as a column of four numbers, in the order of the predictions, as exact decimals.

∂L/∂ŷ

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

To recap: squared error charges the square of each error, its gradient is 2N\frac{2}{N} times the error, and minimizing it is maximum likelihood under Gaussian noise. Classification needs a different loss, built on the same principle, and the first step is to turn scores into probabilities.

From scores to probabilities: softmax

Back to the three letters. Softmax turns a vector of KK logits into KK probabilities:

pi=softmax⁡(z)i=ezi∑j=1Kezj.p_i = \operatorname{softmax}(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}} .

Here KK is the number of classes, ziz_i is the logit of class ii, and the sum in the denominator runs over every class. Exponentiating makes every score positive, and dividing by the total makes them add up to 1. For z=(2,1,0)\mathbf{z} = (2,\allowbreak 1,\allowbreak 0):

  1. Exponentiate: e2≈7.389e^2 \approx 7.389, e1≈2.718e^1 \approx 2.718 and e0=1e^0 = 1.
  2. Add: 7.389+2.718+1=11.1077.389 + 2.718 + 1 = 11.107.
  3. Divide each by the total: 7.389/11.107≈0.6657.389/11.107 \approx 0.665, 2.718/11.107≈0.2452.718/11.107 \approx 0.245 and 1/11.107≈0.0901/11.107 \approx 0.090.
p≈(0.665, 0.245, 0.090).\mathbf{p} \approx (0.665,\ 0.245,\ 0.090).

The name says what it does: it is a soft version of "argmax". The largest logit gets the largest probability, but the others keep a share. Some properties, each with its reason:

  • Order is kept. eze^z is increasing, so a larger logit always means a larger probability.
  • Only differences matter. Adding the same number cc to every logit changes nothing, because ezi+c=ecezie^{z_i + c} = e^c e^{z_i} and the factor ece^c cancels between the top and the bottom. Each unit of difference between two logits multiplies the ratio of their probabilities by e≈2.718e \approx 2.718: here "e" is ee times as likely as "a", and e2e^2 times as likely as "o".
  • Scale matters. Multiplying every logit by 2 does change the answer: the differences double, so the ratios get squared and the distribution gets sharper.

That last property is temperature. Dividing the logits by a temperature T>0T > 0 before the softmax, p=softmax⁡(z/T)\mathbf{p} = \operatorname{softmax}(\mathbf{z}/T), controls how peaked the distribution is. For T=0.5T = 0.5 the logits become (4,2,0)(4,\allowbreak 2,\allowbreak 0); then e4≈54.598e^4 \approx 54.598, e2≈7.389e^2 \approx 7.389 and e0=1e^0 = 1 add up to 62.98762.987, and dividing gives the first row of the table.

TTz/T\mathbf{z}/Tp\mathbf{p}
0.50.5(4,2,0)(4,\allowbreak 2,\allowbreak 0)(0.867,0.117,0.016)(0.867,\allowbreak 0.117,\allowbreak 0.016)
11(2,1,0)(2,\allowbreak 1,\allowbreak 0)(0.665,0.245,0.090)(0.665,\allowbreak 0.245,\allowbreak 0.090)
22(1,0.5,0)(1,\allowbreak 0.5,\allowbreak 0)(0.506,0.307,0.186)(0.506,\allowbreak 0.307,\allowbreak 0.186)

As TT shrinks toward 0 the distribution approaches all of its mass on the largest logit, which is plain argmax. As TT grows, it approaches uniform.

The widget below is this example, live. It answers two questions: how does one logit change every probability, and what does temperature do? The top row has one blue bar per letter, its logit, which you can drag up or down (or focus and move with the arrow keys; Shift moves by 1). Below them the white bars are the probabilities, which always add up to 1, and the dashed lime box marks the correct letter, "e". The readouts give the cross-entropy loss (the next section's subject), the probability of "e", the temperature and the entropy, a measure of how spread out p\mathbf{p} is. Under the bars, a live note says what your last action did.

Now compute a softmax by hand, with the shift that makes it easy.

Work it outA softmax by hand

Compute softmax⁡(z)\operatorname{softmax}(\mathbf{z}) for z=(3,1,2,3)\mathbf{z} = (3,\allowbreak 1,\allowbreak 2,\allowbreak 3). Subtract the largest logit first: it makes the arithmetic easier as well as safer. Enter p\mathbf{p} as a column of four decimals, in the order of the logits, to 3 decimal places.

p

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

And predict what temperature does before computing it.

Quick checkTurning the temperature down

A model's logits for the next token are (4,2,1)(4,\allowbreak 2,\allowbreak 1). At temperature 1 the top token has probability about 0.8440.844. You sample at temperature T=0.5T = 0.5 instead. What happens?

Choose one answer, then check.

Softmax without overflow

The formula has a practical trap. A 64-bit float can hold numbers up to about 1.8×103081.8 \times 10^{308}, which is e709.78e^{709.78}. Logits of 1,000 are unusual but not absurd, and e1000e^{1000} overflows to infinity. Then the softmax computes ∞/∞\infty/\infty, which is "not a number" (nan), and the nan spreads through every later calculation. In 32-bit floats, common on GPUs, the limit is e88.72e^{88.72}.

The fix uses the fact that only differences matter. Subtract the largest logit m=max⁡jzjm = \max_j z_j from every logit first:

ezi−m∑jezj−m=ezi e−m∑jezj e−m=ezi∑jezj.\frac{e^{z_i - m}}{\sum_j e^{z_j - m}} = \frac{e^{z_i}\,e^{-m}}{\sum_j e^{z_j}\,e^{-m}} = \frac{e^{z_i}}{\sum_j e^{z_j}} .

The answer is identical, but now every exponent is at most 0, so nothing overflows, and the largest term is exactly e0=1e^0 = 1, so the denominator is at least 1 and never zero. For z=(1000,1001,1002)\mathbf{z} = (1000,\allowbreak 1001,\allowbreak 1002), subtract 1002 to get (−2,−1,0)(-2,\allowbreak -1,\allowbreak 0), and the answer is (0.090,0.245,0.665)(0.090,\allowbreak 0.245,\allowbreak 0.665).

There is a second trap at the other end. The loss below needs ln⁡py\ln p_y, and a tiny probability can underflow: for z=(0,−800)\mathbf{z} = (0,\allowbreak -800), the true p2=e−800/(1+e−800)p_2 = e^{-800}/(1 + e^{-800}) is about e−800e^{-800}, far below the smallest positive float (about e−744e^{-744}), so it rounds to 0 and ln⁡0=−∞\ln 0 = -\infty. The cure is to never form the probability. Take logs of the softmax formula:

ln⁡pi=zi−ln⁡∑jezj=zi−LSE⁡(z),\ln p_i = z_i - \ln\sum_j e^{z_j} = z_i - \operatorname{LSE}(\mathbf{z}),

where LSE⁡(z)=ln⁡∑jezj\operatorname{LSE}(\mathbf{z}) = \ln\sum_j e^{z_j} is the log-sum-exp function. It is computed stably by pulling the maximum out:

LSE⁡(z)=ln⁡(em∑jezj−m)=m+ln⁡∑jezj−m.\operatorname{LSE}(\mathbf{z}) = \ln\Big(e^{m}\sum_j e^{z_j - m}\Big) = m + \ln\sum_j e^{z_j - m}.

For z=(0,−800)\mathbf{z} = (0,\allowbreak -800) this gives LSE⁡=0+ln⁡(1+e−800)=0\operatorname{LSE} = 0 + \ln(1 + e^{-800}) = 0 to machine precision, so ln⁡p2=−800\ln p_2 = -800 exactly, a perfectly usable number. The function that returns z−LSE⁡(z)\mathbf{z} - \operatorname{LSE}(\mathbf{z}) is called log-softmax, and every framework has one.

⌘+Enter runs · edit freelyPython sleeps until you run code

The naive version fails on the large logits; the stable version survives them; and only log-softmax gets ln⁡p2=−800\ln p_2 = -800 right in the last case, where even the stable softmax has rounded p2p_2 to zero.

To recap: subtract the largest logit before exponentiating, which changes nothing but keeps every exponent at or below 0; and when you need a log-probability, compute zi−LSE⁡(z)z_i - \operatorname{LSE}(\mathbf{z}) directly.

Cross-entropy: the price of a bet

Now score the bet. Say the correct class is yy, an index from 1 to KK (code counts classes from 0, as numpy indexes arrays). The cross-entropy loss charges the model the negative log of the probability it gave to that class:

L=−ln⁡py.L = -\ln p_y .

With p≈(0.665,0.245,0.090)\mathbf{p} \approx (0.665,\allowbreak 0.245,\allowbreak 0.090): if the next letter really is "e", the loss is −ln⁡0.665≈0.408-\ln 0.665 \approx 0.408. If it is "a", the loss is 1.4081.408, and if it is "o", 2.4082.408. (They differ by exactly 1 because ln⁡pi=zi−LSE⁡(z)\ln p_i = z_i - \operatorname{LSE}(\mathbf{z}) and the logits differ by 1.) The loss depends only on the probability of the correct class, and it has the right shape for a penalty:

  • It is 0 when the model gave the right answer probability 1, and it grows without limit as that probability goes to 0.
  • Every halving of pyp_y adds ln⁡2≈0.693\ln 2 \approx 0.693. A model that says "1% chance" for the right answer pays ln⁡100≈4.6\ln 100 \approx 4.6; one that says "one in a million" pays about 13.813.8. Confident mistakes are expensive.

Why "cross-entropy". In information theory, the cross-entropy between a true distribution q\mathbf{q} and a predicted one p\mathbf{p} is H(q,p)=−∑kqkln⁡pkH(\mathbf{q},\allowbreak \mathbf{p}) = -\sum_k q_k \ln p_k. Our target is the one-hot vector y\mathbf{y}, with a 1 in position yy and 0 elsewhere, so every term but one vanishes and H(y,p)=−ln⁡pyH(\mathbf{y},\allowbreak \mathbf{p}) = -\ln p_y. (Some training recipes use a softened target instead of a one-hot one; the general formula covers that.)

Why the log: it is maximum likelihood again. For NN training examples the loss is the average, L=−1N∑nln⁡pynL = -\frac{1}{N}\sum_n \ln p_{y_n}. A log turns a product into a sum, so this equals −1Nln⁡∏npyn-\frac{1}{N}\ln\prod_n p_{y_n}: minimizing the mean cross-entropy is the same as making the model assign the highest possible probability to all the training labels together. Squared error came from the same principle with a Gaussian model; cross-entropy comes from it with a categorical one. The big picture made the same observation about the tokens of a text.

Two numbers to keep in your head.

  • The uniform guess costs ln⁡K\ln K. An untrained classifier with small random weights outputs nearly equal logits, hence py≈1/Kp_y \approx 1/K and a loss near ln⁡K\ln K: ln⁡3≈1.099\ln 3 \approx 1.099, ln⁡10≈2.303\ln 10 \approx 2.303. If the first loss your training loop prints is far from ln⁡K\ln K, something is wrong with the initialization or the loss code, before any learning has happened.
  • Perplexity is eLe^{L}. It converts a loss back into "the model is as uncertain as a uniform choice among this many options". The uniform guess has perplexity eln⁡K=Ke^{\ln K} = K. A loss of 0.4080.408 has perplexity e0.408≈1.50=1/0.665e^{0.408} \approx 1.50 = 1/0.665. For an average over many examples, the perplexity is the geometric mean of 1/pyn1/p_{y_n}. Language models are often compared by perplexity, and you will compute it for your own in module 7.

One number to compute before moving on: the loss of a batch of two, one of them a confident mistake.

Work it outMean cross-entropy of two examples

A classifier sees two examples. It gives the correct class probability 0.80.8 on the first and 0.050.05 on the second. What is the mean cross-entropy loss over the two, in nats (natural log)? Give it to 3 decimal places.

L

Type a number: 0.25, -2, 3/4 and sqrt(2) all work. Enter checks.

The gradient is p minus y

To train, we need the derivative of the loss with respect to each logit. Before deriving it, see the whole computation for our example in one place. The diagram below is a ledger: one column per letter, one row per stage, from the logits at the top to the gradient at the bottom. The row computed in the current step is outlined in lime; earlier rows are dimmed, and later rows wait as empty dashed cells. Press Next to fill in one row at a time.

In general, for any logits and any correct class, the gradient is

∂L∂zi=pi−yi,that is∇zL=p−y,\frac{\partial L}{\partial z_i} = p_i - y_i, \qquad\text{that is}\qquad \nabla_{\mathbf{z}} L = \mathbf{p} - \mathbf{y},

where y\mathbf{y} is the one-hot target: predicted probabilities minus true ones. For our letters with target "e", y=(1,0,0)\mathbf{y} = (1,\allowbreak 0,\allowbreak 0) and

p−y≈(0.665−1, 0.245, 0.090)=(−0.335, 0.245, 0.090).\mathbf{p} - \mathbf{y} \approx (0.665 - 1,\ 0.245,\ 0.090) = (-0.335,\ 0.245,\ 0.090).

The derivation is short from the form L=LSE⁡(z)−zyL = \operatorname{LSE}(\mathbf{z}) - z_y. The derivative of LSE⁡\operatorname{LSE} with respect to ziz_i is ezi∑jezj=pi\frac{e^{z_i}}{\sum_j e^{z_j}} = p_i (the softmax is the gradient of log-sum-exp), and the derivative of zyz_y with respect to ziz_i is 1 if i=yi = y and 0 otherwise, which is yiy_i. Subtract and you have pi−yip_i - y_i. Both routes, in full:

Go slower: Route 1: directly

Step 1, write the loss with logits only. Write S=∑j=1KezjS = \sum_{j=1}^{K} e^{z_j} for the denominator, so L=−ln⁡py=−ln⁡ezySL = -\ln p_y = -\ln\frac{e^{z_y}}{S}. The log of a quotient is the log of the top minus the log of the bottom, so L=−(ln⁡ezy−ln⁡S)L = -(\ln e^{z_y} - \ln S). Since ln⁡ezy=zy\ln e^{z_y} = z_y, this is L=−zy+ln⁡SL = -z_y + \ln S.

Step 2, differentiate the first term. ∂(−zy)∂zi=−1\frac{\partial(-z_y)}{\partial z_i} = -1 if i=yi = y and 00 otherwise. With the one-hot vector, that is −yi-y_i.

Step 3, differentiate the second term. By the chain rule, ∂ln⁡S∂zi=1S∂S∂zi\frac{\partial \ln S}{\partial z_i} = \frac{1}{S}\frac{\partial S}{\partial z_i}. Only the term ezie^{z_i} of SS involves ziz_i, so ∂S∂zi=ezi\frac{\partial S}{\partial z_i} = e^{z_i}, and ∂ln⁡S∂zi=eziS=pi\frac{\partial \ln S}{\partial z_i} = \frac{e^{z_i}}{S} = p_i.

Step 4, add. ∂L∂zi=pi−yi\frac{\partial L}{\partial z_i} = p_i - y_i.

Go slower: Route 2: through the softmax Jacobian

Jacobians and the chain rule derived the softmax Jacobian ∂pk∂zi=pk(δki−pi)\frac{\partial p_k}{\partial z_i} = p_k(\delta_{ki} - p_i), where δki\delta_{ki} is 1 when k=ik = i and 0 otherwise.

Step 1, the loss as a function of the probabilities. L=−∑kykln⁡pkL = -\sum_k y_k \ln p_k, so ∂L∂pk=−ykpk\frac{\partial L}{\partial p_k} = -\frac{y_k}{p_k}.

Step 2, chain rule over every path from ziz_i to LL. ziz_i affects every pkp_k, so ∂L∂zi=∑k=1K∂L∂pk ∂pk∂zi=∑k(−ykpk)pk(δki−pi).\frac{\partial L}{\partial z_i} = \sum_{k=1}^{K}\frac{\partial L}{\partial p_k}\,\frac{\partial p_k}{\partial z_i} = \sum_{k}\left(-\frac{y_k}{p_k}\right)p_k(\delta_{ki} - p_i).

Step 3, cancel the pkp_k. =−∑kyk(δki−pi)=−∑kykδki+pi∑kyk= -\sum_k y_k(\delta_{ki} - p_i) = -\sum_k y_k\delta_{ki} + p_i\sum_k y_k.

Step 4, simplify each sum. ∑kykδki=yi\sum_k y_k\delta_{ki} = y_i, because only k=ik = i survives. And ∑kyk=1\sum_k y_k = 1, because the target is a probability distribution. So ∂L∂zi=−yi+pi\frac{\partial L}{\partial z_i} = -y_i + p_i.

In the vector-Jacobian language of module 5: the upstream gradient is g=−y/p\mathbf{g} = -\mathbf{y}/\mathbf{p} (entry by entry), and the softmax VJP p⊙g−(p⋅g) p\mathbf{p}\odot\mathbf{g} - (\mathbf{p}\cdot\mathbf{g})\,\mathbf{p} gives −y−(−1) p=p−y-\mathbf{y} - (-1)\,\mathbf{p} = \mathbf{p} - \mathbf{y}. The division by p\mathbf{p} and the Jacobian cancel exactly, which is why frameworks fuse the two into one "softmax cross-entropy" operation that never divides by a probability.

Read the result as an instruction to gradient descent, which moves each logit against its gradient:

  • The correct class's logit has gradient py−1≤0p_y - 1 \le 0, so it goes up, by more when the model was less sure of it.
  • Every other logit has gradient pi≥0p_i \ge 0, so it goes down, in proportion to the probability wrongly given to it.
  • The entries sum to 1−1=01 - 1 = 0: the gradient only redistributes belief. Here −0.335+0.245+0.090=0-0.335 + 0.245 + 0.090 = 0.
  • Every entry lies between −1-1 and 11, and the gradient is largest exactly when the model is confidently wrong. It never saturates.

That last point is why classifiers use cross-entropy rather than squared error on probabilities. With a sigmoid output p=σ(z)p = \sigma(z), cross-entropy again gives ∂L/∂z=p−y\partial L/\partial z = p - y (the two-class version of the same result). Squared error (p−y)2(p - y)^2 instead gives 2(p−y) σ′(z)=2(p−y) p(1−p)2(p - y)\,\sigma'(z) = 2(p - y)\,p(1 - p). When the model is confidently wrong, say p=0.01p = 0.01 with y=1y = 1, cross-entropy's gradient is −0.99-0.99, while squared error's is 2(−0.99)(0.01)(0.99)≈−0.01962(-0.99)(0.01)(0.99) \approx -0.0196, fifty times smaller. The sigmoid's flat tail has swallowed the signal exactly when it was needed most.

For a batch, the loss is a mean over NN examples, so each example's gradient is divided by NN: row nn of ∂L/∂Z\partial L/\partial\mathbf{Z} is (pn−yn)/N(\mathbf{p}_n - \mathbf{y}_n)/N.

The same result falls out of a backward pass through the graph of the computation, one node at a time, with no algebra at all. The widget below draws softmax and cross-entropy for two classes as a graph: the logits z1=2z_1 = 2 and z2=1z_2 = 1 on the left, their exponentials e1e_1 and e2e_2, the sum SS, the probability p1=e1/Sp_1 = e_1/S of the correct class, and the loss L=−ln⁡p1L = -\ln p_1 on the right. White numbers are values; amber numbers are gradients. The step counter and the arrow buttons move through the forward pass and then the backward pass, and the panel beside the graph writes out each step's formula with its numbers.

Now derive p−y\mathbf{p} - \mathbf{y} yourself, and take one gradient step with it.

On paperDerive p minus y for three classes

A 3-class classifier outputs logits z=(z1,z2,z3)\mathbf{z} = (z_1,\allowbreak z_2,\allowbreak z_3), and the correct class is class 2.

  1. Write the loss L=−ln⁡p2L = -\ln p_2 using only the logits.
  2. Differentiate it with respect to z1z_1, z2z_2 and z3z_3, and write the result as p−y\mathbf{p} - \mathbf{y}.
  3. Evaluate the gradient for z=(ln⁡2,ln⁡5,ln⁡3)\mathbf{z} = (\ln 2,\allowbreak \ln 5,\allowbreak \ln 3).
  4. Take one gradient descent step with learning rate η=1\eta = 1. What is the new probability of class 2, and the new loss?

To check part 3 before reading the steps, enter the gradient as a column of three decimals, class 1 at the top.

Work it on real paper: writing each step is the point. Then check your final answer here and compare your working with the walk-through.

the gradient in part 3

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Now implement the whole thing, stably, and check the gradient numerically. This is the loss you will use for every classifier and language model in the rest of Part 1.

Code itStable softmax, cross-entropy and a gradient check

Implement the classification loss that the rest of Part 1 uses, stably, and a tool to check any gradient. Z holds logits with one row per example, shape (N, K), and y holds integer class ids.

  1. softmax(Z), row by row, without overflow.
  2. log_softmax(Z) as z−LSE⁡(z)\mathbf{z} - \operatorname{LSE}(\mathbf{z}) per row, never taking the log of a probability.
  3. cross_entropy(Z, y): the mean over rows of −ln⁡pyn-\ln p_{y_n}.
  4. cross_entropy_grad(Z, y): (P−Y)/N(\mathbf{P} - \mathbf{Y})/N.
  5. gradient_check(f, analytic, Z): the largest relative error between analytic and a central-difference gradient of f at Z.
⌘+Enter runsPython sleeps until you run code
Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.

A few more softmax and cross-entropy computations, with fresh numbers each time.

Practice setSoftmax and cross-entropy reps
3 correct in a row completes the set. A miss starts the count again.

Compute p=softmax⁡(z)\mathbf{p} = \operatorname{softmax}(\mathbf{z}) for the logits z=(3,1,−1)\mathbf{z} = (3,\allowbreak 1,\allowbreak -1). Give each probability to 3 decimal places.

p

One entry per box, top to bottom. 0.25, -2, 3/4 and sqrt(2) all work. Enter moves to the next empty box and checks once all are filled.

Finally, explain why classifiers use this loss at all.

In your own wordsWhy cross-entropy for classification

A teammate proposes training a 10-class image classifier by applying softmax and then minimizing the squared error between the probabilities and the one-hot label. Explain why cross-entropy is the standard choice instead. Cover what each loss measures, what happens to each gradient when the model is confidently wrong, and one sanity check that cross-entropy gives you for free.

A few sentences first: 0 of 60 characters.

Saved in this browser as you type.

Next: Backpropagation