Generating text

Suppose a trained character model has read "th". It hands back a probability for every character: e 0.59, a 0.21, t 0.07, and small amounts for the rest. That is a prediction, not text. To write, something has to choose one character, add it to the text, and ask the model again about the longer text, over and over.

The previous lesson built the probabilities and the loss that trains them. This lesson builds the loop that writes with them, and looks closely at the one decision inside it: which character to take. Always the favorite? A random draw? Something in between? Every chat model writes its replies with this same loop, one token at a time, and the temperature setting of a language-model API is one of the knobs you will meet here.

The generation loop

Why a loop? Because the model only ever predicts one character ahead. To get a second character, the first one has to become part of the text the model reads. So generation runs the prediction again and again:

Feed a starting text, the prompt (older papers call it the prime), through the model, updating the state.
Compute $\mathbf{p}$ , the distribution over the next character.
Choose a character from $\mathbf{p}$ and append it to the output.
Feed that character in as the next input and go back to step 2.

Step 4 is where generation differs from training. With teacher forcing, the input at every step was the true character from the text. Here there is no true text, so the model reads its own choices, and a poor choice stays in the text it reads from then on.

The diagram below runs this loop three times. It uses a tiny model with 6 characters, space, a, e, h, n and t, in that order, starting from the prompt "th". Its logits are made up for the example (a trained RNN would compute them from its state), so every number can be checked by hand. The top row holds the text so far. The chart below it shows the round in progress: first the logits, then the probabilities, then their running totals with a random number $u$ drawn across them.

Try it, step by step

Step 1 shows the logits after "th" for (space, a, e, h, n, t): $(-0.50,\allowbreak 0.98,\allowbreak 2.00,\allowbreak -0.82,\allowbreak -0.50,\allowbreak -0.09)$ . The tallest bar is e.
Press Next. Step 2 divides every logit by the temperature $\tau = 0.8$ and applies the softmax. The exponentials add up to 17.91, so e gets $12.18 / 17.91 = 0.68$ , and the whole distribution is $(0.03,\allowbreak 0.19,\allowbreak 0.68,\allowbreak 0.02,\allowbreak 0.03,\allowbreak 0.05)$ . The dashed outlines show the same logits at $\tau = 1$ , where e would get only 0.59.
Step 3 stacks the probabilities into running totals, 0.03, 0.22, 0.90, 0.92, 0.95, 1.00, and draws $u = 0.62$ . The first running total above 0.62 is 0.90, so $u$ lands in e's stretch, from 0.22 to 0.90, which is 0.68 long. Step 4 appends e: the text is now "the", and it is the input for the next round.
Steps 5 and 6 repeat the round. After "the" the favorite is space (0.55), but $u = 0.81$ falls between 0.65 and 0.95, in n's stretch, so the text becomes "then". Sampling picks n here 30% of the time. After "then", $u = 0.37$ is below the first running total, 0.82, so space is picked and the text reads "then ".
Step 7 shows greedy decoding, which always takes the largest bar. It agrees in round 1 (e, 0.68), takes space (0.55) in round 2 where the sample drew n, and takes t (0.51) after "the ". Greedy writes "the t" every time it runs; sampling wrote "then " with these three draws and writes something else with others.

Greedy or sampled

The loop is fixed; the choice in step 3 is ours. There are two basic ways to make it.

Greedy decoding always takes the most likely character. It is deterministic: the same prompt always gives the same text. And it tends to fall into loops. Greedy decoding with the bigram table from the previous lesson, starting from "T", produces "The the the the the the the": after "e" the most likely character is a space, after a space it is "t", after "t" it is "h", after "h" it is "e", and the cycle never breaks.

Greedy also does not find the most likely text, even though it takes the most likely character at every step. A two-character example shows why. Say the first character is A with probability 0.6 or B with 0.4. After A the model is unsure: its best next character has probability 0.4. After B it is confident: its best next character has probability 0.9. Greedy takes A, then the best continuation, for a two-character text of probability $0.6 \times 0.4 = 0.24$ . Starting with B instead gives $0.4 \times 0.9 = 0.36$ . The locally best first choice led to a less likely text. Finding the most likely text in general means searching over whole sequences, which is not what a single greedy pass does.

Sampling draws the character at random with the model's probabilities, so a character with probability 0.3 is chosen 30% of the time. Sampled text is more varied, and on average it looks more like the training text. To see why, we need to know what a trained model's probabilities mean.

What the probabilities mean. A trained model's $p_t(k)$ is its estimate of how often character $k$ follows contexts like the one it is in. The reason is the loss. Cross-entropy rewards honest estimates: on average over many occurrences of a context, the loss is lowest when the model's probabilities match the true frequencies.

Here are numbers first. Suppose that after some context the next character is "e" 30% of the time and something else 70% of the time. A model that says $p = 0.3$ for "e" pays $-\ln 0.3$ in the 30% of cases where "e" comes and $-\ln 0.7$ in the other 70%, so on average $-0.3\ln 0.3 - 0.7\ln 0.7 = 0.361 + 0.250 = 0.611$ nats. Saying $p = 0.5$ costs $-0.3\ln 0.5 - 0.7\ln 0.5 = 0.208 + 0.485 = 0.693$ , and saying $p = 0.1$ costs $-0.3\ln 0.1 - 0.7\ln 0.9 = 0.691 + 0.074 = 0.765$ . The honest 0.3 is cheapest. The box below shows this holds for any true frequency.

Go slower: Why cross-entropy rewards the true frequencies

Take the simplest case, two possible next characters. Suppose that in some context the first one truly follows a fraction $q$ of the time, and the model says $p$ . A fraction $q$ of the time the loss is $-\ln p$ , and the rest of the time it is $-\ln(1 - p)$ , so the expected loss is $E(p) = -q\ln p - (1 - q)\ln(1 - p).$ Differentiate term by term. The derivative of $-q\ln p$ is $-q/p$ . The derivative of $-(1 - q)\ln(1 - p)$ is $-(1 - q) \times \frac{1}{1 - p} \times (-1) = \frac{1 - q}{1 - p}$ , by the chain rule on $\ln(1 - p)$ . So $E'(p) = -\frac{q}{p} + \frac{1 - q}{1 - p}.$ Set it to zero and solve for $p$ , one move per line:

\begin{aligned} \frac{q}{p} &= \frac{1 - q}{1 - p} && \text{move the first term across} \\ q(1 - p) &= p(1 - q) && \text{multiply both sides by } p(1 - p) \\ q - qp &= p - qp && \text{expand both products} \\ q &= p && \text{add } qp \text{ to both sides} \end{aligned}

The second derivative is $E''(p) = \frac{q}{p^2} + \frac{1 - q}{(1 - p)^2}$ , a sum of two positive terms, so $E$ curves upward everywhere and $p = q$ is its minimum. The same holds with any number of outcomes: the expected cross-entropy is smallest when the predicted distribution equals the true one.

So training pushes the probabilities toward the real frequencies, at least on text like the training text. Sampling from them reproduces those frequencies, to the extent the model has learned them: if "e" follows a context 30% of the time in Shakespeare, a well-trained model samples "e" after it about 30% of the time. Greedy decoding throws that information away and keeps only the favorite.

Drawing a sample

Sampling needs a way to pick a character with exactly its probability, using the one kind of randomness a computer offers directly: a uniform random number between 0 and 1. The diagram did it with running totals. Here is why that works.

Lay the probabilities end to end along $[0,\allowbreak 1)$ , each as a stretch as long as its probability. For $\mathbf{p} = (0.5,\allowbreak 0.3,\allowbreak 0.2)$ the stretches are $[0,\allowbreak 0.5)$ , $[0.5,\allowbreak 0.8)$ and $[0.8,\allowbreak 1)$ . Their right ends are the running totals $0.5$ , $0.5 + 0.3 = 0.8$ and $0.8 + 0.2 = 1.0$ . Draw $u$ uniformly from $[0,\allowbreak 1)$ and return the character whose stretch contains it, which is the first character whose running total is above $u$ . For example $u = 0.62$ is not below 0.5 but is below 0.8, so it picks the second character.

Why is that fair? A uniform $u$ lands in any stretch with probability equal to the stretch's length, and each length is exactly that character's probability. The first character is picked whenever $u < 0.5$ , which happens half the time; the second whenever $0.5 \le u < 0.8$ , which happens $0.3$ of the time; the third the remaining $0.2$ .

Try one draw yourself. The trap is to compare $u$ with the probabilities instead of their running totals.

Quick checkOne draw by hand

A model's next-character probabilities are $0.1$ for a, $0.6$ for e and $0.3$ for o, in that order. The sampler draws $u = 0.65$ uniformly from $[0,\allowbreak 1)$ and uses the running-totals method of the lesson. Which character does it return?

Choose one answer, then check.

Temperature

Greedy decoding is dull and repetitive; plain sampling sometimes picks an unlikely character that derails the text. Temperature moves between the two. Divide the logits by a number $\tau > 0$ before the softmax:

p^{(\tau)}(k) = \frac{e^{z_k/\tau}}{\sum_j e^{z_j/\tau}} = \frac{p(k)^{1/\tau}}{\sum_j p(j)^{1/\tau}} .

Here $p(k)$ is the ordinary softmax probability ( $\tau = 1$ ), and $p^{(\tau)}(k)$ is the probability after the temperature. With $\tau = 1$ nothing changes. With $\tau < 1$ the large probabilities grow at the expense of the small ones, and as $\tau \to 0$ all the probability moves to the most likely character: greedy decoding. With $\tau > 1$ the distribution flattens toward uniform. (We write $\tau$ because $T$ is the sequence length; the instrument below calls it $T$ .)

The second form of the formula, with powers of the probabilities, is not obvious from the first. The box shows why they agree.

Go slower: Why dividing the logits is the same as a power of the probabilities

Softmax gives $p(k) = e^{z_k}/S$ with $S = \sum_j e^{z_j}$ . Take logs: $\ln p(k) = z_k - \ln S$ , so $z_k = \ln p(k) + \ln S$ . The logits are the log-probabilities plus one constant $C = \ln S$ , the same for every $k$ .

Divide by $\tau$ and exponentiate, one move at a time: $e^{z_k/\tau} = e^{(\ln p(k) + C)/\tau} = e^{\ln p(k)/\tau}\,e^{C/\tau} = \big(e^{\ln p(k)}\big)^{1/\tau}\,e^{C/\tau} = p(k)^{1/\tau}\,e^{C/\tau}.$ The second step splits $e^{a + b} = e^a e^b$ , and the third uses $e^{a/\tau} = (e^a)^{1/\tau}$ .

The softmax divides each of these by their sum. The factor $e^{C/\tau}$ is in every term, so it cancels: $p^{(\tau)}(k) = \frac{p(k)^{1/\tau}\,e^{C/\tau}}{\sum_j p(j)^{1/\tau}\,e^{C/\tau}} = \frac{p(k)^{1/\tau}}{\sum_j p(j)^{1/\tau}}.$ So you can apply a temperature either to logits or to probabilities. In code, work with $\ln p(k)/\tau$ and a stable softmax: raising probabilities to the power $1/\tau$ directly underflows to 0 for small $\tau$ .

A worked example with $\mathbf{p} = (0.5,\allowbreak 0.3,\allowbreak 0.2)$ .

At $\tau = 0.5$ the power is $1/\tau = 2$ , so each probability is squared: $(0.25,\allowbreak 0.09,\allowbreak 0.04)$ . These sum to $0.38$ , and dividing each by $0.38$ gives $\mathbf{p}^{(0.5)} = (0.658,\allowbreak 0.237,\allowbreak 0.105)$ .
At $\tau = 2$ the power is $1/2$ , so each is square-rooted: $(0.7071,\allowbreak 0.5477,\allowbreak 0.4472)$ . These sum to $1.7020$ , and dividing gives $\mathbf{p}^{(2)} = (0.415,\allowbreak 0.322,\allowbreak 0.263)$ .

The order never changes; only how decisive the distribution is. The first character went from 0.5 to 0.658 when sharpened and to 0.415 when flattened.

The instrument below lets you sample from a distribution and change its temperature. Its three logits, $(0,\allowbreak -0.51,\allowbreak -0.92)$ for (e, a, i), give probabilities of about 0.5, 0.3 and 0.2, the worked example's. The blue bars at the top are the logits and the white bars are the probabilities. Sample draws characters and keeps a running count, drawn as hollow bars beside the white ones, and the strip under the buttons shows the characters drawn.

Try it, step by step

Read the start. The probabilities are 0.500, 0.300 and 0.199 for e, a and i (these logits give 0.1994, not exactly 0.2), the temperature readout is 1, and the entropy is 1.029 nats against 1.099 for a uniform guess.
Press Sample 100. Hollow bars appear beside the white ones, and the line under the bars says how many of the 100 draws were e, near 50. Press it a few more times: with every hundred draws, the hollow bars come closer to the white ones.
Drag the temperature slider to 0.5. The samples clear, because the distribution changed, and the bars become 0.658, 0.237 and 0.105, the worked example's numbers. The line says every gap between two logits grows 2 times. Sample 100 again and count how rarely i appears now.
Drag the temperature to 2. The bars become 0.416, 0.322 and 0.262, which differ from the worked example's 0.415, 0.322 and 0.263 only because these logits give 0.500, 0.300 and 0.199 rather than exactly 0.5, 0.3 and 0.2. The entropy rises to 1.081, close to the uniform 1.099, and a draw now picks a or i more often than e ( $0.322 + 0.262 = 0.584$ ).
Press T = 1 to go back. Drag toward the slider's left end, 0.1: e takes nearly all the probability, and sampling turns into greedy decoding.

Check what a temperature below 1 does to a distribution you have not seen yet.

Quick checkWhat temperature does

A model's next-character probabilities are $(0.6,\allowbreak 0.3,\allowbreak 0.1)$ . What does sampling at temperature $\tau = 0.5$ do?

Choose one answer, then check.

Now put the lesson into code: a temperature that cannot underflow, a sampler that uses exactly one random number per draw, and a generation loop over the bigram table from the previous lesson. You will see greedy decoding fall into "the the the" for yourself.

Code itSampling with a temperature

Generate text from a bigram model.

apply_temperature(probs, temperature): the reshaped distribution, computed in log space so that a tiny temperature cannot underflow. Temperature 0 is greedy.
sample_index(probs, rng): one draw with one uniform random number, by the running-sum method of the lesson.
generate(P, stoi, itos, prime, n, temperature, rng): continue a prime for n characters, each drawn from the row of the bigram table for the previous character.

The code at the bottom builds the Shakespeare bigram table and prints samples at temperatures 1, 0.5 and 0.

Write your code where the starter says raise NotImplementedError, then press Run tests. Each check says what it expects.

To recap: generation is a loop of predict, choose, append; greedy takes the favorite and repeats itself; sampling draws with the model's probabilities, which training makes into honest frequencies; and a temperature sharpens or flattens those probabilities before the draw. The bigram table writes poor English because one character of context is very little. The next lesson trains the RNN, whose state can carry much more.

#The generation loop

#Greedy or sampled

#Drawing a sample

#Temperature

The generation loop

Greedy or sampled

Drawing a sample

Temperature