The map

Part of the training now pushes the model to write customers' messages too, which the product never needs. Counting the loss only on the replies puts all of it on the agent behavior the company wants.

The run turns into pretraining, because the loss now covers every token.

Training fails, because the cross-entropy loss is not defined on customers' messages.

It becomes preference optimization, because the model now sees both sides of each conversation.

It continues the text the way documents on the web tend to continue: it may write a haiku, or it may add more writing prompts, a heading or a forum reply.

It reliably writes a good haiku, because it has read many haiku.

It refuses, because it has not been told that it is allowed to answer.

It produces nothing, because a model cannot generate text until it has been fine-tuned.

Supervised fine-tuning

RLHF with a reward model trained on human comparisons

Reinforcement learning with verifiable rewards

Pretraining

numpy raises a shape error, because (4, 4) and (4,) cannot be combined.

It works: every row ends with mean 0.

It runs without an error, but subtracts row $i$'s mean from column $i$, so the result is wrong.

It centers every column on its own mean instead.

Both reuse the same weights at many positions: across space in a CNN, across time in an RNN.

Both process an entire sequence in parallel in a single step.

Both are built around attention.

Both need a separate set of weights for every position in the input.

It computed the convolutional feature map once for the whole image and cropped features for each proposed region from that map.

It replaced the hand-designed region proposals with a learned region proposal network.

It predicted a pixel mask for each detected object.

It labeled every pixel in one pass, with no region proposals.