Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 37 ยท Training and Scaling

Pretraining Objective and Cross-Entropy

Understand next-token cross-entropy loss, how to compute it from logits, how it relates to perplexity and bits per byte, and why teacher forcing allows parallel training.

Pretraining is the long, expensive phase in which a model with random weights becomes a language model. It uses one objective on huge amounts of unlabeled text: make the real next token more probable. No human labels are required, because the text itself supplies the answer for every position. This lesson defines the loss exactly, shows how to compute it, and connects it to the metrics (perplexity, bits per byte) you will see in papers and model cards.

Teacher forcing: every position is a training example

Take a sequence of tokens t1 ... tn. Feed the model t1 ... t(n-1) and ask it to predict the next token at every position. The "label" at position i is simply t(i+1). One sequence of length n therefore gives n-1 training signals from a single forward pass. This is called teacher forcing: during training the model is always shown the true previous tokens, never its own earlier guesses. The causal mask from Multi-Head Attention and Masking guarantees position i cannot peek at t(i+1).

Python Inputs and targets are the same text, shifted

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Cross-entropy loss

At each position the model outputs logits z, which softmax turns into probabilities p. If the true next token has id y, the loss is the negative log of the probability the model gave it:

loss = -log p[y]

If the model gave the right token probability 1, the loss is 0. If it gave probability 0.01, the loss is about 4.6. Averaging over all positions in the batch gives the training loss. The log makes the loss grow steeply as the model becomes confidently wrong, and it makes the product of probabilities over a sequence into a sum.

Python Cross-entropy from logits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Position 2 is a uniform guess, and its loss is exactly ln 4 โ‰ˆ 1.386. A fresh model starts near ln V (about 10.8 for a 50,000-token vocabulary), and training drives the loss down.

Cross-entropy as compression

The log base chooses the unit. With natural logs the loss is in nats; with log2 it is in bits. Cross-entropy in bits is the average number of bits an ideal coder would need to store each token using the model's probabilities. A better language model is a better compressor, which gives a precise sense in which "predicting the next token well" means "understanding the text's structure".

Because different tokenizers split text into different numbers of tokens, per-token losses of different models cannot be compared directly. A fair comparison uses bits per byte (or per character): total bits divided by the number of bytes of the original text.

Python Loss, perplexity and bits per byte

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The gradient: a surprisingly simple formula

For softmax followed by cross-entropy, the gradient of the loss with respect to the logits is p - onehot(y): the predicted probabilities minus the target. The model is pushed to raise the logit of the correct token by the amount it was missing and lower all others in proportion to their probability. A numerical check confirms it:

Python Check the gradient p - onehot(y)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reading a loss curve

Training logs report the loss over steps. Typical behaviors:

  • Rapid initial drop from about ln V as the model learns token frequencies, then common patterns.
  • Slow power-law decline afterward (see Scaling Laws and Compute).
  • Spikes that signal instability from too high a learning rate or bad data.
  • Train versus validation gap. If validation loss rises while training loss falls, the model is memorizing (overfitting). With very large datasets seen for one pass this is rare.

The pretraining objective

Text is the label The next token is the answer at every position.
Cross-entropy Minus the log of the right token's probability.
1 / 2

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the training label at each position in pretraining?

    The text supplies its own labels: inputs and targets are the same sequence shifted by one.
  2. 2. A model assigns probability 0.5 to the correct token. What is the loss in nats?

    -ln(0.5) = 0.693.
  3. 3. Why use bits per byte to compare models with different tokenizers?

    Normalizing by bytes removes the effect of different token counts.
  4. 4. What is the gradient of cross-entropy with respect to the logits?

    For softmax plus cross-entropy it is the predicted probabilities minus the one-hot label.

Summary

  • Pretraining uses unlabeled text: the shifted sequence provides a target at every position (teacher forcing).
  • Loss is -log p of the true token; an untrained model sits near ln V.
  • Perplexity is e raised to the loss; bits per byte compares models fairly across tokenizers.
  • The logit gradient is p minus the one-hot target.
  • Loss curves drop quickly and then decline slowly; spikes and train/validation gaps are warning signs.

Next: The Training Loop and Gradient Descent