Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 37 ยท Applications

Cross-Entropy and KL Divergence in Machine Learning

See why classifiers minimize cross-entropy: derive KL divergence, softmax loss and its gradient, label smoothing and perplexity, and train a softmax classifier from scratch in NumPy.

Almost every classifier you will meet in machine learning, from logistic regression to the largest language models, is trained by minimizing the same quantity: the cross-entropy between the true labels and the model's predicted probabilities. It is not an arbitrary choice. Cross-entropy comes straight out of information theory, and understanding that connection explains loss functions, evaluation metrics such as perplexity, and why "KL divergence" is everywhere in modern papers.

This lesson connects the entropy you know to the practical formulas: cross-entropy, KL divergence, softmax loss and perplexity. All examples use plain NumPy.

Cross-entropy and KL divergence

Suppose data really follows distribution p, but you encode it with a code designed for a different distribution q. The average code length in bits is the cross-entropy:

TEXT
H(p, q) = - sum_x p(x) log2 q(x)

If q = p this is just the entropy H(p). Otherwise it is larger, and the excess is the Kullback-Leibler divergence:

TEXT
D_KL(p || q) = sum_x p(x) log2( p(x) / q(x) ) = H(p, q) - H(p)

KL divergence is the extra bits per symbol you pay for using the wrong model. Two properties matter:

  • D_KL(p || q) >= 0, with equality only when p = q (Gibbs' inequality).
  • It is not symmetric: D_KL(p || q) != D_KL(q || p) in general, and it is infinite if q(x) = 0 where p(x) > 0. It is a divergence, not a distance.

Since H(p) does not depend on the model, minimizing cross-entropy over q is the same as minimizing KL divergence, and the same as maximum likelihood estimation.

Python Entropy, cross-entropy and KL

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Softmax cross-entropy loss

A neural network outputs raw scores (logits) z. Softmax turns them into probabilities: q_i = exp(z_i) / sum_j exp(z_j). For a one-hot true label y (all probability on the correct class c) the cross-entropy is simply -log q_c: the negative log-probability the model gave to the right answer. In machine learning code the natural logarithm is usual, so the loss is measured in nats.

The gradient of this loss with respect to the logits has a remarkably simple form: q - y, the difference between predicted and true distributions. A small numerical check proves it.

Python Stable softmax loss and its gradient

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first line of softmax is not decoration: exp(1000) overflows, but subtracting the maximum first gives the same probabilities without any overflow.

Training a classifier by minimizing cross-entropy

A softmax regression model is a single linear layer z = X W + b. The gradient of the average loss is X^T (q - y) / n, so a few lines of NumPy are enough to train it.

Python Softmax regression from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At the start the weights are zero, so the model predicts uniformly and the loss is ln 3, the entropy of a 3-way uniform guess. Training pushes the loss toward zero as the model becomes confident and correct.

Perplexity

Language models are scored by perplexity: 2^H (or e^H in nats) where H is the model's average cross-entropy per token on test text. It reads as an "effective branching factor": a perplexity of 50 means the model is as uncertain as if it were choosing uniformly among 50 tokens at each step. A model that guesses uniformly over a vocabulary of size V has perplexity exactly V; a perfect model has perplexity 1.

Label smoothing and regularization

A hard one-hot target asks the model to push the probability of the right class to 1, which drives logits to infinity. Label smoothing replaces y by (1 - ฮต) y + ฮต / K. The best achievable cross-entropy is then the entropy of the smoothed target, not zero, and the model stays a little humble.

Python Perplexity and label smoothing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Mutual information as a feature score

Information theory also helps choose features. A feature that has high mutual information with the label tells you a lot about it; a feature with zero mutual information is useless on its own. Decision trees use the same idea as information gain, the reduction in label entropy after a split.

Related ideas include the Jensen-Shannon divergence, a symmetric and bounded relative of KL used to compare distributions, and the KL term in variational autoencoders, which keeps the encoder's distribution near a simple prior.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the relation between cross-entropy, entropy and KL divergence?

    Cross-entropy equals the true entropy plus the extra bits paid for using the wrong model, which is the KL divergence.
  2. 2. What is the gradient of softmax cross-entropy with respect to the logits?

    The gradient is the predicted distribution minus the one-hot target.
  3. 3. What is the perplexity of a model that guesses uniformly among V tokens?

    Its cross-entropy is log2 V bits, so 2^(log2 V) = V.
  4. 4. Why subtract the maximum logit before computing softmax?

    Softmax is unchanged when a constant is subtracted from all logits, and this prevents exp from overflowing.

Summary

  • Cross-entropy H(p, q) is the average code length when coding data from p with a code for q.
  • H(p, q) = H(p) + KL(p || q); KL is non-negative, asymmetric and measures wasted bits.
  • Minimizing cross-entropy equals maximum likelihood and minimizing KL.
  • Softmax cross-entropy has the gradient q - y.
  • Perplexity is 2^H, the effective number of equally likely choices.

Next: Information Theory in Cryptography