Skip to content
Tutorials Center
My learning

Topics

Lesson 42 of 111 ยท Model Evaluation and Tuning

Loss Functions and Cross-Entropy

What a loss function is, and how MSE, MAE, Huber, hinge, log loss, categorical and sparse categorical cross-entropy compare, with NumPy code you can run.

Training a model means choosing parameters that make some number as small as possible. That number is the loss function: it scores how wrong the model's predictions are on the training data. The optimizer (gradient descent and its variants) only sees the loss. It cannot see what you meant. So the choice of loss silently defines what "good" means to the model: it decides how outliers are treated, whether confident mistakes are punished, and whether the output behaves like a probability.

This lesson walks through the losses you will meet most often, for regression and for classification, and explains the difference between a loss (what the model minimizes) and a metric (what you report to people).

Loss versus metric

  • A loss must be (almost everywhere) differentiable and give useful gradients, because training depends on it. Log loss, mean squared error and hinge loss are losses.
  • A metric is whatever answers your real question. Accuracy, F1 or the area under the ROC curve are metrics, but accuracy is a terrible loss: it only changes when a prediction flips, so its gradient is zero almost everywhere.

It is normal to train with one function and report another. Just make sure they agree about what is important (see regression metrics and classification metrics).

Regression losses

For true values y and predictions ลท, with error e = y - ลท:

  • Mean squared error (MSE): the average of eยฒ. It is smooth and punishes big errors very strongly. That is right when large errors are really much worse, and wrong when a few bad data points should not dominate. The model that minimizes MSE predicts the mean.
  • Mean absolute error (MAE): the average of |e|. Every unit of error counts the same, so it is robust to outliers. The model that minimizes MAE predicts the median. Its gradient has constant size, which makes it jumpy near zero.
  • Huber loss: quadratic for small errors (|e| โ‰ค ฮด), linear for large ones. It combines the smooth behavior of MSE near the target with the outlier resistance of MAE. HuberRegressor and SGDRegressor(loss="huber") use it.

The next example computes all three on a small set of predictions that contains one terrible error (a true value of 60 predicted as 6), then again without that point.

Python MSE, MAE and Huber on data with one outlier

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One bad point multiplies the MSE by more than a thousand (0.36 to 486.30), but the MAE and the Huber loss by only about 17 and 50. During training, that single row would produce almost all of the gradient under MSE. If outliers are measurement errors, choose MAE or Huber; if large errors are truly costly (for example, an overshooting forecast that damages a power grid), keep MSE.

Classification losses

Log loss (binary cross-entropy)

A classifier outputs a probability p that the label is 1. Binary cross-entropy, also called log loss, is

TEXT
loss = -[ y ยท log(p) + (1 - y) ยท log(1 - p) ]       averaged over the examples

For a true label of 1 it equals -log(p): the loss is small when p is near 1 and grows without bound as p goes to 0. A confident wrong answer is punished far more than an unsure one:

Probability given to the correct class 0.9 0.5 0.1 0.01 0.001
Loss -log(p) 0.105 0.693 2.303 4.605 6.908

This is the negative log-likelihood of the model under a Bernoulli distribution, so minimizing it is maximum likelihood estimation (see density estimation and MLE). It is the loss behind logistic regression, and it also rewards well-calibrated probabilities (calibration).

Categorical and sparse categorical cross-entropy

With K classes the model outputs K probabilities that sum to 1 (a softmax output). Categorical cross-entropy compares them with a one-hot target such as [0, 1, 0]:

TEXT
loss = - sum over classes of ( y_k ยท log(p_k) )

Because the one-hot vector has a single 1, only the log-probability of the true class survives. Sparse categorical cross-entropy is the same computation with the labels given as integers (1) instead of one-hot vectors ([0, 1, 0]). The two give identical numbers; the sparse form just saves memory and a conversion step when there are hundreds or thousands of classes (a vocabulary of 50,000 words, for example). Deep learning libraries expose both names (categorical_crossentropy and sparse_categorical_crossentropy), and in scikit-learn log_loss accepts integer labels directly.

Python Binary, categorical and sparse categorical cross-entropy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All three numbers agree, which is the whole point: sparse and non-sparse cross-entropy are two ways to feed the same loss.

Hinge loss

The support vector machine uses the hinge loss. With labels y in {-1, +1} and a raw score s, the loss is max(0, 1 - yยทs). A point that is on the right side with a margin of at least 1 costs nothing; everything else costs more the further it is on the wrong side. Unlike log loss it gives no probabilities and stops caring about points that are already well classified.

Numerical stability and logits

Softmax uses exp, which overflows for large scores. Libraries therefore compute the loss directly from the raw scores (logits) with the log-sum-exp trick, subtracting the largest logit before exponentiating. If you write your own loss, do the same.

Python Why you subtract the maximum logit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Choosing a loss

Task Typical loss Notes
Regression, clean data MSE Predicts the mean; sensitive to outliers
Regression with outliers MAE or Huber MAE predicts the median
Binary classification Log loss Gives probabilities
Multiclass, one-hot labels Categorical cross-entropy Softmax output
Multiclass, integer labels Sparse categorical cross-entropy Same value, less memory
Max-margin classifiers Hinge No probabilities
Imbalanced classes Weighted log loss See imbalanced classes

When the standard losses do not match your costs (a missed fraud is 100 times worse than a false alarm), weight the examples or the classes, or tune the decision threshold afterwards (thresholds).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which statistic does a model trained with MSE tend to predict, and which one for MAE?

    Squared error is minimized by the mean, absolute error by the median, which is why MAE is more robust to outliers.
  2. 2. What is the difference between categorical and sparse categorical cross-entropy?

    Only the label format differs. Integer labels save memory when there are many classes.
  3. 3. A binary classifier gives probability 0.001 to the true class. How does the log loss for that example compare with the loss at probability 0.5?

    -log(0.001) is about 6.91, while -log(0.5) is 0.69. Confident mistakes are punished heavily.
  4. 4. Why is accuracy rarely used as a training loss?

    Gradient-based training needs a smooth loss. Train with log loss and report accuracy as a metric.

Summary

  • The loss is what the optimizer minimizes; the metric is what you report. They need not be the same, but they should agree.
  • MSE punishes large errors and predicts the mean; MAE is robust and predicts the median; Huber blends the two.
  • Log loss (binary cross-entropy) and categorical cross-entropy are negative log-likelihoods that reward good probabilities; sparse categorical cross-entropy takes integer labels and gives the same value.
  • Hinge loss belongs to margin-based models such as SVMs.
  • Compute softmax-based losses from logits with the log-sum-exp trick to avoid overflow.

Next: overfitting and underfitting, and how to tell whether the loss on training data really predicts the loss on new data.