Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 73 ยท Neural Networks From Scratch

Loss Functions

Learn how loss functions score a network's mistakes: mean squared error, binary and categorical cross-entropy, and stable log-softmax, with NumPy code.

A network's forward pass produces predictions. Before it can learn anything, something has to say how good or bad those predictions are, and say it with a single number. That number is the loss (also called the cost or the objective). Training is nothing more than a search for parameters that make the loss small, so the loss function you choose defines what "good" means for your model.

This lesson covers the three losses you will use in nearly every project: mean squared error for predicting numbers, and binary and categorical cross-entropy for predicting classes. You will also see why cross-entropy beats squared error for classification, and how to compute it without numerical overflow. The next lesson, Gradient Descent, uses the loss to improve the parameters.

One number for "how wrong"

Regression Predict a quantity. Use squared error.
Two classes One probability out. Use binary cross-entropy.
Many classes A probability per class. Use categorical cross-entropy.

The loss must be differentiable, so we can follow its slope downhill.

1 / 2

Mean squared error

For regression, where the target is a number such as a price or a temperature, the standard loss is the mean squared error (MSE). For each example you take the difference between the prediction and the target, square it, and average over the batch:

TEXT
MSE = (1/N) * sum( (prediction - target)^2 )

Squaring does two things. It makes every error positive, so over-predictions and under-predictions cannot cancel each other out, and it punishes large errors much more than small ones: an error of 4 costs sixteen times as much as an error of 1. The gradient is also simple. For one example the derivative with respect to the prediction is 2 * (prediction - target), a straight line through zero.

A close cousin is the mean absolute error (MAE), which uses the absolute value instead of the square. It treats all errors proportionally, so one wild outlier does not dominate the average. The trade-off is that its gradient has constant size, which makes it jumpy near the minimum. Try both on data that contains an outlier.

Python MSE versus MAE with an outlier

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The single miss of 10 units raises MAE from 0.3 to 2.3, but it raises MSE from 0.15 to 20.15, and the gradient almost entirely belongs to that one example. MSE tells the network "fix the worst mistake first", which is usually what you want, unless your data has errors that should be ignored.

Why not squared error for classification?

Suppose a network outputs the probability that an image shows a cat. You could treat the label (0 or 1) as a number and use MSE. It works in principle, but it learns badly, for a reason that shows up in the gradient.

With a sigmoid output p, the MSE gradient with respect to the raw score contains a factor p * (1 - p). When the network is confidently wrong (for example p is 0.001 but the label is 1), that factor is nearly zero, so the gradient nearly vanishes and the network barely corrects its worst mistakes. Cross-entropy removes this factor. Its gradient with respect to the raw score is simply p - label: the bigger the mistake, the stronger the push.

Binary cross-entropy

For two classes, the network outputs one probability p that the label is 1. The binary cross-entropy (BCE), also called log loss, for one example with label y is:

TEXT
BCE = -( y * log(p) + (1 - y) * log(1 - p) )

Only one of the two terms is active. If y = 1, the loss is -log(p): it is 0 when p = 1 and grows without bound as p approaches 0. If y = 0, the loss is -log(1 - p). The logarithm turns "probability assigned to the right answer" into a penalty that explodes for confident mistakes.

The name comes from information theory. Cross-entropy measures how many nats (units based on the natural logarithm) you would need on average to encode the true labels using the model's beliefs. A model that puts probability 1 on the truth needs zero.

Python Binary cross-entropy against squared error

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Squared error never goes above 1, so a model that is 99.9 percent sure of the wrong answer is punished only about as much as one that is simply unsure. Cross-entropy keeps climbing, which is exactly the pressure you want.

Implementations never call log(0). They clip probabilities to a tiny range such as 1e-12 to 1 - 1e-12, or better, work from the raw scores (logits), as shown at the end of this lesson.

Categorical cross-entropy

With more than two classes, the network produces one raw score (a logit) per class, and the softmax function turns them into probabilities that are positive and sum to 1:

TEXT
softmax(z)_k = exp(z_k) / sum_j exp(z_j)

The label is stored either as an integer class index or as a one-hot vector, with a 1 in the correct position and zeros elsewhere. The categorical cross-entropy is the negative log of the probability given to the correct class:

TEXT
CE = -log( p[correct class] )

Because a one-hot vector zeroes out every other term of the general formula -sum(y_k * log(p_k)), only the correct class contributes. Binary cross-entropy is the two-class special case. Averaged over a batch, it looks like this in NumPy.

Python Softmax and categorical cross-entropy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last line is a useful sanity check. A model that knows nothing assigns probability 1/K to each of K classes and gets a loss of log(K). If your freshly initialized classifier does not start near that value, something is off. Remember it when you debug in Debugging Training Runs.

The numerically stable way

The code above subtracts the row maximum before calling exp. That is not decoration. Logits of 1000 would make exp(1000) overflow to infinity, and then everything becomes nan. Subtracting a constant from all logits does not change the softmax, so subtracting the largest one makes the biggest exponent exp(0) = 1.

The same trouble hits the logarithm: log(softmax(z)) can underflow to log(0). The fix is to compute the log-softmax directly with the log-sum-exp trick:

TEXT
log_softmax(z)_k = z_k - max(z) - log( sum_j exp(z_j - max(z)) )

No probability is ever formed, so nothing underflows. Frameworks combine softmax and cross-entropy into a single function (PyTorch's CrossEntropyLoss takes logits, not probabilities) for exactly this reason. Compare the naive and stable versions on extreme logits.

Python Naive versus stable log-softmax

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The naive version returns nan for the huge logits and inf for the hopeless prediction, while the stable version gives the correct finite answers (almost exactly 0 for the huge logits, where class 0 wins easily, and 1000.6931 for the hopeless prediction).

The loss as a surface

A loss is a function of the parameters. With two parameters you can draw it as a landscape: height is the loss, and the horizontal axes are the weights. Training is a walk downhill on that landscape. The following example fits a one-input logistic model p = sigmoid(w * x + b) to a small dataset and draws the BCE surface as a contour map, with the best point marked.

Python A loss surface

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Binary cross-entropy for a logistic model is a smooth, bowl-shaped (convex) surface, so any downhill walk reaches the same bottom. Deep networks have far more parameters and their surfaces have ridges, plateaus and many valleys, but the idea is identical.

More losses: Poisson, cosine proximity, perceptual and CTC

Four more losses appear often enough that you should recognize them.

Poisson loss is for counts (calls per hour, clicks per day). The model predicts a rate ฮป = exp(z) and the loss is the negative log-likelihood of a Poisson distribution, mean(ฮป โˆ’ yยทlog ฮป) (the constant log y! is dropped). Using exp keeps the rate positive. For a model with a single constant output, the best rate is exactly the mean of the counts.

Cosine proximity loss is โˆ’cos(ฮธ), where ฮธ is the angle between the prediction and the target. It ignores length and only cares about direction, so it is used for embeddings, retrieval and similarity learning. It ranges from โˆ’1 (same direction, perfect) through 0 (orthogonal) to +1 (opposite).

Perceptual loss compares two images through the feature maps of a pretrained network (often VGG) instead of pixel by pixel. A pixel loss punishes a sharp image that is shifted by one pixel as badly as a different image, and it favors blurry averages. Feature maps respond to textures and shapes, so the perceptual loss is small for "looks the same" and large for "looks different". It is used in super-resolution, style transfer and as an extra term when training generative models.

CTC loss (connectionist temporal classification) trains sequence models when you know what the output is but not where each symbol occurs, such as speech or handwriting recognition. The model emits, for every time step, a distribution over the symbols plus a special blank. A path of symbols maps to a label by collapsing repeats and then removing blanks (a a _ b b becomes ab). The probability of a label is the sum over every path that collapses to it, and the loss is its negative log. Real implementations compute this sum with a dynamic program (the forward algorithm); the program below enumerates the paths by brute force so you can see the definition.

Python Poisson, cosine, CTC and perceptual losses

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In PyTorch these are nn.PoissonNLLLoss, nn.CosineEmbeddingLoss and nn.CTCLoss; in Keras they are keras.losses.Poisson, CosineSimilarity and ctc_batch_cost. A perceptual loss is built by running both images through the first layers of a frozen pretrained network and applying MSE to the outputs.

Choosing a loss

Match the loss to the output layer and the task:

Task Output layer Loss
Predict a number Linear (no activation) MSE, or MAE for outlier-heavy data
Two classes One sigmoid unit Binary cross-entropy
One of K classes K logits, softmax Categorical cross-entropy
Several labels at once K sigmoid units Binary cross-entropy per label

Two practical notes. First, always average over the batch (not sum), so the loss and the learning rate stay independent of batch size. Second, the loss is a training signal, not always your final report card: you will also track accuracy, precision and similar numbers in Evaluating Models.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does mean squared error punish large errors so strongly?

    Squaring grows faster than linear growth: an error of 4 costs 16 times as much as an error of 1. That is why one outlier can dominate MSE.
  2. 2. A 3-class classifier knows nothing and predicts 1/3 for every class. What is its categorical cross-entropy?

    The correct class always has probability 1/3, so the loss is -log(1/3) = log(3). This is the loss to expect from an untrained 3-class network.
  3. 3. Why is cross-entropy preferred over squared error for classification with a sigmoid or softmax output?

    With cross-entropy the gradient at the logits is p minus the label, so big mistakes get big corrections. With squared error a saturated output multiplies the gradient by a factor near zero.
  4. 4. What does the log-sum-exp trick avoid?

    Subtracting the maximum logit keeps every exponent at most 0, and computing log-softmax directly never forms tiny probabilities that round to zero.

Summary

  • The loss turns predictions and targets into one differentiable number; training minimizes it.
  • Use MSE (or MAE) for regression, binary cross-entropy for two classes or multi-label problems, and categorical cross-entropy for one-of-K classes.
  • Cross-entropy is -log of the probability given to the correct answer, so confident mistakes cost a lot; an untrained K-class model starts near log(K).
  • Compute softmax and log-softmax with the max subtracted (log-sum-exp) so extreme logits do not give nan or inf.
  • Think of the loss as a landscape over the parameters: gradient descent walks downhill on it.

Next lesson: Gradient Descent.