Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 40 ยท Training and Evaluation

Loss Functions

Choose and understand Keras losses: MSE, MAE, Huber, binary and categorical cross-entropy, from_logits, the p minus y gradient, with runnable NumPy versions.

The loss is the single number training tries to make small. It defines what "good" means for your model, so choosing it wrongly trains the model to do the wrong thing, however long you run it. A loss must be differentiable (gradients must flow through it), and it should match the task: numbers to predict call for one family, class labels for another. This lesson walks through the losses you will use, shows their formulas, and lets you run each one.

What a loss does

For each example the loss compares the model's prediction y_pred with the target y_true and returns a non-negative number: 0 for perfect, larger for worse. Keras averages over the batch (by default) to get one scalar, and the optimizer follows its gradient.

Python
loss_fn = keras.losses.MeanSquaredError()
loss = loss_fn(y_true, y_pred)           # a scalar tensor

You can pass a loss to compile by object (keras.losses.MeanSquaredError()), which lets you set options, or by string name ("mse") as a shortcut.

Regression losses

Mean squared error (MSE): mean((y_pred - y_true)**2). Squares the error, so big mistakes are punished heavily. Smooth and the default for regression. Sensitive to outliers.

Mean absolute error (MAE): mean(|y_pred - y_true|). Treats all errors linearly, so it is robust to outliers, but its gradient has constant size, which can make the final convergence jumpy.

Huber: quadratic for small errors, linear for large ones, with a threshold delta. It combines the smoothness of MSE near the target with the robustness of MAE for outliers: keras.losses.Huber(delta=1.0).

Python Stand-in: MSE, MAE and Huber with an outlier (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The single outlier multiplies MSE enormously but MAE and Huber much less. Choose by how you want to treat rare big errors.

Classification losses: cross-entropy

For classes, the standard loss is cross-entropy, which measures how well the predicted probability distribution matches the true one. For one example whose true class is c, it is simply

loss = -log(p_c)

the negative log of the probability the model assigned to the correct class. If the model gives the right class probability 1, loss is 0. At 0.5, loss is 0.69. At 0.01, loss is 4.6. It punishes confident wrong answers severely, which is exactly what you want.

Python Stand-in: -log(p) as a penalty (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three cross-entropy variants exist in Keras. Pick by label format and number of classes:

Task Last layer Labels Loss
Binary (yes/no), or multi-label Dense(k, sigmoid) 0/1 (per label) BinaryCrossentropy
Multi-class, one-hot labels Dense(k, softmax) [0,0,1,0] CategoricalCrossentropy
Multi-class, integer labels Dense(k, softmax) 2 SparseCategoricalCrossentropy
Python Stand-in: the three cross-entropies (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last two give identical numbers; they differ only in the label format they accept. Mixing them up (integer labels with the categorical loss) raises a shape error.

from_logits

If the final layer has no activation (it outputs raw scores, called logits), pass from_logits=True. Keras then applies sigmoid or softmax inside the loss using a numerically stable formula (log-sum-exp), which avoids log(0) and overflow. If the last layer already applies softmax or sigmoid, leave from_logits=False.

Python
# Style A: activation in the model, probabilities out
layers.Dense(10, activation="softmax")
loss = keras.losses.SparseCategoricalCrossentropy()            # from_logits=False

# Style B: logits out (recommended: more stable)
layers.Dense(10)
loss = keras.losses.SparseCategoricalCrossentropy(from_logits=True)

Getting this wrong is silent: training runs, but accuracy is poor. When a classifier "inexplicably" learns badly, check this first.

The beautiful gradient

For softmax followed by cross-entropy, the gradient of the loss with respect to the logits is simply p - y (predicted probabilities minus the one-hot target). That is why the combination is so popular: the gradient is large when the model is wrong and shrinks to zero as it becomes right, and there is no vanishing-gradient flat spot at the output. Verify it numerically:

Python Stand-in: the gradient of softmax + cross-entropy is p - y (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Other losses worth knowing

  • Hinge (keras.losses.Hinge): for margin-based classifiers like SVMs, labels in {-1, 1}.
  • KL divergence: how one distribution differs from another; used in knowledge distillation and VAEs.
  • Cosine similarity: for embedding models, where direction matters more than magnitude.
  • Focal loss (BinaryFocalCrossentropy): down-weights easy examples when classes are very imbalanced.
  • Custom losses: any function loss(y_true, y_pred) returning a per-example or scalar tensor built from tensor operations.
Python
def weighted_mse(y_true, y_pred):
    weights = keras.ops.where(y_true > 100, 2.0, 1.0)     # penalize big targets double
    return keras.ops.mean(weights * keras.ops.square(y_true - y_pred))

model.compile(optimizer="adam", loss=weighted_mse)

Class imbalance and sample weights

When one class is rare, plain cross-entropy lets the model ignore it. Give rare classes more weight with fit(..., class_weight={0: 1.0, 1: 10.0}), or give per-example weights with sample_weight. Metrics other than accuracy matter too (Metrics and Validation).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which loss is the most robust to large outliers?

    MAE grows linearly with the error, while MSE grows with its square.
  2. 2. A true class has predicted probability 0.01. What is the cross-entropy for that example?

    -log(0.01) is about 4.6: confident wrong predictions are punished heavily.
  3. 3. The last layer is Dense(10) with no activation and labels are integers 0-9. Which loss setting is right?

    Integer labels need the sparse variant, and raw scores need from_logits=True.
  4. 4. What is the gradient of softmax + cross-entropy with respect to the logits?

    It is predicted probabilities minus the one-hot target, large when wrong and zero when right.

Summary

  • The loss defines the training objective; it must be differentiable and match the task.
  • Regression: MSE (default), MAE (robust), Huber (both).
  • Classification: binary cross-entropy for yes/no and multi-label, categorical for one-hot, sparse categorical for integer labels.
  • Use from_logits=True when the last layer has no activation; mismatches fail silently.
  • Softmax plus cross-entropy has the clean gradient p - y.

Next: Optimizers