Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 45 ยท Building Models

Activation Functions and Loss Functions

Learn ReLU, GELU, sigmoid, tanh and softmax, plus MSELoss, CrossEntropyLoss and BCEWithLogitsLoss, logits vs probabilities and numerical stability, live.

A network made only of linear layers can only compute linear functions, no matter how many you stack: two matrix multiplications in a row are still one matrix multiplication. Activation functions break that limit. Placed between layers, a simple nonlinearity such as max(0, x) lets the network bend and fold its input space and approximate far more complicated functions. Loss functions are the other half: they turn "the model's output versus the truth" into the single number that autograd differentiates and the optimizer minimizes.

Choosing these correctly matters more than it first seems. The most frequent PyTorch mistakes in classification are about logits versus probabilities: putting a softmax before CrossEntropyLoss, or a sigmoid before BCEWithLogitsLoss. This lesson explains why, with live NumPy implementations of the functions, plots, and the numerical-stability tricks PyTorch uses inside its losses. The PyTorch code is read-only; the NumPy code is a stand-in that computes the same formulas you can run and inspect.

Common activations

ReLU max(0, x). The default for hidden layers.
GELU A smooth ReLU. Used in transformers.
Sigmoid Squashes to 0..1. For one yes/no output.
Tanh Squashes to -1..1. Centered at zero.
1 / 2

Activation functions

Each activation is applied elementwise, so its output has the same shape as its input.

Python
import torch
from torch import nn

x = torch.linspace(-3, 3, 7)
print(torch.relu(x))        # tensor([0., 0., 0., 0., 1., 2., 3.])
print(torch.sigmoid(x))     # values between 0 and 1
print(torch.tanh(x))        # values between -1 and 1
print(nn.functional.gelu(x))
  • ReLU, max(0, x): cheap, does not saturate for positive inputs, and its gradient is exactly 1 there. Its weakness is that a unit whose input stays negative gets gradient 0 and may never recover ("dying ReLU"). nn.LeakyReLU(0.01) keeps a small slope for negatives to avoid this.
  • GELU, x * ฮฆ(x) where ฮฆ is the standard normal cumulative distribution: like a ReLU with a smooth corner and a tiny dip below zero. It is the standard choice in transformers (nn.GELU()). nn.SiLU() (also called swish, x * sigmoid(x)) is a close cousin.
  • Sigmoid, 1 / (1 + e^-x): maps any number to (0, 1), so it reads as a probability. It saturates: for large |x| its slope is almost zero, which starves earlier layers of gradient. Today it is mainly used for the output of a binary classifier (and often not even there, see below) and inside gates of LSTMs.
  • Tanh: the same S shape scaled to (-1, 1) and centered at zero. Also saturates.
  • Softmax is different: it acts on a whole vector, turning scores z into probabilities exp(z_i) / sum(exp(z_j)) that are positive and add up to 1. Always say which dimension: torch.softmax(z, dim=-1).

For hidden layers, start with ReLU (or GELU for transformer-style models). Save sigmoid and softmax for outputs.

The first live example plots the four elementwise functions and their slopes (computed numerically). The slope plot explains a lot: ReLU's slope is 0 or 1, while sigmoid's slope never exceeds 0.25 and vanishes at both ends.

Python Activation functions and their slopes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Loss functions

A loss takes the model output and the target and returns a scalar (by default the mean over the batch; the reduction argument can change that to "sum" or "none").

Regression: nn.MSELoss. The mean of squared differences, mean((pred - y)**2). Outputs are plain real numbers with no activation at the end. nn.L1Loss (absolute error) is less sensitive to outliers; nn.HuberLoss blends the two.

Multi-class classification: nn.CrossEntropyLoss. For C classes the model outputs C raw scores per sample, the logits, shape (N, C). The targets are integer class indices of shape (N,) with dtype torch.long. The loss is the negative log of the probability that softmax assigns to the correct class.

Python
loss_fn = nn.CrossEntropyLoss()
logits = torch.tensor([[2.0, 0.5, -1.0],
                       [0.1, 0.2, 3.0]])     # raw model outputs, shape (2, 3)
target = torch.tensor([0, 2])                # correct class per sample
print(loss_fn(logits, target))               # small loss, both are right

Binary and multi-label classification: nn.BCEWithLogitsLoss. For a yes/no output, the model emits one logit per sample, and the target is a float 0. or 1. of the same shape. For multi-label problems (a photo can be "beach" and "dog" at once) emit one logit per label; each is treated as its own yes/no question.

Python
loss_fn = nn.BCEWithLogitsLoss()
logit = torch.tensor([1.5, -2.0, 0.3])
target = torch.tensor([1.0, 0.0, 1.0])       # float, same shape as the logits
print(loss_fn(logit, target))

Logits versus probabilities

The word logit means the raw, unnormalized score before softmax or sigmoid. Both CrossEntropyLoss and BCEWithLogitsLoss expect logits, because they compute the softmax/sigmoid inside, combined with the logarithm, in a numerically safe way (next section). Consequences:

  • Do not put nn.Softmax or nn.Sigmoid as the last layer of a model you train with these losses. Applying softmax twice squashes the scores and makes learning slow and the loss meaningless.
  • At prediction time, apply softmax or sigmoid yourself if you want probabilities: torch.softmax(logits, dim=-1). If you only need the predicted class, logits.argmax(dim=-1) already gives it, since softmax does not change the order.
  • The older nn.BCELoss and nn.NLLLoss do expect probabilities and log-probabilities respectively. Prefer the combined versions: BCEWithLogitsLoss, CrossEntropyLoss.

Numerical stability

Computers represent floating-point numbers with limited range. exp(1000) overflows to infinity, and log(0) is minus infinity, so a naive softmax or cross-entropy breaks on large scores. Two standard tricks fix that.

  1. Subtract the maximum before exponentiating. softmax(z) = softmax(z - max(z)), because the common factor cancels. After the shift the largest exponent is exp(0) = 1, so nothing overflows.
  2. Use the log-sum-exp form. Cross-entropy for target class t is log(sum(exp(z))) - z[t], computed with the same shift, so we never take the log of a tiny probability.

For binary cross-entropy with logit z and target y, the stable form is max(z, 0) - z*y + log(1 + exp(-|z|)). It is exactly equal to -[y log ฯƒ(z) + (1-y) log(1-ฯƒ(z))] but never evaluates log(0). This is what BCEWithLogitsLoss computes, and the reason you should not write sigmoid followed by BCELoss yourself.

The next editor compares the naive formulas with the stable ones, first on harmless inputs where they agree, then on extreme inputs where the naive ones fail. It also checks the famous gradient of softmax cross-entropy, softmax(z) - one_hot(target), against finite differences. That neat formula is why this loss trains so well: the gradient is simply "what you predicted minus what was true".

Python Stable softmax and cross-entropy versus naive

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the last row: a confidently wrong prediction (logit 40, target 0) should give a large but finite loss of about 40. The naive version computes sigmoid(40), which rounds to exactly 1.0 in floating point, then takes log(0) and returns infinity. An infinite loss yields nan gradients and a ruined training run.

Why cross-entropy and not MSE for classification?

You could in principle train a classifier with mean squared error on probabilities. It works poorly because of saturation: if a sigmoid outputs 0.001 for a target of 1 (a confident mistake), the MSE gradient is multiplied by the sigmoid's tiny slope, so the model barely learns exactly when it is most wrong. Binary cross-entropy has a gradient p - y with respect to the logit, which is largest when the error is largest. The next example plots both losses as a function of the logit so you can see the difference.

Python MSE on a sigmoid versus cross-entropy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On the left, the MSE curve flattens at 1 for very wrong predictions while cross-entropy keeps growing. On the right, the MSE gradient fades to zero when the model is confidently wrong, while the cross-entropy gradient stays close to -1, so the model keeps getting a strong push toward the right answer.

(New to layers and neurons? Read Neurons, Perceptrons and Feedforward Networks first.)

More activations: tanh, the ReLU family and softmax in detail

Tanh maps any input into (-1, 1) and is zero-centered: negative inputs give negative outputs. That symmetry makes the next layer's inputs balanced around zero, which usually helps gradient descent compared with sigmoid (whose outputs are always positive). Its gradient is 1 - tanh(x)**2, at most 1 at x = 0 (four times sigmoid's best slope of 0.25). The price is saturation: for large |x| the output sticks to -1 or 1 and the gradient vanishes, and the exponentials are costlier than ReLU's max. Use tanh in small networks, in recurrent cells (see Recurrent Networks) and wherever a bounded, centered output is wanted, and prefer ReLU-like functions in deep feedforward stacks.

The ReLU family. Plain ReLU is simple, fast, sparse (many exact zeros) and does not saturate for positive inputs. Its drawbacks are dying ReLU (a unit whose input is always negative gets zero gradient and never recovers), a non-differentiable corner at 0 (PyTorch simply uses 0 there), outputs that are never negative, and unbounded outputs that can grow large without normalization. The fixes keep a small slope on the negative side:

Function Formula for x < 0 PyTorch
ReLU 0 nn.ReLU()
Leaky ReLU 0.01 * x (fixed slope) nn.LeakyReLU(0.01)
Parametric ReLU (PReLU) a * x, where a is learned nn.PReLU()
ELU alpha * (exp(x) - 1) nn.ELU()

PReLU adds one learnable parameter per layer (or per channel with num_parameters=C), so the network chooses the negative slope itself. If you have no reason to do otherwise: ReLU for first attempts, Leaky ReLU or GELU when you see dead units, tanh or sigmoid only where their ranges are needed.

Softmax in detail. For scores z, softmax(z)_i = exp(z_i) / sum_j exp(z_j). The exponential makes every output positive and amplifies the largest score; the division makes them sum to 1, so the result reads as a probability distribution. Three practical points:

  • Stability. exp overflows for large scores, so implementations subtract the maximum first. The result is identical because the constant cancels. torch.softmax and nn.CrossEntropyLoss do this for you.
  • Temperature. Dividing the scores by a temperature T before softmax controls sharpness: T < 1 makes the distribution peaked (more confident), T > 1 flattens it. Language models use it to trade determinism against variety, and distillation uses it to soften teacher outputs.
  • Overconfidence. Softmax always assigns probability to some class, even for inputs unlike anything seen in training, and trained networks are often more confident than they are accurate. Outputs are scores, not guaranteed calibrated probabilities.

Softmax also appears in attention (weights over positions) and in reinforcement learning (a policy over actions).

Python Tanh, PReLU and softmax with a temperature

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

More loss functions

Beyond MSE, cross-entropy and BCE, PyTorch ships losses for other situations. The table gives the idea; the editor below computes each by hand.

Loss Use it for PyTorch
Binary cross-entropy -[y log p + (1-y) log(1-p)] Yes/no or multi-label outputs nn.BCEWithLogitsLoss
KL divergence sum p log(p / q) Making one distribution match another: distillation, VAEs nn.KLDivLoss
Hinge embedding Similar (+1) or dissimilar (-1) pairs by distance nn.HingeEmbeddingLoss
Margin ranking max(0, -y (x1 - x2) + margin) Ranking: the first score should beat the second by a margin nn.MarginRankingLoss
CTC Sequence labelling when input and output lengths differ and alignment is unknown (speech, handwriting) nn.CTCLoss

Binary cross-entropy behaves as a probabilistic score: confident and right costs almost nothing, confident and wrong costs a lot. For a target of 1 and a predicted probability of 0.9 the loss is -log(0.9) = 0.105; at 0.1 it is 2.303.

KL divergence measures how much a distribution q differs from a target p. It is zero only when they are equal and is not symmetric. A PyTorch quirk: nn.KLDivLoss expects the input as log-probabilities and the target as probabilities:

Python
loss_fn = nn.KLDivLoss(reduction="batchmean")           # "batchmean" matches the math
student_logp = torch.log_softmax(student_logits, dim=-1)
teacher_p = torch.softmax(teacher_logits, dim=-1)
loss = loss_fn(student_logp, teacher_p)

Hinge embedding loss takes a distance (or score) x and a label y of +1 or -1. For similar pairs (y = 1) the loss is x, pulling the distance toward zero; for dissimilar pairs (y = -1) it is max(0, margin - x), pushing the distance beyond the margin and then stopping.

Margin ranking loss takes two scores and a label y of +1 (first should be larger) or -1: max(0, -y * (x1 - x2) + margin). It is zero when the order is correct by at least the margin. It trains recommenders and search ranking. Its cousin nn.TripletMarginLoss compares an anchor with a positive and a negative example.

CTC loss (connectionist temporal classification) sums over all alignments between a long sequence of per-frame predictions and a shorter target label sequence, using a special "blank" symbol. It is what makes speech recognizers trainable without frame-by-frame labels:

Python
ctc = nn.CTCLoss(blank=0)
log_probs = model(audio).log_softmax(2)        # (time, batch, classes)
loss = ctc(log_probs, targets, input_lengths, target_lengths)
Python KL, hinge embedding and margin ranking by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Practical checklist

  • Last layer: nothing (regression), C logits (multi-class), 1 logit (binary), C logits (multi-label).
  • Loss for that case: MSELoss, CrossEntropyLoss, BCEWithLogitsLoss, BCEWithLogitsLoss.
  • CrossEntropyLoss targets are long class indices (N,); BCEWithLogitsLoss targets are floats shaped like the logits.
  • Imbalanced classes: CrossEntropyLoss(weight=...) or BCEWithLogitsLoss(pos_weight=...).
  • CrossEntropyLoss(label_smoothing=0.1) softens targets slightly and often helps generalization.
  • When the loss is nan, suspect the learning rate first, then a hand-written log or exp (see Debugging PyTorch Code).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Your classifier ends with nn.Softmax and you train it with nn.CrossEntropyLoss. What is wrong?

    CrossEntropyLoss expects raw logits and applies log-softmax internally. Remove the final softmax during training, and apply it only when you want probabilities at prediction time.
  2. 2. What dtype and shape should the targets of nn.CrossEntropyLoss have for logits of shape (N, C)?

    The standard form takes one integer class index per sample. (Recent versions also accept probability targets of shape (N, C), but class indices are the everyday case.)
  3. 3. Why subtract the maximum logit before computing exp in softmax?

    softmax(z) equals softmax(z - c) for any constant c. Choosing c as the maximum keeps all exponents at most 0, so nothing overflows.
  4. 4. Which activation is the usual default for hidden layers of a feed-forward network?

    ReLU is cheap, does not saturate for positive values and trains well. Without an activation, stacked linear layers collapse into one linear map.
  5. 5. A multi-label model must tag each photo with any subset of 5 labels. Which output and loss fit?

    Each label is an independent yes/no question, so every logit gets its own sigmoid-based binary loss. Softmax would force the five probabilities to compete.

Summary

  • Activations add nonlinearity between linear layers: ReLU (default), GELU/SiLU (transformers), sigmoid and tanh (saturating, mostly for outputs and gates); softmax turns a vector of scores into probabilities.
  • Losses by task: MSELoss for regression, CrossEntropyLoss for one-of-many, BCEWithLogitsLoss for yes/no and multi-label.
  • Classification losses expect raw logits; do not add softmax or sigmoid before them during training.
  • Stable implementations subtract the maximum and use log-sum-exp, so large scores do not overflow; the gradient of softmax cross-entropy is softmax(z) - one_hot.
  • Cross-entropy keeps a strong gradient when the model is confidently wrong, unlike MSE on probabilities.

Next, make deep networks train reliably: Initialization, Normalization and Dropout.