Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 73 Β· Neural Networks From Scratch

Activation Functions

Compare sigmoid, tanh, ReLU, leaky ReLU, GELU and softmax in NumPy. Plot each with its derivative, see saturation and dying ReLU, and learn how to choose.

An activation function is the small non-linear step applied after each layer's weighted sum. As the previous lesson showed, without it a stack of layers collapses into one. With it, the network can bend and fold its decision boundaries.

Which function you choose affects more than the shape of the curve. During training you will need the derivative of the activation, and its size decides whether learning signals flow back through the layers smoothly or fade away. This lesson plots six functions together with their derivatives and gives you a practical rule for picking one.

Six activations at a glance

Sigmoid Squashes to (0, 1). Good for probabilities.
Tanh Squashes to (-1, 1). Zero-centered.
ReLU max(0, z). The default for hidden layers.

Sigmoid and tanh saturate; ReLU does not for positive inputs.

1 / 2

Sigmoid and tanh

The sigmoid Οƒ(z) = 1 / (1 + e^(βˆ’z)) maps any real number into (0, 1). Large negative inputs give nearly 0, large positive inputs give nearly 1, and 0 gives exactly 0.5. Its derivative has a neat form: Οƒβ€²(z) = Οƒ(z)(1 βˆ’ Οƒ(z)), which peaks at 0.25 when z = 0.

Tanh is a stretched and shifted sigmoid: tanh(z) = 2Οƒ(2z) βˆ’ 1. It maps to (βˆ’1, 1), is centered on zero, and has a peak derivative of 1: tanhβ€²(z) = 1 βˆ’ tanhΒ²(z). Centered outputs usually make optimization a little easier than sigmoid outputs, which are always positive.

Both are saturating: for large |z| the curve becomes flat and the derivative approaches zero. We return to why that hurts below.

ReLU and its relatives

The rectified linear unit ReLU(z) = max(0, z) outputs zero for negative inputs and passes positive inputs through unchanged. Its derivative is 0 for z < 0 and 1 for z > 0 (at exactly 0 the slope is undefined, and implementations simply pick 0). It is cheap to compute, does not saturate for positive inputs, and is the standard choice for hidden layers.

Two variants repair its weak spot:

  • Leaky ReLU: z if z > 0, otherwise 0.01 z. A small slope on the negative side keeps a trickle of gradient.
  • GELU (Gaussian error linear unit): z Β· Ξ¦(z), where Ξ¦ is the standard normal cumulative distribution function. It looks like a smooth ReLU that dips slightly below zero before rising, and it is used in most transformer models. A common approximation is 0.5 z (1 + tanh(√(2/Ο€)(z + 0.044715 zΒ³))).
Python Activation functions and their derivatives

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The top row shows each function and the bottom row its slope. Notice how sigmoid and tanh slopes die out in the tails, while ReLU's slope is exactly 1 for every positive input.

Saturation and vanishing signals

When a unit saturates, its derivative is almost zero. During backpropagation the gradient is multiplied by this derivative at each layer (the chain rule). With sigmoid, the derivative never exceeds 0.25, so even in the best case each layer shrinks the learning signal to a quarter. After ten sigmoid layers the signal is at most about 0.25¹⁰ β‰ˆ 10⁻⁢ of its original size. The early layers barely learn. This is the vanishing gradient problem, and a main reason deep networks were hard to train until ReLU became common.

Python Gradients shrinking through sigmoid layers

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Dying ReLU

ReLU has its own failure mode. If a neuron's weights drift so that z is negative for every example, its output is always 0 and its derivative is always 0. No gradient flows through it, so its weights never change again: the unit is "dead". A large learning rate, a bad initialization or an unlucky big update can kill a sizable fraction of the units in a layer.

Leaky ReLU and GELU avoid exact zeros in the gradient, which is why they are popular when dead units are a problem.

Python Dead units in a ReLU layer

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The units with the strongly negative bias get a gradient on 0 percent of the examples. Those two units would stay dead through training.

Softmax for many classes

The activations above work on each number separately. Softmax works on a whole vector of scores (called logits) and turns it into a probability distribution:

softmax(z)α΅’ = e^(zα΅’) / Ξ£β±Ό e^(zβ±Ό)

All outputs are positive and sum to 1, and the largest score gets the largest probability. Use it on the output layer for single-label multi-class classification. Softmax is shift invariant: adding a constant to every logit changes nothing. We exploit this by subtracting the maximum first, which prevents e^z from overflowing.

Python Softmax and numerical stability

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

More activations: PReLU, ELU, SELU and Swish

Leaky ReLU fixes the dead-unit problem with a fixed small slope. Several relatives push the idea further.

  • PReLU (parametric ReLU) uses z for positive inputs and aΒ·z for negative ones, where a is a learned parameter, one per channel or one for the whole layer. The network decides how much negative signal to keep. The gradient with respect to a is the sum of the incoming gradient times z over the negative inputs.
  • ELU (exponential linear unit) is z for z > 0 and Ξ±(eΛ£ βˆ’ 1) for z ≀ 0. The negative side is a smooth curve that levels off at βˆ’Ξ±, so average outputs sit closer to zero than with ReLU. The cost is an exponential.
  • SELU (scaled ELU) is ELU multiplied by a constant Ξ» β‰ˆ 1.0507, with Ξ± β‰ˆ 1.6733. These two numbers were derived so that, with LeCun-normal weights (std = 1/√fan_in) and no dropout surprises, the mean and variance of activations are driven toward 0 and 1 from layer to layer. The network self-normalizes, so very deep plain stacks can train without batch normalization.
  • Swish / SiLU is z Β· sigmoid(z): smooth, slightly negative for small negative inputs, and used in EfficientNet and many language models (GELU is its close cousin).

Tanh deserves a plain verdict too. It is zero-centered, which makes optimization easier than with sigmoid, but it saturates at both ends, so deep stacks of tanh layers suffer from vanishing gradients. Use it where a bounded output is the point (inside LSTM and GRU cells, or an output scaled to [-1, 1]), not as a general hidden-layer default.

Python PReLU, ELU, SELU and Swish

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the last three lines: with ordinary ReLU and a weight scale that is too small, the signal fades to nothing in 30 layers, and He initialization repairs it. SELU keeps mean 0 and standard deviation 1 with no special care beyond its matching initialization. Treat SELU as a niche tool: it requires dense layers and exactly that initialization, and convolutional nets and transformers rely on normalization layers instead.

Choosing an activation

A simple rule covers most cases.

Where Typical choice Why
Hidden layers (default) ReLU Fast, no saturation for positive values
Hidden layers with dead units Leaky ReLU Keeps a small gradient
Transformer and modern language models GELU (or SiLU, a close cousin) Smooth, works well in practice
Output: regression None (identity) Output any real number
Output: yes or no Sigmoid A probability in (0, 1)
Output: one of many classes Softmax Probabilities that sum to 1
Recurrent networks (later) Tanh, sigmoid inside gates Bounded states

Do not tune the activation first. Start with ReLU in the hidden layers, get the whole pipeline working, and try alternatives only if training shows a specific problem.

Check your understanding

Quick check Β· 5 questions

of 5 answered You scored / 5 Β· perfect!

  1. 1. What is the largest possible value of the sigmoid's derivative?

    sigma'(z) = sigma(z)(1 - sigma(z)) is maximal at z = 0, where it equals 0.5 * 0.5 = 0.25.
  2. 2. What is a 'dead' ReLU unit?

    With z < 0 for every input, the output is 0 and the gradient is 0, so the unit's weights stop updating.
  3. 3. Which output activation fits a model that picks exactly one of five classes?

    Softmax converts five logits into five probabilities that sum to 1.
  4. 4. Why subtract the maximum logit before exponentiating in softmax?

    Softmax is shift invariant, so subtracting max(z) keeps the exponentials small and avoids inf values, with identical output.
  5. 5. Why do deep stacks of sigmoid layers train slowly?

    The derivative is at most 0.25, so after many layers the gradient reaching early layers is tiny: the vanishing gradient problem.

Summary

  • Activations add the non-linearity that makes depth useful; training also needs their derivatives.
  • Sigmoid and tanh saturate: flat tails give near-zero gradients and cause vanishing signals.
  • ReLU is the default hidden activation; leaky ReLU and GELU avoid dead units.
  • Dying ReLU happens when a unit's input is negative for every example.
  • Softmax turns logits into probabilities; always subtract the maximum for stability.
  • Pick the output activation by task: identity, sigmoid or softmax.

Next: Loss Functions.