Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 40 ยท Building Models with Keras

Activation Functions

Compare ReLU, leaky ReLU, sigmoid, tanh, softmax, GELU and swish in Keras: formulas, derivatives, output layer choices and how to use them in a model.

Without activation functions a neural network is just matrix multiplication, and a stack of matrix multiplications is itself one matrix multiplication. Nonlinear activations are what let depth do anything useful. They also play a second role at the output layer: shaping predictions into probabilities or bounded values. This lesson covers the activations you will actually use, how to apply them in Keras, and how to choose.

Why nonlinearity

Two linear layers: (x @ W1) @ W2 = x @ (W1 @ W2). The second layer added nothing. Insert a nonlinear function between them and the network can bend its decision boundaries and approximate curved functions. A network with enough hidden units and a nonlinear activation can approximate almost any reasonable function (the universal approximation theorem).

Python Stand-in: two linear layers collapse into one (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The activations

Hidden-layer activations

ReLU max(0, z). The default choice.
Leaky ReLU Small slope for z below 0.
GELU / Swish Smooth ReLU variants.

Start with ReLU.

1 / 2

ReLU (relu): max(0, z). Cheap, does not saturate for positive inputs, and trains fast. Its weakness is "dead" units whose input is always negative, so their gradient is 0 forever.

Leaky ReLU (leaky_relu): z if z > 0 else alpha * z with a small alpha such as 0.01-0.2. Keeps a little gradient for negative inputs.

Sigmoid (sigmoid): 1 / (1 + exp(-z)), squashes to (0, 1). Perfect for a single yes/no probability at the output. Avoid it in deep hidden stacks because it saturates: for large |z| the derivative is near 0, so gradients vanish.

Tanh (tanh): squashes to (-1, 1), zero-centered. Used inside recurrent layers.

Softmax (softmax): turns a vector of scores into probabilities that sum to 1. The output activation for multi-class classification (one correct class).

GELU (gelu) and Swish/SiLU (silu, swish): smooth ReLU-like curves used in transformers and modern CNNs.

ELU, SELU, softplus and others exist; use them when a paper or architecture asks for them.

Values and slopes

Slopes matter because gradients flow through them. Print values and derivatives for a few inputs:

Python Stand-in: activation values and derivatives (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the sigmoid derivative at z = ยฑ6: about 0.0025. Stack ten sigmoid layers and the gradient shrinks by a factor of 0.25 or less at each, vanishing before it reaches the early layers. ReLU's derivative is exactly 1 for positive inputs, which is why it enabled much deeper networks.

Softmax in detail

For scores z, softmax(z)_i = exp(z_i) / sum_j exp(z_j). The largest score gets the largest probability, and the output always sums to 1. The raw scores before softmax are called logits.

Python Stand-in: softmax, temperature and logits (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Using activations in Keras

There are three equivalent ways to apply one.

Python
from keras import layers, activations

# 1. as an argument (most common)
layers.Dense(64, activation="relu")

# 2. as a separate layer (needed when you want BatchNorm before the activation)
layers.Dense(64, use_bias=False)
layers.BatchNormalization()
layers.Activation("relu")      # or layers.ReLU()

# 3. as a function
activations.relu(x)

Some activations are also dedicated layers with options: layers.LeakyReLU(negative_slope=0.1), layers.PReLU() (learns the slope), layers.Softmax(axis=-1).

Choosing: a decision table

Where Task Activation
Hidden layers default relu
Hidden layers dead-ReLU trouble leaky_relu, gelu, silu
Output binary classification sigmoid (1 unit)
Output multi-class (one label) softmax (k units)
Output multi-label (several labels) sigmoid (k units)
Output regression (any value) None
Output regression (positive only) softplus or relu

Logits versus probabilities

A common trick for numerical stability: leave the last layer without activation (it outputs logits) and tell the loss from_logits=True. The loss then combines the sigmoid or softmax with the cross-entropy in one stable formula.

Python
model = keras.Sequential([
    keras.Input(shape=(784,)),
    layers.Dense(128, activation="relu"),
    layers.Dense(10),                       # logits, no activation
])

model.compile(
    optimizer="adam",
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=["accuracy"],
)

When you then need probabilities, apply keras.ops.softmax(model.predict(x)) yourself. Both styles work; just keep the from_logits flag consistent with the last layer, or the loss will be wrong without any error message. This is one of the most common silent Keras bugs.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why can't a network of only linear layers solve XOR?

    Matrix products compose into a single matrix, so depth adds no expressive power without nonlinearity.
  2. 2. Which output activation fits a multi-class problem where exactly one class is correct?

    Softmax produces a probability distribution over k classes that sums to 1.
  3. 3. Why is sigmoid a poor choice for deep hidden stacks?

    For large |z| the derivative approaches 0, and these small factors multiply through many layers.
  4. 4. A model ends with Dense(10) with no activation. What loss setting is needed?

    The outputs are logits, so the loss must apply the softmax internally with from_logits=True.

Summary

  • Nonlinear activations are what make deep networks more than one linear map.
  • Use ReLU in hidden layers by default; sigmoid, softmax or no activation at the output depending on the task.
  • Sigmoid and tanh saturate and cause vanishing gradients; ReLU does not for positive inputs.
  • Apply activations with the activation= argument, an Activation layer or a function.
  • Keep from_logits consistent with whether the last layer has an activation.

Next: The Sequential API