Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 40 ยท Building Models with Keras

Building a Dense Layer from Scratch

Build a Dense layer with forward and backward passes in NumPy, check its gradients, then map every piece to keras.layers.Dense and its arguments.

The Dense (fully connected) layer is the workhorse of neural networks, and the best way to understand any Keras layer is to build one. In this lesson you will implement a Dense layer with a forward pass and a backward pass in NumPy, verify its gradients numerically, stack two of them, and then see that keras.layers.Dense is the same thing with extra conveniences.

What a Dense layer computes

Given an input batch x of shape (batch, n_in), a Dense layer with units neurons has a weight matrix W of shape (n_in, units) (Keras calls it the kernel) and a bias b of shape (units,). It outputs

output = activation(x @ W + b)

Every input feature connects to every unit, hence "fully connected". The layer has n_in * units + units parameters, so Dense layers on large inputs (a flattened 224x224x3 image has 150,528 values) get big fast, which motivates convolutional layers in Chapter 5.

Forward and backward

The forward pass computes the output. The backward pass receives dout, the gradient of the loss with respect to the layer's output, and must produce:

  • dW = x.T @ dout: the gradient for the weights;
  • db = dout.sum(axis=0): the gradient for the bias;
  • dx = dout @ W.T: the gradient to hand to the previous layer.

The three formulas follow from the chain rule applied to x @ W + b. The layer must remember its input x during the forward pass because the backward pass needs it. (This is why training needs more memory than inference.)

flowchart LR
    X["x"] --> F["forward:<br/>z = xW + b"]
    F --> Z["z"]
    Z -.->|"loss"| D["dout"]
    D --> B["backward"]
    B --> DW["dW = x.T dout"]
    B --> DB["db = sum dout"]
    B --> DX["dx = dout W.T"]

The forward pass caches x; the backward pass produces three gradients.

The layer in NumPy

We keep the activation separate from the linear part so each piece is simple. First the linear (Dense) part:

Python Stand-in: a Dense layer with forward and backward (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A useful rule: every gradient has the same shape as the thing it is the gradient of. If shapes ever disagree, the formula is wrong.

Check the gradients numerically

Let us verify dW, db and dx against finite differences, using the loss L = sum(out * coeff) for fixed random coefficients, so that dL/dout = coeff.

Python Stand-in: gradient check for the Dense layer (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All three differences are around 1e-9 or smaller, so the formulas are right.

Stacking layers

Chain two Dense layers with a ReLU between them. The backward pass goes through the layers in reverse order, and ReLU's backward pass simply zeroes the gradient where the input was not positive.

Python Stand-in: forward and backward through two layers (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss should fall clearly. A purely linear model could not fit x1 * x2; the ReLU hidden layer lets the network approximate it.

keras.layers.Dense

The real layer has the same core with many conveniences:

Python
from keras import layers

layer = layers.Dense(
    units=64,
    activation="relu",               # activation folded in
    use_bias=True,
    kernel_initializer="glorot_uniform",
    bias_initializer="zeros",
    kernel_regularizer=None,
    name="hidden_1",
)

Argument by argument:

  • units: the number of neurons (output size).
  • activation: a name or function applied after the linear part. None gives a linear layer, used for regression outputs and logits.
  • use_bias: omit the bias when a normalization layer follows.
  • kernel_initializer / bias_initializer: how weights start; see Initializers and Regularizers.
  • kernel_regularizer: an optional penalty on large weights.
  • name: a label for summary() and for finding the layer later.

Keras layers are built lazily: the weights are created when the input shape is known, either from a keras.Input or on the first call. After that, the layer expects the same number of input features.

Python
layer = layers.Dense(4)
x = np.ones((10, 3), dtype="float32")
y = layer(x)                       # builds weights with n_in = 3
print(y.shape)                     # (10, 4)
print(layer.kernel.shape)          # (3, 4)

Passing a tensor of shape (10, 5) to the same layer afterwards fails with a shape-mismatch error, because the kernel is (3, 4).

Dense on higher-rank inputs

If the input has more than two axes, say (batch, time, features), Dense applies to the last axis only, with the same weights at every position. That makes Dense usable inside sequence models, and it is how transformers apply their feed-forward blocks.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How many parameters does Dense(10) have for an input of 100 features?

    100 x 10 weights plus 10 biases is 1010.
  2. 2. Why does the layer cache its input during the forward pass?

    dW = x.T @ dout, so x must be remembered until the backward pass.
  3. 3. What shape does dx have for input x of shape (batch, n_in)?

    A gradient has the same shape as the quantity it belongs to, so dx matches x.
  4. 4. With Dense applied to input of shape (batch, time, features), which axis does it transform?

    Dense multiplies along the last axis, applying the same weights at every time step.

Summary

  • A Dense layer computes activation(x @ W + b) with n_in * units + units parameters.
  • Backward pass: dW = x.T @ dout, db = dout.sum(0), dx = dout @ W.T; gradients match the shapes of their targets.
  • Stacked layers run backward in reverse order; ReLU passes gradient only where its input was positive.
  • keras.layers.Dense builds the same layer lazily and adds initializers, regularizers and naming.

Next: Activation Functions