Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 73 ยท Neural Networks From Scratch

Layers and the Forward Pass

Stack dense layers in NumPy: weight matrix shapes, batches as rows, a two-layer forward pass, parameter counts, and why depth needs a non-linearity.

A single neuron draws one straight line. To do more, we put many neurons side by side in a layer, then feed the outputs of one layer into the next. The act of pushing an input through all the layers to get a prediction is called the forward pass. It is the part of a network you use every time you ask a trained model a question.

This lesson is mostly about bookkeeping: which matrix has which shape, how many parameters you end up with, and why the layers must be separated by a non-linear function or the whole stack collapses into a single layer. Getting comfortable with shapes now will save hours of debugging later.

A dense layer

A dense (or fully connected) layer with n_in inputs and n_out neurons has:

  • a weight matrix W of shape (n_in, n_out), where column j holds the weights of neuron j;
  • a bias vector b of shape (n_out,), one bias per neuron.

For a batch X of shape (batch, n_in), the layer computes

Z = X @ W + b, then A = ฯ†(Z)

and A has shape (batch, n_out). Every example is a row; every neuron is a column. The bias vector broadcasts across the rows.

Shapes in a dense layer

X: (batch, n_in) One example per row.
W: (n_in, n_out) One column per neuron.
b: (n_out,) One bias per neuron.
A: (batch, n_out) One output row per example.

Layer sizes chain together, so n_out of one layer is n_in of the next.

1 / 2
Python One dense layer and its shapes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stacking layers

A network with an input layer, one hidden layer and an output layer is a "two-layer network" (we count layers that have weights; the input is not counted). The forward pass is a chain:

H = ฯ†(X @ W1 + b1), then Y = g(H @ W2 + b2)

flowchart LR
    X["X<br/>(batch, 3)"] --> L1["W1: (3, 4)<br/>b1: (4,)"]
    L1 --> H["H<br/>(batch, 4)"]
    H --> L2["W2: (4, 2)<br/>b2: (2,)"]
    L2 --> Y["Y<br/>(batch, 2)"]

Shapes flow through a network with 3 inputs, 4 hidden units and 2 outputs.

Terminology you will meet constantly:

  • Input layer: the data itself, with no computation.
  • Hidden layers: everything between the input and the output; their values are not directly observed.
  • Output layer: produces the prediction. Its activation g depends on the task: nothing (identity) for regression, sigmoid for a yes/no answer, softmax for one-of-many classification.
Python A two-layer forward pass

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The probabilities are meaningless right now because the weights are random. Training, which starts with Loss Functions and Gradient Descent, is what makes the numbers useful.

Counting parameters

Each dense layer contributes n_in ร— n_out weights plus n_out biases. The parameter count is a rough measure of a model's size and memory cost. For the 3-4-2 network above:

  • layer 1: 3 ร— 4 + 4 = 16
  • layer 2: 4 ร— 2 + 2 = 10
  • total: 26

Parameter counts grow quickly. An image of 28 ร— 28 pixels has 784 inputs; a hidden layer of 512 units already has 784 ร— 512 + 512 = 401,920 parameters. That is why later chapters introduce convolutions, which share weights and need far fewer.

Python Counting parameters of any network

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Why depth needs a non-linearity

Suppose we skip the activation function between two layers. Then

Y = (X @ W1 + b1) @ W2 + b2 = X @ (W1 @ W2) + (b1 @ W2 + b2)

That is exactly one layer with weights W1 @ W2 and bias b1 @ W2 + b2. A stack of purely linear layers, however tall, is equivalent to a single linear layer and can only draw straight boundaries. The activation function breaks this. After a non-linear step, the next layer receives a bent version of the data, and many stacked bends can approximate very complicated functions.

Forgetting the activation between hidden layers is a classic beginner bug. The network trains without errors, but it performs no better than linear regression or logistic regression.

Run the check below to see the collapse with your own eyes, and see that adding a ReLU makes the two-layer version genuinely different.

Python Two linear layers collapse into one

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A tiny reusable network

Putting it together, a forward pass over any list of layers is just a loop. This is the skeleton we will grow into a full training framework over the coming lessons.

Python A forward pass over any list of layers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The odd-looking np.sqrt(2 / n_in) scaling keeps the signal from shrinking or exploding as it passes through layers; Weight Initialization explains why.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A dense layer takes 10 inputs and has 6 neurons. What are the shapes of W and b?

    W has one row per input and one column per neuron, so (10, 6). There is one bias per neuron, so b has shape (6,).
  2. 2. How many parameters does a 5-8-3 network have?

    Layer 1 has 58 weights plus 8 biases (48). Layer 2 has 83 weights plus 3 biases (27). Total 75.
  3. 3. Why does a stack of layers with no activation functions behave like a single layer?

    (X W1) W2 = X (W1 W2), so two linear layers equal one linear layer with weights W1 W2.
  4. 4. In the convention used here, what does each row of X represent?

    Examples are rows and features are columns, so X @ W processes the whole batch at once.

Summary

  • A dense layer computes A = ฯ†(X @ W + b) with W shaped (n_in, n_out) and b shaped (n_out,).
  • Examples are rows; the output of one layer is the input of the next, so sizes must chain.
  • The forward pass is a loop over layers; the output activation depends on the task.
  • Parameters per layer: n_in * n_out + n_out.
  • Without a non-linear activation between them, stacked layers collapse into a single linear layer.

Next: Activation Functions.