Skip to content
Tutorials Center
My learning

Topics

Lesson 45 of 49 ยท Projects

Project: A Tiny Neural Network

Build a two-layer neural network with only NumPy. Write the forward pass, softmax and cross-entropy, backpropagation with matrices, a training loop and a decision boundary plot.

Deep learning frameworks can feel like black boxes, but a neural network is mostly matrix multiplication with a few simple functions in between. Training it is repeated calculus, and NumPy can do all of it. If you build a small network yourself once, every framework suddenly makes sense: tensors are arrays, layers are matrix products, and loss.backward() is the chain rule applied for you.

In this project you will teach a network the classic XOR problem: two classes arranged so that no straight line can separate them. You will generate the data, write the forward pass, add softmax and cross-entropy, derive backpropagation using matrix shapes as your guide, verify the gradients numerically, train in a loop, and draw the decision boundary the network learned. You should be comfortable with matrix multiplication and broadcasting.

The architecture

Our network has two layers. The input is a point (x1, x2). A hidden layer with 8 units computes a1 = tanh(X @ W1 + b1). The output layer computes scores z2 = a1 @ W2 + b2 for the two classes, and softmax turns the scores into probabilities. For a batch of n points the shapes are:

Array Shape Meaning
X (n, 2) input points
W1, b1 (2, 8), (8,) hidden layer weights and biases
a1 (n, 8) hidden activations
W2, b2 (8, 2), (2,) output layer weights and biases
p (n, 2) class probabilities, each row sums to 1
flowchart LR
    X["X<br/>(n, 2)"] --> Z1["z1 = X @ W1 + b1<br/>(n, 8)"]
    Z1 --> A1["a1 = tanh(z1)<br/>(n, 8)"]
    A1 --> Z2["z2 = a1 @ W2 + b2<br/>(n, 2)"]
    Z2 --> P["p = softmax(z2)<br/>(n, 2)"]
    P --> L["loss<br/>cross-entropy"]

The forward pass: two matrix products with a nonlinearity in between, then softmax and a loss.

The tanh between the two matrix products is essential. Without a nonlinearity, two stacked linear layers collapse into one linear layer, and a single line cannot solve XOR.

Step 1: XOR data

XOR means "exactly one of the inputs is 1". We scatter noisy points around the four corners of a unit square. Points near (0, 0) and (1, 1) get class 0; points near (0, 1) and (1, 0) get class 1. The two classes sit on the two diagonals, and no single line separates them.

Python Generate XOR blobs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

np.repeat(labels, 50) produces 50 zeros, 50 zeros, 50 ones, 50 ones, matching the order of the corners. Always check that labels and points line up; a shuffled mismatch makes learning impossible.

Step 2: forward pass, softmax and cross-entropy

Softmax turns a row of scores into probabilities: exp(z) / sum(exp(z)). A naive version overflows for large scores (np.exp(1000) is inf). The standard fix is to subtract the row maximum first, which does not change the result because the factor cancels. keepdims=True keeps the maximum as a column of shape (n, 1) so broadcasting subtracts it from each row.

Cross-entropy measures how surprised the model is by the true label: the average of -log(probability assigned to the correct class). We pick those probabilities with fancy indexing: p[np.arange(n), y].

Python The forward pass and the starting loss

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

An untrained network should have a loss near ln(2) = 0.693, which is what a model that answers "50/50" for everything would score. If your starting loss is far off (for example 5 or nan), something is wrong with the weights or the code. This quick check catches many bugs before training even starts.

Step 3: backpropagation with matrices

Training needs the gradient of the loss with respect to every weight. Backpropagation is the chain rule, applied from the output backwards. The surprise is how clean softmax plus cross-entropy is: the gradient of the loss with respect to the scores z2 is simply

TEXT
dz2 = (p - Y) / n        (Y is the one-hot version of the labels)

From there, each step is guided by shapes. A gradient always has the same shape as the thing it belongs to:

  • dW2 = a1.T @ dz2 has shape (8, n) @ (n, 2) = (8, 2), the shape of W2.
  • db2 = dz2.sum(axis=0) has shape (2,): the bias was broadcast over all rows, so its gradient is summed over them.
  • da1 = dz2 @ W2.T pushes the error back to the hidden layer, shape (n, 8).
  • dz1 = da1 * (1 - a1**2) multiplies by the derivative of tanh, which is 1 - tanh(z)**2.
  • dW1 = X.T @ dz1 and db1 = dz1.sum(axis=0).

How do you know your calculus is right? Compare with a numerical gradient: nudge one weight by a tiny amount, recompute the loss, and divide the change by the nudge. If the two agree, the backward pass is correct. This is called a gradient check, and it is the single most useful debugging tool when writing backpropagation by hand.

Python Backward pass and a numerical gradient check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every parameter group reports a difference below 1e-7, so the hand-written backward pass is correct. Notice the trick dz2[np.arange(n), y] -= 1: subtracting a one-hot matrix is the same as subtracting 1 at the correct class of each row, without building the one-hot matrix at all.

Step 4: the training loop and the decision boundary

Training is the same cycle as in linear regression: forward pass, loss, backward pass, update each parameter by -learning_rate * gradient. We loop over the whole data set each time (full-batch gradient descent), which is fine for 200 points. Weights start as random numbers; starting them all at zero would make every hidden unit identical and learn the same thing.

After training, we draw the decision boundary. We build a grid of points covering the plane with np.meshgrid, stack the grid into an (m, 2) array, run the network on all of it in one forward pass, and color each grid cell by the predicted class (see Creating Arrays).

Python Train the network and plot what it learned

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss drops from around 0.8 to nearly zero and the accuracy reaches 100 percent. The decision plot shows two diagonal bands, and the boundary curves between the clusters: something a single straight line could never do. The four corner predictions are [0 0 1 1], exactly XOR.

Ideas to extend the project

  • Replace tanh with ReLU: a1 = np.maximum(0, z1), derivative (z1 > 0).
  • Train on mini-batches by sampling rng.choice(n, 32) each step.
  • Make the data harder (two interleaving spirals) and increase hidden.
  • Add accuracy to the training log and a separate test set.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the network need tanh (a nonlinearity) between the two matrix products?

    A product of two matrices is just another matrix, so stacking linear layers adds no power. The nonlinearity lets the network bend its decision boundary.
  2. 2. Why do we subtract the row maximum inside softmax?

    Subtracting the same number from every score in a row cancels out in the ratio, but it keeps the exponentials small enough to avoid inf.
  3. 3. A bias b2 of shape (2,) is added to a (n, 2) array. What is the shape of its gradient db2?

    The bias was broadcast across n rows, so its gradient is the sum of the row gradients: dz2.sum(axis=0), which has shape (2,).
  4. 4. What does a gradient check tell you?

    By nudging a parameter and measuring the loss change, you get a slow but reliable numerical gradient. Agreement with your backpropagation code means the backward pass is almost certainly correct.

Summary

  • A tiny network is tanh(X @ W1 + b1) @ W2 + b2 followed by softmax; shapes guide every step.
  • Softmax needs the max-subtraction trick; cross-entropy is -log of the probability of the correct class, picked with fancy indexing.
  • Backpropagation applies the chain rule backwards: dz2 = p - Y, then dW = input.T @ dz, db = dz.sum(axis=0), and dz1 = (dz2 @ W2.T) * (1 - a1**2).
  • Verify gradients numerically before trusting a training loop.
  • Evaluate a network on a meshgrid to draw its decision boundary.

Next project: K-Means Clustering.