Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 73 ยท Neural Networks From Scratch

Your First Trained Network

Train your first two-layer neural network in NumPy on XOR and the two-moons dataset: full training loop, loss and accuracy curves, and decision boundaries.

Everything is ready. You have layers and the forward pass, a loss, a gradient-descent update and a verified backward pass. In this lesson you put them together and train a network that does something a single neuron provably cannot: solve XOR, and then separate two interleaved moons.

You will also write the thing that every training script is made of: the loop. Forward, loss, backward, update, record, repeat. Once you have written it by hand, a framework's fit() or loss.backward() will look very familiar.

The recipe

Initialize Small random weights, zero biases.
Forward Inputs through the layers to a prediction.
Loss Compare prediction and label.
Backward Gradients for every parameter.

Then step downhill and repeat for hundreds of iterations.

1 / 2

The building blocks

The classes below are the same as in Backpropagation in NumPy, plus a Tanh layer and a small Network wrapper that holds the list of layers and does the forward and backward loops. Every later example reuses them.

Python
class Network:
    def __init__(self, layers):
        self.layers = layers

    def forward(self, X):
        for layer in self.layers:
            X = layer.forward(X)
        return X

    def backward(self, grad):
        for layer in reversed(self.layers):
            grad = layer.backward(grad)

    def step(self, lr):
        for layer in self.layers:
            if hasattr(layer, "W"):
                layer.W -= lr * layer.dW
                layer.b -= lr * layer.db

The loss is kept outside the network: it needs the labels, and it starts the backward pass. That mirrors how frameworks separate the model from the criterion.

XOR: the problem a single neuron cannot solve

XOR ("exclusive or") outputs 1 when exactly one of its two inputs is 1. Plotted on a plane, the points (0,0) and (1,1) belong to class 0 and (0,1) and (1,0) to class 1. No straight line separates them, which is the famous limit of the single-layer perceptron. A hidden layer fixes this: each hidden unit draws one line, and the output unit combines the two lines.

Here is a complete network with two inputs, four tanh hidden units and one sigmoid output, trained with binary cross-entropy. For a sigmoid output with BCE, the gradient at the pre-activation is simply (p - y) / N.

Python Training a network to solve XOR

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss starts at about 1.4 (this seed begins with confidently wrong guesses; an undecided network would sit near log 2 = 0.693) and drops to about 0.003, and the four predictions land close to 0, 1, 1, 0. The right-hand plot shows how: the network folded the plane so that the two diagonal corners share a region.

Training can occasionally stall for a while in a plateau at a loss near 0.69 before it breaks out, and a different random seed can behave differently. Small networks on tiny problems are sensitive to their starting weights. Weight Initialization explains why.

Two moons

XOR has four points. A more realistic toy is make_moons from scikit-learn: two crescents of noisy points that interlock, which no line can split. We will use a larger network, softmax with cross-entropy for the output (two classes, two logits), and the standard discipline of evaluating on held-out data.

First, split the data into a training set, which the network learns from, and a test set, which it never sees during training. Accuracy on the test set tells you how well the network generalizes. We track both losses and both accuracies at every step.

Python Training on the two-moons dataset

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Even a randomly initialized ReLU network bends the plane a little, so test accuracy starts at 84 percent, then climbs to about 95 percent while the training loss keeps falling. The decision boundary is a curve that follows the gap between the crescents, something a linear model cannot draw. Because the test points were never used for the updates, their accuracy is an honest estimate of how the network will behave on new data.

Read the three plots together. If the training loss falls but the test loss starts rising, the network is memorizing noise, a problem called overfitting that gets its own lesson later. If both are flat and high, the model is too small or the learning rate is off.

How big should the hidden layer be?

The hidden layer's width controls how flexible the boundary can be. Train the same network with 1, 2, 4 and 32 hidden units and compare the test accuracy.

Python Effect of the hidden layer width

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With one to four hidden ReLU units the boundary can only bend a few times, and training accuracy stays near 86 percent, close to what a straight line achieves. With 32 units the network has enough pieces to trace both crescents and reaches about 97 percent. More units is not always better: a very wide network on very little data can fit noise. Choosing capacity is a balancing act you will keep practicing.

What you built

Take a moment to appreciate the pieces. About 40 lines of NumPy define layers with forward and backward passes, a loss, and a training loop, and that is already enough to learn non-linear decision boundaries. Everything in the rest of the tutorial is an improvement to one of these parts: better initialization, smarter optimizers, regularization, convolutions, attention.

Next, make training more reliable by choosing starting weights well in Weight Initialization.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does XOR need a hidden layer?

    XOR is not linearly separable. Hidden units each draw a line, and the output unit combines them into a non-linear boundary.
  2. 2. What is the purpose of the held-out test set?

    Accuracy on training data can be inflated by memorization. Held-out data gives an honest estimate of generalization.
  3. 3. What loss value should a freshly initialized 2-class network start near?

    A network that starts out predicting about 50/50 gets -log(0.5) = log 2 for each example.
  4. 4. What is the correct order of one training iteration?

    Predict first, score the prediction, compute the gradients from that score, and only then change the parameters.
  5. 5. In the width experiment, why did a network with one hidden unit perform poorly on the two moons?

    Capacity limits the shapes the network can draw. With few units it behaves almost like a line.

Summary

  • A training script is a loop of forward pass, loss, backward pass and parameter update, with metrics recorded at each step.
  • A network with one hidden layer solves XOR and the two-moons problem, which no single neuron can.
  • Split the data: train on one part and report accuracy on a held-out test set.
  • Loss curves, accuracy curves and a decision-boundary plot together tell you whether training is healthy.
  • Hidden-layer width controls flexibility; too few units underfit.

Next lesson: Weight Initialization.