Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 40 ยท Getting Started

How Neural Networks Learn

Understand neurons, layers, loss and gradient descent, then train a tiny network on XOR with a NumPy stand-in before using Keras's compile and fit.

A neural network is a stack of simple functions with adjustable numbers. Learning is a loop: make predictions, measure the error, work out how each number contributed to it, and change every number a little in the direction that reduces the error. Keras automates this loop, but understanding it makes every later choice (learning rate, loss, batch size) far less mysterious.

This lesson builds that understanding with a small network on the classic XOR problem. The NumPy code is a stand-in for Keras; the real Keras equivalent follows it.

Neurons and layers

A neuron takes numbers in, multiplies each by a weight, adds a bias and passes the sum through a nonlinear activation:

output = activation(w1*x1 + w2*x2 + ... + b)

A layer is many neurons side by side, which is just a matrix multiplication: activation(X @ W + b). A network stacks layers so the output of one feeds the next. The nonlinearity matters. Without it, any stack of layers collapses into a single linear function, because a product of matrices is itself one matrix.

flowchart LR
    I["Input x"] --> L1["Layer 1<br/>relu(xW1 + b1)"]
    L1 --> L2["Layer 2<br/>sigmoid(hW2 + b2)"]
    L2 --> P["Prediction"]
    P --> Loss["Loss vs. target"]

Data flows forward; the loss measures the final error.

The four ingredients of learning

  1. Forward pass: compute predictions from the current weights.
  2. Loss: reduce all errors to one number. Smaller is better.
  3. Gradients: for each weight, the slope of the loss, telling which direction increases the loss fastest. Backpropagation computes all of them efficiently by applying the chain rule backwards through the layers.
  4. Update: move each weight a small step against its gradient: w = w - learning_rate * gradient.

Repeat for many batches and epochs. This is gradient descent, and every optimizer in Keras is a refinement of it.

One training step

Forward Compute predictions.
Loss Measure the error.
Backward Compute gradients.
Update Nudge the weights.

Repeat thousands of times.

1 / 2

Why XOR?

XOR outputs 1 when the two inputs differ and 0 when they are equal. No single straight line separates the 1s from the 0s, so a model with no hidden layer cannot solve it. A tiny network with one hidden layer can. It is the smallest problem that proves you need nonlinearity and depth.

A network that learns XOR (NumPy stand-in)

This is a full training loop: forward pass, loss, hand-written backpropagation and updates. Study it; the next chapters turn each part into a reusable idea.

Python Stand-in: training a 2-4-1 network on XOR (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss drops steadily and the rounded predictions become [0 1 1 0]. Try changing lr to 0.01 and watch learning crawl, or to 20 and watch it become unstable. The learning rate is the most important setting in any training run.

The same network in Keras (read-only)

Python
import numpy as np
import keras
from keras import layers

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype="float32")
y = np.array([0, 1, 1, 0], dtype="float32")

model = keras.Sequential([
    keras.Input(shape=(2,)),
    layers.Dense(4, activation="tanh"),
    layers.Dense(1, activation="sigmoid"),
])

model.compile(optimizer=keras.optimizers.SGD(learning_rate=0.5),
              loss="mse")

model.fit(X, y, epochs=2000, batch_size=4, verbose=0)
print(model.predict(X).round(2))

Everything you wrote by hand has a Keras name: the Dense layers hold W and b, loss="mse" is the loss, SGD is the update rule and fit is the loop. Exact numbers vary with initialization, and plain SGD on XOR can occasionally get stuck for an unlucky start; rerunning with a different seed or using "adam" usually fixes it.

Overfitting: the one thing that can go wrong

A network with enough weights can memorize its training examples, including noise, and then fail on new data. That is overfitting. This is why we always keep data aside the model never trains on (validation and test sets). The idea is explored in Metrics and Validation and Overfitting and Callbacks.

Key vocabulary

  • Epoch: one pass over the training data.
  • Batch: a small group of examples used for one update.
  • Learning rate: the step size of the update.
  • Hyperparameter: a setting you choose (layers, units, learning rate), as opposed to a parameter the model learns (weights, biases).
  • Gradient: the slope of the loss with respect to a parameter.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does backpropagation compute?

    Backpropagation applies the chain rule backwards to get every weight's gradient efficiently.
  2. 2. Why can't a model without a hidden layer (or nonlinear activation) solve XOR?

    One straight line cannot split XOR's classes. Nonlinear hidden units make it possible.
  3. 3. What happens if the learning rate is far too large?

    Oversized steps jump past the minimum, so the loss may oscillate or blow up.
  4. 4. Which is a hyperparameter?

    You choose the number of units before training. Weights and biases are learned parameters.

Summary

  • A layer is activation(X @ W + b); a network stacks layers, and nonlinearity makes depth useful.
  • Learning loops over forward pass, loss, backpropagation and an update step.
  • XOR shows why hidden layers matter; the NumPy stand-in trains one end to end.
  • Keras names each part: Dense, loss, optimizer, fit.
  • Overfitting is the main risk, so keep unseen data for validation.

Next: The Keras Workflow and Backends