Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 73 ยท First Steps

The Perceptron Learning Rule

Implement the perceptron algorithm in NumPy, teach it AND and OR, watch its decision boundary move, and see why XOR is impossible for a single neuron.

In the previous lesson someone had to choose the weights by hand. The perceptron, invented by Frank Rosenblatt in the late 1950s, is the first algorithm that finds them automatically. It is simple enough to write in ten lines, and it already contains the core idea of all neural network training: look at a mistake, then nudge the parameters to correct it.

It also has a famous limitation, which in 1969 helped cool research on neural networks for years. Understanding exactly why it fails on one tiny problem, XOR, motivates everything in the chapters that follow.

The perceptron model

A perceptron is a neuron whose activation is a step function: it outputs 1 if the weighted sum is zero or more, and 0 otherwise.

ลท = 1 if w ยท x + b โ‰ฅ 0, else 0

Labels are 0 or 1. The perceptron is therefore a classifier that draws a straight line (or hyperplane) and says "1" on one side and "0" on the other.

The learning rule

Go through the training examples one at a time. For each example compute the prediction ลท and compare with the true label y:

w โ† w + ฮท (y โˆ’ ลท) x and b โ† b + ฮท (y โˆ’ ลท)

Here ฮท is the learning rate, a small positive number. Look at what the rule does:

  • If the prediction is right, y โˆ’ ลท = 0 and nothing changes.
  • If the neuron said 0 but the answer is 1, y โˆ’ ลท = +1, so x is added to w. The weight vector turns toward x, which raises w ยท x next time.
  • If the neuron said 1 but the answer is 0, y โˆ’ ลท = โˆ’1, so x is subtracted, lowering w ยท x.

One pass of the rule

Predict Compute w ยท x + b and apply the step.
Compare Error is label minus prediction: -1, 0 or +1.
Correct Add or subtract the input from the weights.
Repeat Loop over the data until no mistakes remain.

One full loop over the data is called an epoch.

1 / 2

One loop over all the examples is an epoch. Training repeats epochs until an epoch has no mistakes (or a limit is reached).

Learning AND and OR

AND outputs 1 only when both inputs are 1. OR outputs 1 when at least one is. Each has four rows in its truth table, and each is linearly separable: a single straight line can put the 1s on one side and the 0s on the other.

Python A perceptron learns AND and OR

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each gate is learned within a handful of epochs. The weights it ends with are not unique: many lines separate the same points, and which one you get depends on the starting values and the order of the data.

Watching the boundary move

The best way to see learning is to draw the decision boundary after each epoch. On a two-dimensional dataset we can plot the line wโ‚xโ‚ + wโ‚‚xโ‚‚ + b = 0 and watch it turn and slide until it splits the classes.

Python The decision boundary during training

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The printout shows 2 mistakes in the first epoch and none in the second, and the plot shows the lines: the starting guess cuts through the wrong place, one correction later the line sits between the two clouds. With classes closer together you would see more epochs of zig-zagging before the mistakes reach zero.

Does it always converge?

The perceptron convergence theorem gives a guarantee: if the data are linearly separable, the perceptron rule finds a separating line in a finite number of mistakes. The bound depends on how far apart the classes are (the margin) and how large the inputs are, not on how many examples there are. Well-separated data converge fast; classes that nearly touch need many more corrections.

If the data are not linearly separable, there is no line with zero mistakes, so the algorithm never stops on its own. The weights keep jumping around. That is why the code uses a maximum number of epochs.

Why XOR is impossible

XOR ("exclusive or") outputs 1 when the inputs differ. Plot the four points and try to draw one straight line with (0,1) and (1,0) on one side and (0,0) and (1,1) on the other. You cannot: the two 1s sit on one diagonal and the two 0s on the other.

You can also prove it with the neuron formula. We would need b < 0 (for input 0,0), wโ‚ + b โ‰ฅ 0 and wโ‚‚ + b โ‰ฅ 0 (for the two 1s), and wโ‚ + wโ‚‚ + b < 0 (for input 1,1). Adding the middle two gives wโ‚ + wโ‚‚ + 2b โ‰ฅ 0, so wโ‚ + wโ‚‚ + b โ‰ฅ โˆ’b > 0, which contradicts the last requirement.

Python The perceptron cannot learn XOR

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

However many epochs you allow, the mistake count never reaches zero (in this run it is still 4 at epochs 3, 10, 50 and 100, and the final weights get only 2 of the 4 predictions right). XOR is not an exotic puzzle: it is the simplest example of a pattern that needs more than one line.

The way out

The fix, which you build in the next chapter, is to use several neurons in layers. One hidden neuron can compute "OR", another "AND", and a final neuron combines them into "OR but not AND", which is XOR. The catch is that the perceptron rule does not know how to train hidden neurons, because the step function has zero slope almost everywhere. We need smooth activations and the chain rule: backpropagation.

Rosenblatt's perceptron was used with a physical machine, the Mark I Perceptron, to recognize simple images. The criticism that single-layer perceptrons cannot learn XOR was accurate, but it was widely misread as a limit on all neural networks. Multi-layer networks trained by backpropagation removed it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A perceptron predicts 0 but the true label is 1. What happens to the weights?

    The error y - y_hat is +1, so w becomes w + eta * x, turning the weight vector toward the input and raising w . x next time.
  2. 2. When is the perceptron guaranteed to converge?

    The convergence theorem applies to linearly separable data. Otherwise no line has zero mistakes.
  3. 3. Which of these functions can a single perceptron NOT represent?

    XOR is not linearly separable: no straight line puts (0,1) and (1,0) apart from (0,0) and (1,1).
  4. 4. One full pass of the training loop over the whole dataset is called...

    An epoch is one complete pass over the training data.

Summary

  • The perceptron is a neuron with a step activation, trained by w โ† w + ฮท (y โˆ’ ลท) x.
  • Correct predictions change nothing; mistakes pull the boundary toward the misclassified point.
  • On linearly separable data (AND, OR, many real datasets) the algorithm provably converges.
  • XOR is not linearly separable, so a single perceptron can never learn it.
  • Hidden layers fix the limitation, but need smooth activations and backpropagation to train.

Next: Layers and the Forward Pass.