Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 73 ยท First Steps

The Artificial Neuron

Build an artificial neuron in NumPy: inputs, weights, bias, weighted sum and activation. Plot its decision boundary and run it on a whole batch at once.

The artificial neuron is the smallest building block of every neural network. It takes some numbers in, combines them into one number, and passes that through a simple function. That is all. A network with a billion parameters is made of exactly this unit repeated many times and wired together.

The name comes from a loose analogy with brain cells, but the analogy is not worth pushing. Think of a neuron as a tiny, adjustable decision-maker: it weighs several pieces of evidence and says "yes", "no" or "how much".

Inputs, weights and bias

A neuron with three inputs xโ‚, xโ‚‚, xโ‚ƒ owns three weights wโ‚, wโ‚‚, wโ‚ƒ and one bias b. It computes in two steps.

  1. The weighted sum (also called the pre-activation): z = wโ‚xโ‚ + wโ‚‚xโ‚‚ + wโ‚ƒxโ‚ƒ + b, or in vector form z = w ยท x + b.
  2. The activation: a = ฯ†(z), where ฯ† is an activation function.

Anatomy of a neuron

Inputs The numbers coming in, x.
Weights How much each input matters.
Bias A shift that sets the firing threshold.
Activation A function applied to the sum.

Weights and bias are the learned parameters; everything else is fixed.

1 / 2

The weights say how much each input matters and in which direction. The bias is a constant added regardless of the input. Without it, a neuron whose inputs are all zero would always output ฯ†(0); with it, the neuron can have a built-in tendency to fire or stay quiet.

flowchart LR
    x1["x1"] -->|"w1"| S(("ฮฃ + b"))
    x2["x2"] -->|"w2"| S
    x3["x3"] -->|"w3"| S
    S -->|"z"| A["activation ฯ†"]
    A --> O["output a"]

A neuron: weigh each input, add them with the bias, then apply the activation.

A neuron in NumPy

The activation we will use for now is the sigmoid, ฯƒ(z) = 1 / (1 + e^(โˆ’z)). It squashes any number into the range between 0 and 1, so the output can be read as a probability or a degree of "yes". Lesson Activation Functions compares it to the alternatives.

Python One neuron, step by step

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The weighted terms are [1.8, 1.5, 1.0], so z = 4.3 โˆ’ 1.5 = 2.8 and ฯƒ(2.8) โ‰ˆ 0.9427. The bias moves the whole output up or down without looking at the inputs at all.

The decision boundary

A neuron with two inputs divides the plane into two regions. The line where the output is exactly 0.5 (for a sigmoid, where z = 0) is the decision boundary. On one side the neuron leans "yes"; on the other, "no".

Setting z = wโ‚xโ‚ + wโ‚‚xโ‚‚ + b = 0 gives a straight line. The weights decide its direction (the weight vector is perpendicular to the line) and the bias decides its position. Increase the bias and the line slides away from the weight vector's direction.

This is also why a single neuron has limits: it can only draw one straight line. Later lessons stack neurons to build curved boundaries.

Python Plotting a neuron's decision boundary

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The thick black line is the boundary, and the arrow is the weight vector, perpendicular to it. Try changing b to 1.5 and rerun: the line moves toward the bottom left while keeping its tilt.

Vectorizing over a batch

Real training feeds many examples at once. Instead of looping over examples, stack them as the rows of a matrix X with shape (n_samples, n_features). Then one expression computes every output:

z = X @ w + b, then a = sigmoid(z)

The shapes are (n, d) @ (d,) giving (n,); the bias is a single number that broadcasts (is added to every entry). This is dramatically faster than a Python loop, because NumPy runs the arithmetic in compiled code.

With several neurons in a layer, w becomes a matrix W of shape (d, k) and b a vector of shape (k,). Then X @ W + b has shape (n, k): one column per neuron. The next lesson stacks such layers.

Python Loop versus vectorized batch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Keep the convention examples are rows, features are columns everywhere. It makes X @ W the natural way to write a layer, and every framework you will meet uses it.

The McCulloch-Pitts neuron: where it started

The first mathematical neuron came in 1943, from the neurophysiologist Warren McCulloch and the logician Walter Pitts. Their McCulloch-Pitts (MP) neuron is simpler than the one above, and it is worth knowing because every modern unit is a relaxed version of it:

  • All inputs and the output are binary (0 or 1).
  • There are no learned weights. Each excitatory input simply counts as 1.
  • An inhibitory input has absolute veto power: if any inhibitory input is 1, the neuron outputs 0 no matter what else arrives.
  • The neuron fires (outputs 1) when the number of active excitatory inputs reaches a fixed threshold ฮธ.

Different thresholds give different logic gates. With two inputs, ฮธ = 2 is AND (both must be on) and ฮธ = 1 is OR (at least one). Make x2 inhibitory with ฮธ = 1 and you get "x1 AND NOT x2". Since AND, OR and NOT are enough to build any Boolean function, a network of MP neurons can compute anything a digital circuit can. What the MP neuron cannot do is learn: a person has to choose the threshold and the wiring by hand. The perceptron fixed that by adding real-valued weights that are adjusted from examples, and the modern neuron added a smooth activation so that gradient descent works.

Python McCulloch-Pitts gates and XOR

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The search in the middle of that program finds no single unit that computes XOR, which is the limit the next section explains. A second layer fixes it, just as it does for modern networks.

What the neuron cannot do yet

Everything here is fixed: someone chose w and b by hand. The point of deep learning is to learn them. The next lesson, The Perceptron Learning Rule, shows the oldest algorithm for doing that, and why a single neuron is not enough for every problem.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the correct pre-activation value z of a neuron?

    z = w . x + b. The activation function is applied to z afterwards.
  2. 2. For a neuron with two inputs, what does the bias change about the decision boundary?

    The weights set the direction of the line. The bias shifts it.
  3. 3. X has shape (200, 5) and w has shape (5,). What does X @ w + b produce?

    Each of the 200 rows is dotted with w, giving one pre-activation per example. The scalar bias broadcasts.
  4. 4. Why is a single sigmoid neuron unable to separate two moon-shaped clusters?

    z = 0 defines a straight line (a plane in higher dimensions), so curved boundaries need several neurons in several layers.

Summary

  • A neuron computes a weighted sum of its inputs plus a bias, then applies an activation function.
  • Weights set how strongly and in which direction each input matters; the bias shifts the threshold.
  • With two inputs, the decision boundary w ยท x + b = 0 is a straight line: weights set its direction, the bias its position.
  • Write batches as rows and compute sigmoid(X @ w + b) instead of looping.
  • A layer of neurons just replaces w by a matrix W and b by a vector.
  • A single neuron can only draw straight boundaries.

Next: The Perceptron Learning Rule.