Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 45 ยท Building Models

Neurons, Perceptrons and Feedforward Networks

Learn the ideas behind every PyTorch model: the artificial neuron, the perceptron and its limits, multi-layer perceptrons, forward propagation and the main network types.

Before you write nn.Module classes it helps to know what a neural network actually is. At heart it is a pile of very simple units, each one multiplying its inputs by weights, adding a bias and passing the result through a non-linear function. Stack enough of them and the pile can recognize faces, translate sentences or predict prices. This lesson starts from a single neuron, builds the classic perceptron, shows why one layer is not enough, and then follows a signal through a multi-layer perceptron (MLP) in a forward pass.

PyTorch cannot run in the browser, so the PyTorch snippets are read-only. The live editors use NumPy to do the same arithmetic by hand, so every number you see is real, but it is a stand-in for the library, not the library itself.

From a biological neuron to an artificial one

A biological neuron collects electrical signals through its dendrites, sums them in the cell body and, if the total is strong enough, fires a signal down its axon. The earliest mathematical model, the McCulloch-Pitts neuron (1943), copied exactly that: binary inputs are added up and the neuron outputs 1 when the sum reaches a threshold, otherwise 0. It had no learning, but it showed that simple threshold units can compute logic.

A modern artificial neuron keeps the idea and adds two upgrades: real-valued weights that say how much each input matters, and a bias that shifts the firing point.

Text
output = activation(w1*x1 + w2*x2 + ... + wn*xn + b)

In PyTorch a whole layer of such neurons is one nn.Linear followed by an activation function (see Activation Functions and Loss Functions). The analogy with the brain is loose; do not read more into it than "units that sum weighted inputs and fire".

The perceptron

Frank Rosenblatt's perceptron (1958) is one neuron with a step activation and, crucially, a learning rule. Its parts are:

  • Inputs x, the features of one example.
  • Weights w and a bias b, the learnable numbers.
  • A weighted sum z = w . x + b.
  • A step function that outputs 1 if z >= 0 and 0 otherwise.

Training is a loop over the examples. For each one, compute the prediction, compare it to the target, and nudge the weights only if it was wrong:

Text
w = w + lr * (target - prediction) * x
b = b + lr * (target - prediction)

If the prediction was too low the weights move toward the input, if too high they move away, and if correct nothing changes. When the classes can be separated by a straight line (or a flat plane in higher dimensions), this rule is guaranteed to find a separating line in a finite number of steps.

The PyTorch version uses a single linear layer; the step function has no useful gradient, so in practice you train with a smooth function and a loss instead:

Python
import torch.nn as nn

perceptron_like = nn.Sequential(nn.Linear(2, 1), nn.Sigmoid())   # one neuron, smooth output
Python A perceptron learns AND, OR and fails at XOR

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

AND and OR are linearly separable, so the perceptron reaches 100 percent. XOR is not: no single straight line puts (0,0) and (1,1) on one side and (0,1) and (1,0) on the other. In 1969 Minsky and Papert pointed this limitation out, and it slowed neural network research for years. The fix is to add a layer.

Multi-layer perceptrons

A multi-layer perceptron places one or more hidden layers between the input and output. Each hidden layer is a set of neurons that all see the same inputs; their outputs become the inputs of the next layer. The hidden layer builds new features (for XOR: "at least one input is on" and "both inputs are on"), and the output layer separates the classes in that new feature space.

The non-linear activation between layers is essential. Without it, two stacked linear layers collapse into a single linear layer (matrix times matrix is still a matrix), and the network is no more expressive than a perceptron.

Perceptron Multi-layer perceptron
Layers One neuron layer At least one hidden layer
Activation Step ReLU, tanh, sigmoid and others
Learns Linearly separable problems Non-linear problems (XOR and far beyond)
Training Perceptron rule Backpropagation with gradient descent

The universal approximation theorem says an MLP with one hidden layer and enough neurons can approximate any continuous function on a bounded region to any accuracy. It says nothing about how many neurons or how to find the weights, which is why deeper networks and good training habits matter in practice.

Forward propagation

Forward propagation is the journey of an input through the network to the output. For a layer with weights W, bias b and activation f, the step is a = f(W x + b), and the output a is the input of the next layer. For a batch, x becomes a matrix with one row per example, and the layer is f(X @ W.T + b), exactly what nn.Linear does.

Python
import torch
import torch.nn as nn

mlp = nn.Sequential(
    nn.Linear(2, 2), nn.ReLU(),     # hidden layer: 2 inputs -> 2 neurons
    nn.Linear(2, 1), nn.Sigmoid(),  # output layer: 2 neurons -> 1 probability
)
y = mlp(torch.tensor([[0., 1.]]))   # a forward pass on one example

The forward pass computes the prediction and stores intermediate values; the backward pass then uses them to compute gradients (see Computational Graphs and the Chain Rule). The next editor does a forward pass for XOR with weights chosen by hand, so you can see the hidden layer solving what a single neuron could not.

Python Forward pass of a two-layer network solving XOR

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In a real network you do not choose weights by hand: you start from random values and let gradient descent find them, as in Your First Model: Linear Regression and The Training Loop.

Shallow versus deep networks

A shallow network has one (or very few) hidden layers; a deep network has many. Shallow networks are quick to train, need little data and are easy to inspect, so they are still a good choice for small tabular problems and simple regressions. Their weakness is that representing complicated functions may need an enormous hidden layer. Deep networks reuse earlier layers: each layer builds on the features of the previous one, from edges to shapes to objects in an image. That hierarchical feature learning is why depth wins on images, speech and text, at the price of more data, more compute and harder optimization.

Fully connected layers

A fully connected (dense) layer connects every input to every output neuron: nn.Linear(in_features, out_features) has in * out weights plus out biases. It is the most flexible layer (any input can influence any output) and also the most expensive, because the parameter count grows with the product of sizes. A 224 by 224 RGB image flattened into 150,528 numbers feeding a 1,000-neuron layer needs about 150 million weights. That is why convolutional layers (Convolutions and CNNs) take over early in image models and dense layers stay for the end, where they turn features into class scores.

Dense layers cannot handle variable-sized input, ignore the spatial and sequential structure of the data, and overfit easily without regularization.

The feedforward network

An MLP is the simplest feedforward neural network: information moves one way, input to output, with no loops. This is the right model for fixed-size inputs such as a row of a table: credit scoring, price prediction, simple classification. When the input has order (text, audio) you need recurrence or attention (Recurrent Networks, Attention), and when it has a grid layout (images) you want convolutions.

Network families at a glance

Network Idea Typical use
Feedforward / MLP Dense layers, one direction Tabular data, simple tasks
Convolutional (CNN) Shared local kernels Images, video, audio spectrograms
Recurrent (RNN, LSTM, GRU) A hidden state carried through time Sequences, time series
Transformer Attention between all positions Text, vision, almost everything now
Autoencoder Compress then reconstruct Denoising, anomaly detection (Autoencoders)
GAN A generator and a discriminator compete Image generation (GANs)
Radial basis function network Neurons respond to distance from a center Function approximation, small classifiers
Self-organizing map Competitive learning on a grid Clustering, visualization
Deep belief network Stacked restricted Boltzmann machines Historic pretraining method
Siamese network Two twin branches with shared weights Similarity: faces, signatures, duplicates
Capsule network Groups of neurons encode pose and presence Research on viewpoint-robust vision
Spiking network Neurons communicate with discrete spikes in time Energy-efficient neuromorphic hardware

Only the first six are common in everyday PyTorch work; the rest are worth knowing by name because you will meet them in papers and interviews. A Siamese network is easy to build in PyTorch: apply the same module to two inputs and compare the embeddings.

Python
class Siamese(nn.Module):
    def __init__(self, encoder):
        super().__init__()
        self.encoder = encoder                     # the SAME weights for both inputs

    def forward(self, a, b):
        return nn.functional.pairwise_distance(self.encoder(a), self.encoder(b))

Learning paradigms

How a network learns depends on the data it gets. In supervised learning every example has a label and the loss measures the gap between prediction and label. In unsupervised learning there are no labels; the network finds structure itself (autoencoders, clustering). In reinforcement learning an agent acts in an environment and learns from rewards rather than labeled answers. Most of this track is supervised, with unsupervised ideas appearing in the generative lessons.

Advantages and limitations

Neural networks learn features automatically, handle messy high-dimensional input, and improve with more data and compute. They also need lots of labeled data, are expensive to train, behave like black boxes that are hard to explain, and can overfit. They are not always the best tool: for a small table, gradient-boosted trees often win.

Note: A "layer count" is not standardized. Some people count the input layer, some count only layers with weights. In this track, "a two-layer network" means two weight layers (nn.Linear modules).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why can a single perceptron not learn XOR?

    A perceptron draws one straight line. XOR's two classes cannot be split by any single line, so a hidden layer is needed.
  2. 2. What happens if you stack linear layers with no activation function between them?

    The product of weight matrices is just another matrix, so without non-linearities extra layers add nothing.
  3. 3. What does forward propagation compute?

    Forward propagation applies each layer in turn. Gradients are computed afterwards in the backward pass.
  4. 4. How many parameters does nn.Linear(100, 10) have?

    100 * 10 = 1,000 weights plus 10 biases gives 1,010.
  5. 5. What is the defining trait of a Siamese network?

    Both inputs are encoded by the same weights so their embeddings can be compared directly.

Summary

  • A neuron computes an activation of a weighted sum plus a bias; a layer is many neurons, nn.Linear plus an activation.
  • The perceptron learns linearly separable problems with a simple update rule but cannot learn XOR.
  • Hidden layers with non-linear activations make multi-layer perceptrons universal approximators.
  • Forward propagation applies the layers in order; backpropagation then computes gradients.
  • Dense layers are flexible but parameter-hungry; CNNs, RNNs and transformers add structure for images and sequences.
  • Know the family names (RBF, SOM, DBN, Siamese, capsule, spiking) even if you rarely build them.

Next: choose the non-linearities and the loss in Activation Functions and Loss Functions.