Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 43 ยท Models of Cognition

Connectionism: Networks of Simple Units

Understand artificial neurons, perceptrons, why XOR needs a hidden layer, how backpropagation learns, and the debate over whether networks explain language.

A single neuron in your brain is not clever. It adds up signals from other neurons and, if the total is large enough, fires. Yet billions of them together recognize faces, read this sentence and plan tomorrow. Connectionism takes that observation seriously: build models from many simple, neuron-like units, let them influence each other through weighted connections, and see how far learning can take you.

This lesson explains how one unit computes, how layers of units learn, and why the approach reshaped cognitive science in the 1980s. It also covers the strongest objections, because they are part of the story.

Four steps: inputs, weights, sum plus bias, activation.
One artificial unit multiplies inputs by weights, adds a bias and passes the total through an activation function.

What you will learn

  • How an artificial unit turns inputs into an output.
  • What a perceptron can learn and why it cannot learn XOR.
  • How a hidden layer and backpropagation overcome that limit.
  • What distributed representations and graceful degradation mean.
  • What the past-tense debate and the systematicity objection argued.

Key terms

Unit: a simple processing element with an activation value.

Weight: the strength of the connection between two units, which can be positive (excitatory) or negative (inhibitory).

Hidden layer: units between input and output that learn their own internal representations.

Backpropagation: a method that adjusts weights by passing the output error backward through the network.

Distributed representation: a concept represented by a pattern across many units rather than by one unit.

From neurons to units

In 1943 Warren McCulloch and Walter Pitts showed that networks of simplified threshold neurons could compute logical functions. In 1958 Frank Rosenblatt built the perceptron, a unit with adjustable weights and a learning rule: when the output is wrong, nudge each weight in the direction that would have reduced the error. A perceptron computes

output = 1 if (w1*x1 + w2*x2 + ... + bias) > 0, otherwise 0

and learns by w = w + learning_rate * (target - output) * x. If the classes can be separated by a straight line (or flat plane), this procedure is guaranteed to find a separator.

The XOR problem

In 1969 Marvin Minsky and Seymour Papert's book Perceptrons proved that a single-layer perceptron cannot compute functions that are not linearly separable. The classic example is XOR (exclusive or): output 1 when exactly one of the two inputs is 1. No straight line separates {(0,1), (1,0)} from {(0,0), (1,1)}.

The result is often told as if it killed neural network research. A fairer reading: the book was right about single-layer networks, and it reduced funding and interest for years, even though people knew multi-layer networks could in principle do more. What was missing was a practical way to train them.

Hidden layers and backpropagation

Add a hidden layer and XOR becomes easy: one hidden unit can detect "at least one input on", another "both on", and the output says "first but not second". The question is how to learn those hidden weights. Backpropagation, which uses the chain rule of calculus to compute how each weight contributed to the error, was derived several times. It became widely known after David Rumelhart, Geoffrey Hinton and Ronald Williams published it in 1986, and the two-volume Parallel Distributed Processing books from Rumelhart, James McClelland and the PDP Research Group launched connectionism as a movement.

Training works by repetition. Show an example, compute the error, adjust the weights slightly, repeat thousands of times. The hidden units end up detecting features nobody programmed.

What made networks attractive as models of mind

  • Distributed representations. A concept is a pattern across many units. Similar concepts get similar patterns, so generalization is natural.
  • Graceful degradation. Remove a few units and performance drops slightly, like a person with partial brain damage, rather than crashing.
  • Learning. The same mechanism acquires knowledge from examples, which suits perception and language acquisition.
  • Soft constraints. Many weak influences are combined, as when you resolve an ambiguous letter using its neighbors.

Famous models include McClelland and Rumelhart's interactive activation model of letter perception (1981) and NETtalk (Sejnowski and Rosenberg, 1987), which learned to pronounce English text.

The past-tense debate

In 1986 Rumelhart and McClelland trained a network to produce past-tense English verbs. It showed a U-shaped learning curve similar to children, who first say "went", later overregularize to "goed" and finally recover. The authors argued no explicit rule was needed.

Steven Pinker and Alan Prince replied in 1988 with detailed criticisms of the model's design and results, and argued for a hybrid: rules for regular verbs, memory (possibly associative) for irregulars. McClelland and others responded with improved networks. The debate has continued for decades, and it is a model case of how modeling pushes both sides to be more precise.

The systematicity objection

Jerry Fodor and Zenon Pylyshyn argued in 1988 that human thought is systematic: anyone who can think "John loves Mary" can think "Mary loves John", and that this requires structured, combinable representations, which plain networks do not guarantee. Connectionists replied with architectures that do handle structure. Modern neural language models show surprising compositional behavior, though experiments still find systematic failures. The honest summary: the question is empirical and unsettled.

Cards comparing symbolic and connectionist models.
Two answers to the question of what knowledge is.

Common misconceptions

"Artificial neural networks are accurate copies of the brain." They borrow an idea from neurons but leave out most of the biology: neurotransmitter types, spikes, dendritic processing and more. Their strength is as abstract models of learning.

"Minsky and Papert proved neural networks do not work." They proved limits of single-layer perceptrons. Multi-layer networks escape those limits.

Try it

Python A perceptron learns AND and OR but not XOR

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python A hidden layer solves XOR (backpropagation)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Play with one unit

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does a perceptron do when its output is wrong?

    The perceptron rule nudges each weight by learning rate times error times input.
  2. 2. Why can a single-layer perceptron not learn XOR?

    No straight line separates the XOR classes, and a single-layer perceptron can only draw a straight line.
  3. 3. Which pair is credited with popularizing backpropagation in 1986?

    Rumelhart, Hinton and Williams published the influential 1986 account, though the method had earlier roots.
  4. 4. What is graceful degradation?

    Because knowledge is distributed, losing a few units causes a small drop instead of total failure.
  5. 5. What was the core of Fodor and Pylyshyn's 1988 objection?

    They argued that systematic, combinable thought requires structured representations, which plain networks do not guarantee.

Summary

  • Connectionist models use many simple units joined by weighted connections.
  • A single-layer perceptron learns only linearly separable tasks, so it fails on XOR.
  • Hidden layers plus backpropagation allow networks to learn their own features.
  • Distributed representations give generalization and graceful degradation.
  • Debates over the past tense and systematicity are still instructive.

Next: The Bayesian brain