Skip to content
Tutorials Center
My learning

Topics

Lesson 46 of 73 ยท Generative and Other Models

Classic Unsupervised Networks

Meet three classic networks that learn without labels: restricted Boltzmann machines, self-organizing maps and Hopfield memories, each built in NumPy.

Most of this course trained networks with labels and backpropagation. Before that recipe took over, researchers built other kinds of networks, inspired by physics and by the brain. They learn without labels, and each one answers a different question. A restricted Boltzmann machine learns a probability model of the data. A self-organizing map squeezes data onto a small grid so that neighbors on the grid are similar. A Hopfield network stores patterns and recalls the whole pattern from a damaged piece of it.

These networks are rarely the best tool today, but they are worth an hour. RBMs were stacked into the first deep networks that trained well (deep belief networks, around 2006), Hopfield networks introduced the idea of an energy landscape that still shows up in modern research, and the self-organizing map remains a handy way to look at high-dimensional data. Each section below has a compact NumPy implementation. For the modern descendants, see Autoencoders and VAEs and GANs.

Three networks, three jobs

RBM Learns a probability model of binary data.
SOM Maps data onto a grid that keeps neighbors close.
Hopfield Stores patterns and recalls them from noisy cues.

None of them needs a label.

1 / 2

Restricted Boltzmann machines

An RBM has two layers of binary units: visible units v that hold the data, and hidden units h that learn features. Every visible unit connects to every hidden unit through a weight matrix W, and there are no connections inside a layer (that is the "restricted"). Each layer also has biases, a for visible and b for hidden.

The network defines an energy for every pair (v, h):

TEXT
E(v, h) = -a.v - b.h - v.W.h

and a probability p(v, h) proportional to exp(-E): low-energy configurations are likely. Training adjusts W, a and b so that the data has low energy. The restriction makes the conditionals easy. Given v, the hidden units are independent:

TEXT
p(h_j = 1 | v) = sigmoid(b_j + v.W[:, j])
p(v_i = 1 | h) = sigmoid(a_i + W[i, :].h)

Computing the exact gradient of the likelihood is intractable, so Hinton's contrastive divergence (CD-1) approximates it with one short back-and-forth:

  1. Start with a data vector v0. Compute hidden probabilities, and sample hidden states h0.
  2. Reconstruct: compute visible probabilities from h0 and sample v1, then hidden probabilities h1 from v1.
  3. Update the weights by the difference of two correlations: dW = lr * (v0 h0^T - v1 h1^T), and similarly for the biases.

The first term strengthens connections that are active together on real data ("positive phase"). The second term weakens those that are active together in the model's own reconstruction ("negative phase"). When the model's dreams look like the data, the two terms cancel and learning stops.

The program trains an RBM with 8 hidden units on noisy copies of three 6 by 6 pictures (a "T", an "L" and a plus sign), then asks it to repair a damaged picture.

Python An RBM trained with contrastive divergence

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The reconstruction error falls from 0.25 (random weights) to about 0.05 within 200 epochs, close to the 5% of pixels that were flipped as noise, which is the best any model can do on noisy data. The damaged "T" had 6 wrong pixels, and one pass through the hidden layer gives back a clean T with none: the 8 hidden units have learned that the data consists of three clean shapes. (Your numbers will match these only with the same random seed.)

Once trained, the hidden units act as feature detectors, and one pass through them can denoise input (as above), generate new samples (alternate sampling h and v for many steps), or provide features for a classifier. A deep belief network stacks RBMs, training each on the hidden activations of the one below. Modern generative models replaced RBMs because they are easier to train and scale, but the energy view, in which "plausible" means "low energy", lives on in energy-based models.

Self-organizing maps

A self-organizing map (SOM), or Kohonen map, is a grid of units, each holding a weight vector with the same dimension as the data. It does competitive learning, in three steps repeated for every sample x:

  1. Find the best matching unit (BMU): the grid unit whose weight vector is closest to x.
  2. Compute a neighborhood around the BMU on the grid (not in data space): units near the BMU on the grid get a strong update, distant ones a weak one. A Gaussian is typical: h = exp(-d_grid^2 / (2 sigma^2)).
  3. Move every unit toward x in proportion to that neighborhood: w += lr * h * (x - w).

Both the learning rate and the neighborhood radius sigma shrink over time: first the map unfolds across the data, then each unit settles on a local patch. The result is a topology-preserving map. Data points that are close in the original space land on units that are close on the grid, so a 2D grid becomes a picture of a high-dimensional data set.

Python Training a self-organizing map

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Training lowers the average distance from a sample to its best matching unit (the quantization error) from 0.16 to 0.047. The printed map shows the three clusters as three solid regions of the grid: A in the upper left, C on the right and B along the bottom. Grid neighbors have weight vectors about 0.11 apart on average, while two random units are about 0.53 apart, so the grid really is organized by similarity.

The dots in the map are units that no sample picks; they sit in the gaps between clusters, where the grid stretches across empty space. In practice SOMs are used for visualization (colored maps and "U-matrices" of distances between neighbors), for clustering, and for compressing data to a small codebook of prototype vectors.

Hopfield networks

A Hopfield network is a memory. It has N binary neurons with values +1 or -1, all connected to all the others by symmetric weights W (with a zero diagonal). To store patterns x^1, ..., x^P, you set the weights in one step with the Hebbian rule, "neurons that fire together wire together":

TEXT
W = (1/N) * sum over patterns of  x x^T      (diagonal set to 0)

To recall, you set the neurons to a (possibly corrupted) cue and update them repeatedly: each neuron takes the sign of its weighted input, s_i = sign(sum_j W[i, j] * s_j). Updating one neuron at a time guarantees that the network's energy

TEXT
E(s) = -1/2 * s.W.s

never increases. So the state slides downhill on an energy landscape until it reaches a local minimum, and the stored patterns are (ideally) the minima. A noisy picture lies on the slope of the right valley and rolls to the bottom: this is associative memory, retrieval by content rather than by address.

Python A Hopfield memory recalls damaged patterns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The cue (an O with 5 of 25 pixels flipped) has a high energy of -4.16. After two sweeps, the network lands exactly on the stored O at energy -14.24, the lowest of the three stored patterns' energies (the X and the plus sign sit at -13.6). The capacity experiment shows the limit: 100 neurons recall 3 or 8 random patterns every time, 12 patterns about 85% of the time, 20 patterns only 30% and 30 patterns never.

Hopfield networks are limited. Storing too many patterns creates spurious minima (blends of stored patterns), and the capacity is only about 0.14 * N random patterns, shown in the last experiment. Similar patterns interfere: the Hebbian rule works best when memories are roughly orthogonal. Modern Hopfield networks (dense associative memories) raise the capacity enormously by sharpening the energy function, and their update rule turns out to be closely related to the attention mechanism in Transformers.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is the Boltzmann machine 'restricted' in an RBM?

    Connections exist only between the visible and hidden layers. That makes p(h | v) and p(v | h) simple products of sigmoids, which makes contrastive divergence practical.
  2. 2. In a self-organizing map, where is the neighborhood of the winning unit measured?

    Units close to the winner on the grid are pulled toward the sample. Because grid neighbors move together, nearby grid positions end up representing similar data.
  3. 3. What guarantees that a Hopfield network settles into a stable state?

    With symmetric weights and zero diagonal, flipping a neuron to match its input can only keep or lower the energy, so the state ends in a local minimum.
  4. 4. Roughly how many random patterns can a Hopfield network of N neurons store reliably?

    Beyond roughly 14 percent of N, crosstalk between memories creates spurious minima and recall fails. Modern variants raise this limit.

Summary

  • RBMs have a visible and a hidden layer with no connections inside a layer; contrastive divergence compares data correlations with reconstruction correlations to learn a probability model.
  • Hidden units of a trained RBM are feature detectors that can denoise, generate samples and feed classifiers; stacked RBMs formed the first deep belief networks.
  • Self-organizing maps pick a best matching unit, then pull it and its grid neighbors toward the sample, producing a topology-preserving map for visualization and clustering.
  • Hopfield networks store patterns in symmetric Hebbian weights and recall them by descending an energy landscape from a corrupted cue.
  • Their capacity is about 0.14 N random patterns; modern Hopfield networks lift this and relate to attention.

Next: RBF, Siamese, Capsule and Spiking Networks.