Skip to content
Tutorials Center
My learning

Topics

Lesson 47 of 73 ยท Generative and Other Models

RBF, Siamese, Capsule and Spiking Networks

Meet four specialist network types in NumPy: RBF networks, Siamese networks with contrastive and triplet loss, capsule routing, and spiking neurons.

The networks in the main part of this course (dense, convolutional, recurrent, attention) cover most practical work. But the field has produced many other designs, each built around one clear idea. Four of them come up often enough that you should recognize them: radial basis function (RBF) networks, which classify by distance to prototypes; Siamese networks, which learn whether two things are alike; capsule networks, which pass vectors instead of single numbers; and spiking networks, which communicate with timed pulses like real neurons.

This lesson gives each one a section, a short explanation and a small program you can run. All four are real implementations in NumPy, scaled down so that the idea is visible. None of them is a drop-in replacement for the main architectures, and the last two are mainly active research areas, but each teaches something you can reuse.

Four networks, four ideas

RBF network Hidden units fire near a center.
Siamese network Twin encoders, shared weights, compare the outputs.
Capsule network Vector outputs and routing by agreement.
Spiking network Neurons send timed pulses.
1 / 2

RBF networks

A radial basis function network has one hidden layer with a special kind of unit. Instead of a weighted sum followed by an activation, hidden unit k measures the distance from the input x to its own center c_k and responds with a bump:

phi_k(x) = exp(-||x - c_k||^2 / (2 s^2))

The unit outputs close to 1 when x sits on its center and falls toward 0 as x moves away. The width s controls how far the influence reaches. The output layer is an ordinary linear layer on top of these bump values. So an RBF network answers a question by asking, "which of my prototypes is this input near?", and it can carve out curved regions with a handful of centers.

Training can be split into two easy stages, which is why RBF networks were popular before deep learning. First place the centers, for example with k-means on the training inputs. Then, with the centers fixed, the output weights are a linear problem that has a closed-form solution (here, ridge regression). No backpropagation is needed, although you can also train everything end to end.

Python An RBF network on two interleaving moons

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two moons cannot be split by a straight line, and the linear model manages 87.8% on the test data. With 10 centers of width 0.5, the RBF network reaches 99.0%. The other rows show the two knobs. Two or four centers are too few to bend the boundary (75.8% and 83.8%). A width of 0.15 makes every bump so narrow that the space between the centers is not covered (93.8%), and a width of 2.0 blurs everything into one broad hill (88.2%). Choosing centers and width is the real design work of an RBF network.

RBF units are local: each one responds only near its center, unlike a ReLU unit that responds across a whole half-space. That makes RBF networks good at smooth interpolation in low dimensions and poor in high dimensions, where "nearby" stops meaning much (the curse of dimensionality). The same bump appears in support vector machines with an RBF kernel, in Gaussian processes and in radial-basis interpolation.

Siamese networks

A Siamese network is not a different kind of layer but a different way of using one. Two inputs pass through the same encoder with the same weights, which gives two embedding vectors. The model's answer is a function of the distance between them. The twin copies share weights because "similar" must mean the same thing for both sides.

This design suits tasks where you have many classes, few examples per class, or classes that appear later: face verification, signature checking, product matching, duplicate-question detection and image retrieval. You are not teaching the network what a particular person looks like, but how to measure whether two photos show the same person. A new person can then be recognized from a single reference photo, which is called one-shot learning.

Two losses teach the encoder to build a useful distance:

  • Contrastive loss works on pairs. For a pair with label same = 1 it penalizes the squared distance d^2. For a pair with same = 0 it penalizes max(0, m - d)^2, which pushes different items at least a margin m apart and ignores them once they are far enough.
  • Triplet loss works on triples: an anchor a, a positive p of the same class and a negative n of a different class. The loss is max(0, d(a, p)^2 - d(a, n)^2 + margin): the positive must be closer to the anchor than the negative, by a margin. Choosing triplets that are hard (the negative is almost as close as the positive) matters a lot, and many systems mine them from each batch.

The example builds a synthetic version of face verification. Each "photo" has 20 measurements, but only 3 of them depend on who is pictured; the other 17 are nuisance (think lighting and background) with a large spread. The encoder is a single linear layer trained with each loss on 8 people. We then test it on 5 people it has never seen.

Python Learning a distance with triplet and contrastive loss

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Raw Euclidean distance is swamped by the nuisance measurements: 80.6% same-or-different accuracy is no better than always answering "different" (about 80%, because only one pair in five is a match), and one-shot recognition is 24.5%, hardly above guessing. After training with triplet loss, a 3-dimensional embedding reaches 88.8% and recognizes 74.5% of photos of the new people from a single example each; contrastive loss gets 86.0% and 59.5%. The last printout shows how: the learned matrix gives the three informative measurements weights of 0.9 to 1.6 and the nuisance ones about 0.05, so it has learned to ignore what should not matter, and that knowledge transfers to people it has never seen. The numbers depend on the seed and on the learning rate; the lesson is the pattern, not the exact percentages.

For images, text and audio you would replace the linear layer with a CNN or a transformer encoder. Modern embedding models for search and recommendation use the same idea at scale, often with a variant such as in-batch negatives (other items in the batch serve as negatives) that gets triplet-like signal very cheaply.

Capsule networks

Convolutional networks detect features, and max pooling throws away where exactly they were. A face detector can be fooled by a bag of parts in the wrong arrangement. Capsule networks (Hinton and colleagues, 2017) try to keep the structure. A capsule outputs a vector instead of a single number. The vector's length says how likely the entity is to be present, and its direction encodes its properties, such as pose, size or orientation. A squash function rescales any vector to a length between 0 and 1 while keeping its direction.

Higher capsules (a "face") are built from lower ones (an "eye", a "nose", a "mouth"). Each lower capsule predicts the output of every higher capsule by multiplying its vector by a learned matrix. Routing by agreement then decides who talks to whom: a lower capsule sends more of its output to those higher capsules whose result agrees with its own prediction, and less to those it disagrees with. This is repeated for a few iterations. If an eye, a nose and a mouth all predict the same pose for a face, the face capsule becomes confident.

Python Squash and routing by agreement, and a spiking neuron

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The squash numbers show the intended behavior: a tiny vector is squeezed almost to nothing (0.010), a long one saturates just below 1 (0.962). In the routing example, four lower capsules agree on the pose of the "face" and point in scattered directions for the "house". In the first iteration they split their votes evenly. By the third, 90% of the vote has moved to the face, whose output length rises from 0.86 to 0.95, and the house output stays at 0.00. Nothing was trained here; this is only the routing loop itself.

Capsule networks produced good results on small digit datasets and are interesting for understanding viewpoint changes, but the routing loop is slow and they did not scale to large images as well as CNNs and vision transformers. Think of them as an influential idea rather than a tool.

Spiking neural networks

The neurons in every other model in this course output a real number at every step. A biological neuron does not: it accumulates input, and when its membrane voltage crosses a threshold it sends a brief pulse, a spike, and resets. Information is carried by when spikes occur and how many there are. A spiking neural network (SNN) copies that behavior.

The standard model is the leaky integrate-and-fire (LIF) neuron. Its voltage v leaks back toward zero and is pushed up by its input current:

TEXT
v = v + (-v + input) / tau
if v >= threshold: emit a spike, then v = 0

The second half of the example above runs it. An input of 0.9 stays below the threshold of 1.0 and never fires. Inputs of 1.2, 1.6, 2.5 and 5.0 fire 5, 10, 20 and 40 times in 200 steps, so the rate tells you the strength, and the first spike arrives at step 34, 19, 9 and 4, so the latency carries the same information faster.

Because spikes are sparse and events happen only when something changes, an SNN can run on neuromorphic hardware (chips such as Intel's Loihi or IBM's TrueNorth) that uses very little power. The difficulty is training. The spike is a hard threshold, so its derivative is zero almost everywhere and ordinary backpropagation fails. Researchers use surrogate gradients (pretend the step function was a smooth sigmoid when going backward), spike-timing-dependent plasticity (a local learning rule inspired by biology), or train a normal network and convert it. SNNs are most useful today for low-power sensing, such as event cameras, and in neuroscience models.

Choosing among them

Network Core idea Typical strength Typical limitation
RBF hidden units respond near learned centers simple, fast to train, smooth interpolation weak in high dimensions, needs good centers
Siamese shared encoder, compare embeddings by distance few or new classes, verification, retrieval needs good pair or triplet selection
Capsule vector outputs, routing by agreement keeps part-whole and pose information slow routing, did not scale
Spiking timed pulses, leaky integrate-and-fire very low power on special hardware hard to train, little tooling

If you meet a problem that looks like "same or different?", start with a Siamese setup or an embedding model. If you meet "tiny data, smooth function, few inputs", an RBF network or a kernel method is a reasonable baseline. For the other two, read the papers and watch where the research goes.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does a hidden unit in an RBF network compute?

    An RBF unit measures the distance from the input to its center and responds with a Gaussian bump, so it is a local detector.
  2. 2. Why do the two branches of a Siamese network share their weights?

    Similarity only makes sense if both inputs are mapped into the same space by the same encoder.
  3. 3. What does triplet loss require of an anchor, a positive and a negative?

    The loss max(0, d(a,p)^2 - d(a,n)^2 + margin) is zero only when the positive is closer to the anchor than the negative is, by the margin.
  4. 4. In a capsule network, what does the length of a capsule's output vector represent?

    The direction encodes properties such as pose, while the length (squashed into 0 to 1) says how likely the entity is to be there.
  5. 5. Why can't ordinary backpropagation train a spiking neuron directly?

    A step function gives no useful gradient. Surrogate gradients replace its derivative with a smooth stand-in during the backward pass.

Summary

  • An RBF network uses Gaussian bump units around centers (often found with k-means) and a linear output layer; width and number of centers are its key settings.
  • A Siamese network passes two inputs through one shared encoder and compares the embeddings; contrastive loss uses pairs and triplet loss uses anchor, positive and negative with a margin.
  • Siamese models learn a distance that transfers to classes never seen in training, which enables one-shot recognition and similarity search.
  • A capsule outputs a vector (length is presence, direction is pose); routing by agreement sends votes to the higher capsules that agree with the lower capsules' predictions.
  • A spiking neuron integrates input, leaks, fires a spike at a threshold and resets, and information is carried by spike rate and timing; training needs surrogate gradients because the spike is not differentiable.

Next: Graph Neural Networks.