Skip to content
Tutorials Center
My learning

Topics

Lesson 86 of 111 ยท Beyond the Basics

Self-Supervised Learning

Learn from unlabeled data: pretext tasks, masked prediction and contrastive learning, with a runnable scikit-learn demo and an InfoNCE loss in NumPy.

Labels are expensive. Someone has to mark every photo, read every document and diagnose every scan. Unlabeled data, on the other hand, is nearly free: millions of images, pages of text, hours of audio. Self-supervised learning (SSL) turns unlabeled data into a training signal by inventing a task whose answer is already hidden in the data. Hide part of the input and predict it; take two views of one item and learn that they belong together. The model is never told what a "3" or a "dog" is, yet it has to learn useful structure to solve the invented task.

Afterwards you reuse what the model learned, its internal representation, as the starting point for a real task that has only a handful of labels. This is how modern language models (predict the next word) and many vision models are built. This lesson explains the idea, builds a masked-prediction demo with scikit-learn, and writes the contrastive InfoNCE loss from scratch in NumPy. It sits next to semi-supervised and active learning, which also stretches a small label budget.

Self-supervised in three steps

Invent a task Hide part of the data, or make two views.
Pretrain Solve it on lots of unlabeled data.
Reuse Train a small model on few labels.
1 / 2

Pretext tasks

The invented task is called a pretext task. Its answers come for free from the data:

Data Pretext task What the model must learn
Text Predict the next word, or a masked word Grammar, facts, meaning
Images Fill in a masked patch, or restore a corrupted image Shapes, textures, objects
Images Predict how much an image was rotated Which way is up, what objects look like
Audio Predict a hidden slice of the signal Speech sounds, rhythm
Tables Predict a masked column from the others Relationships between features

Nobody cares about the pretext answers. The value is the representation: the numbers produced by the model's hidden layers. If solving the pretext task requires understanding the data, those numbers describe it well, and a simple classifier on top of them can reach good accuracy from few labels.

A masked-prediction demo

We use scikit-learn's 8x8 handwritten digits (64 pixels per image). The experiment:

  1. Pretrain a small neural network on all 1,797 images without labels. We randomly erase 40% of the pixels of each image and ask the network to reconstruct the full image. This is a denoising / masked autoencoder, the simplest masked-prediction model.
  2. Take the network's 64 hidden-layer activations as a learned representation of each image.
  3. Train a logistic regression on a small labeled set (5 or 20 labeled examples per digit) using three inputs: raw pixels, the learned representation, and a representation from the same network before pretraining (random weights) as a control.

The test set is a separate 600 images. Because everything uses one fixed random seed, the numbers below are reproducible, but they will move by a few points with other seeds, so look at the pattern rather than the last digit.

Python Masked pretraining, then few labels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The pretext task works: from a copy with 40% of the pixels erased, the network restores the image with an Rยฒ of about 0.48 on images it has never seen. The downstream results tell an honest story. With 5 labels per digit, logistic regression gets 0.859 on raw pixels, 0.774 on features from the untrained network and 0.850 on features from the pretrained one; with 20 labels per digit the scores are 0.926, 0.892 and 0.919. So pretraining clearly beats random features (about +8 points at 5 labels), but on this easy data set it only matches raw pixels, it does not beat them.

On a data set this clean, raw pixels are already a strong representation, so do not expect a dramatic win; self-supervision pays off most where raw inputs are poor features (pixels of large natural photos, raw audio, word IDs) and unlabeled data is plentiful. The control row matters: it shows that the gain, if any, comes from training on the pretext task and not just from having a wider hidden layer.

Contrastive learning

The second family does not predict anything. It learns by comparison. Take one item (a photo) and create two augmented views of it (different crops, color changes, noise). Views of the same item form a positive pair; views of different items in the batch are negatives. Train an encoder so that positives are close together in embedding space and negatives are far apart. Methods such as SimCLR, MoCo and CLIP (which pairs an image with its caption) follow this recipe.

The standard loss is InfoNCE (also called NT-Xent). For a batch of N items with embeddings z1[i] and z2[i] for the two views, normalize each embedding to unit length and compute the cosine similarities between every z1 and every z2. For item i, the correct match is z2[i], so apply a softmax over row i and take the negative log of the probability given to the correct match:

loss_i = โˆ’log( exp(sim(i, i) / ฯ„) / ฮฃโฑผ exp(sim(i, j) / ฯ„) )

ฯ„ (tau) is the temperature: a small value sharpens the softmax so the model is punished harder for confusing similar items. This is just cross-entropy where the "classes" are the other items in the batch and the right class is the matching view. If the embeddings carry no information, every item looks equally likely and the loss equals ln N.

In the next example we have no network to train, so we evaluate the loss for different encoders, which teaches what it measures. Two views of each digit are made by erasing random pixels and adding noise. We compare random projections of the pixels with a PCA projection fitted on unlabeled images, and also report how often the nearest z2 really is the other view of the same image.

Python InfoNCE loss from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table from the top. An encoder that outputs the same vector for everything scores exactly ln 128 = 4.852: it cannot tell items apart. Random vectors do even worse (7.15), because at a low temperature random similarities create confident wrong answers. The random projection of the pixels (4.35) is slightly better than nothing, raw pixels do better (3.55) and the PCA projection is best (2.80), picking the right partner for 31% of items out of 128 candidates (chance would be under 1%). All of these encoders were fixed, so they only differ in how well their geometry survives noise and erasing; training an encoder to minimize this loss is what contrastive pretraining does. The last block shows the temperature: it changes the loss a lot and has a sweet spot (here near 0.1), which is why it is a hyperparameter you tune.

A real contrastive method wraps this loss around a trainable encoder (a neural network) and updates the weights to lower it, using an autograd library; the next track, deep learning, shows how networks are trained. What you have built here is the loss itself, which is the whole idea.

Using what you learned

  • Pretrain, then fine-tune. Train on unlabeled data once, then either fit a small classifier on frozen features (a linear probe, like our logistic regression) or continue training the whole network on labeled data (fine-tuning).
  • Pick a pretext task that needs what you care about. If rotation is irrelevant for your images, a rotation-prediction task may teach little; masking and contrastive tasks are safer defaults.
  • Augmentations are the real design choice in contrastive learning: they define what the model should ignore. If your augmentation changes the label (a heavy crop that removes the object), the model learns the wrong thing.
  • Beware of shortcuts. A model can solve a pretext task with a trick that is useless later, such as using image borders instead of content.
  • Evaluate on labels. Judge a representation by how it performs on held-out labeled data, exactly as you would any other model.

Large pretrained models you download (BERT-style text encoders, vision transformers, CLIP) are self-supervised models trained at huge scale. Using one as a feature extractor is the industrial version of this lesson. See Pretrained Models for text.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is a pretext task?

    Pretext tasks such as predicting masked pixels or the next word need no human labels. The goal is the representation the model learns while solving them.
  2. 2. In contrastive learning, which pairs are treated as positives?

    Positives are different views of one item. Views of other items in the batch serve as negatives.
  3. 3. An InfoNCE loss on a batch of 128 items is about 4.85 (that is ln 128). What does that suggest about the encoder?

    With uninformative embeddings every candidate looks equally likely, so the cross-entropy over 128 candidates is ln 128. A useful encoder gets a lower loss.
  4. 4. Why include a control with random, untrained hidden weights in the masked-prediction experiment?

    A random network also produces 64 hidden features. Comparing against it isolates the benefit of learning them from the pretext task.

Summary

  • Self-supervised learning creates labels from the data itself, so it can use huge unlabeled data sets.
  • A pretext task (predict masked parts, restore corrupted inputs, match two views) forces the model to learn a representation.
  • The representation is reused: a linear probe on frozen features, or fine-tuning, needs far fewer labels.
  • Contrastive learning pulls positive pairs (two views of one item) together and pushes other items apart; InfoNCE is a softmax cross-entropy over the batch with a temperature.
  • Augmentations decide what the model learns to ignore; evaluate representations on held-out labeled data.
  • Gains are largest when raw inputs are poor features and unlabeled data is plentiful, not on tiny clean tables.

Next, look at what happens after a model is trained and shipped in MLOps and Model Monitoring.