Skip to content
Tutorials Center
My learning

Topics

Lesson 76 of 111 ยท Beyond the Basics

Semi-Supervised Learning and Active Learning

Learn when labels are scarce: self-training, label propagation, and an uncertainty-sampling active learning loop that spends a labeling budget wisely.

Labels are usually the most expensive part of a machine learning project. Collecting thousands of photos, log lines or customer records is easy; getting a doctor, a lawyer or a careful annotator to label each one is slow and costs real money. Two families of techniques attack this problem from different sides.

Semi-supervised learning uses a small labeled set together with a large pile of unlabeled data. Active learning goes one step further: the model chooses which unlabeled examples a human should label next, so every label bought teaches it as much as possible. In this lesson you build both with scikit-learn and NumPy, and you will see when they help and when they quietly do harm.

Three data situations

Fully labeled Ordinary supervised learning.
Few labels, many rows Semi-supervised learning.
Labels can be bought Active learning.

The cheaper the unlabeled data, the more these ideas pay off.

1 / 2

Marking unlabeled data

scikit-learn's semi-supervised tools use one convention: an unlabeled sample has the label -1. You pass the full y array, with -1 in the positions whose label is unknown. The features of those rows are still used.

To practice, we hide most labels of a dataset whose true labels we know. That lets us measure honestly how much the hidden rows helped. In real life you could not do this check, which is why a labeled test set is still essential (see Train, Validation and Test Sets).

Python A tiny labeled set versus a bigger pile of unlabeled rows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The gap between those two numbers is the room semi-supervised learning is trying to close. It cannot beat having every label, but it can recover part of the difference using only the features.

Self-training

Self-training is the simplest idea: train on the labeled rows, predict the unlabeled ones, keep the predictions the model is confident about, treat them as if they were real labels, and train again. Repeat until nothing new passes the confidence bar.

flowchart LR
    A["Train on<br/>labeled rows"] --> B["Predict<br/>unlabeled rows"]
    B --> C{"Confidence<br/>above threshold?"}
    C -->|"yes"| D["Add as<br/>pseudo-labels"]
    C -->|"no"| E["Leave<br/>unlabeled"]
    D --> A

Each round, the most confident guesses join the training set.

SelfTrainingClassifier wraps any classifier that can output probabilities (predict_proba). Its threshold is the confidence needed to accept a pseudo-label. After fitting, labeled_iter_ records in which round each sample received its label (0 means it was labeled from the start, -1 means it never got one).

Python Self-training with a confidence threshold

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the output as a trade-off. A low threshold adds many pseudo-labels quickly, but some of them are wrong, and the model then learns from its own mistakes. A high threshold adds few labels, which are almost all correct, but may stop early. Counting wrong pseudo-labels is only possible here because we secretly know the truth. Notice also that the gain over the supervised baseline is small, and at some thresholds absent: self-training is not guaranteed to help, so always compare against the plain model.

Self-training can confirm its own bias. If the first model is confidently wrong about a region of the data, every pseudo-label there reinforces the error. Watch the test score, not just the number of accepted labels.

Label propagation

Label propagation takes a different view. It builds a graph in which every row (labeled or not) is a node connected to similar rows, then lets the known labels "flow" along the edges to their neighbors. Rows near a labeled point of class 1 end up as class 1; the labels follow the shape of the data.

scikit-learn offers LabelPropagation and the more forgiving LabelSpreading. Both accept kernel="knn" (connect each row to its n_neighbors nearest) or kernel="rbf" (weight by a Gaussian of distance, controlled by gamma). LabelSpreading also has alpha between 0 and 1: the share of its neighbors' influence a row accepts, so a small alpha clings to the original labels and spreads little. The settings matter a great deal: with only 7 neighbors and a small alpha the same two labels reach barely more than half of these rows correctly, so tune them on a labeled validation set.

Python Labels flowing along two moons

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With one label per moon, a plain classifier has nothing to go on, yet the graph carries each label across its own curved shape. This works because the data really does form two clusters that match the classes. This is the cluster assumption: points that sit close together tend to share a label. When the assumption fails, for example when classes overlap heavily, propagation spreads wrong labels just as efficiently.

Two practical notes:

  • Both methods are transductive: transduction_ holds the labels for the rows you fit on. They also provide predict for new rows, but the quality is best on data similar to the graph.
  • The graph is n x n, so memory grows quadratically. Use the knn kernel for anything larger than a few thousand rows.

Co-training and graph-based learning

Self-training and label propagation are two of several semi-supervised ideas. Two more complete the picture.

Co-training needs data that has two different views: two sets of features, each sufficient on its own to predict the label, and not strongly dependent given the label. A web page can be described by its own text (view 1) and by the text of the links pointing to it (view 2); a product by its photo and by its description. Two classifiers are trained, one per view, on the small labeled set. Then each classifier labels the unlabeled examples it is most confident about, and those pseudo-labels are added to the other classifier's training set. Because the views differ, one classifier can teach the other things it would not find on its own, which makes co-training less prone than plain self-training to reinforcing its own mistakes. If the views are really the same information, the benefit vanishes.

Graph-based learning treats every example (labeled or not) as a node and connects similar examples with weighted edges. Labels flow from labeled nodes to their neighbors along the edges, on the assumption that connected points share labels. Label propagation and label spreading (described above) are the standard algorithms. Graph methods suit data with a natural network (social networks, citation graphs, molecules) as well as data where a similarity graph can be built from the features.

Scikit-learn has self-training and label propagation but no co-training class, so the example below writes the loop by hand: each round, each view's classifier gives its 10 most confident rows per class to the other view's training pool. The data has two views of 6 features each, both informative about the label and independent of each other given the label (the ideal case); only 10 of 1,800 training rows are labeled.

Python Co-training with two feature views

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With only 10 labels the baseline reaches 0.81. After co-training the model uses about 400 rows, 98% of whose pseudo-labels are correct, and reaches 0.865, close to the 0.872 of a model trained on all 1,800 labels. The share of correct pseudo-labels is the number to watch: pseudo-labels are only useful if almost all of them are right. (When we let the classifiers pick their most confident rows without balancing the classes, one run collapsed to 0.69 accuracy, because the confident rows all came from one class. That is why the loop takes the same number of rows per class.) The result also depends on the views really being informative and different. In practice keep a small labeled validation set to confirm the gain, and stop when it stops improving.

Active learning: choosing what to label

Now flip the question. Instead of making do with whatever labels exist, suppose an annotator can label, say, 40 more rows. Which 40? Random rows are a fine default, but many of them will be easy cases the model already gets right. Uncertainty sampling asks for the rows the current model is least sure about, which are usually near the decision boundary and carry the most information.

The loop looks like this:

flowchart LR
    A["Train on<br/>current labels"] --> B["Score the<br/>unlabeled pool"]
    B --> C["Pick the most<br/>uncertain rows"]
    C --> D["Human labels<br/>them"]
    D --> A

The model and the annotator take turns until the budget runs out.

For a binary classifier, uncertainty is highest when predict_proba is close to 0.5. A simple score for any number of classes is 1 - max probability: the larger it is, the less sure the model is. We simulate the human by looking up the true label.

Python Uncertainty sampling versus random sampling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Averaging over ten starting sets matters: a single run of either strategy is noisy, and a lucky random draw can look better than a smart one. Compare the two columns at equal numbers of labels. If uncertainty sampling reaches a given accuracy with noticeably fewer labels, it saved annotation cost. The curves can also cross or stay close, because active learning is no magic: it helps most when labels are costly and the model is not yet near its ceiling.

Labeling budgets in practice

Real projects add a few more considerations:

  • Batch size. Humans label in batches, not one by one. Choosing the top 10 most uncertain rows can pick 10 near-duplicates; production systems add a diversity rule, such as clustering the uncertain rows and taking one per cluster.
  • Sampling bias. Rows picked by uncertainty are not a random sample of the world. Never evaluate on them; keep a separate, randomly drawn, labeled test set.
  • Stopping rule. Stop when the validation score stops improving by more than the cost of another batch is worth. Plotting the curve, as above, is the easiest way to see this.
  • Annotator noise. A model that asks about confusing rows gets confusing rows labeled, and humans disagree most on exactly those. Budget for two annotators on a sample.
  • Cold start. The first model needs some labels from every class. Start with a small random batch, then switch strategies.

Combine the two ideas. Use label propagation or self-training to make use of unlabeled rows, and active learning to decide where to spend human attention. Always keep a clean, randomly sampled labeled test set aside.

When these methods help

Semi-supervised learning helps when (1) unlabeled data is plentiful, (2) labeled data is truly scarce, and (3) the cluster assumption roughly holds. It can hurt when the labeled set is unrepresentative or the model is overconfident. Active learning helps when labels have a price and a reasonable starting model exists. Neither replaces the habits from The Machine Learning Workflow: always compare against a plain supervised model trained on the same labeled rows, as the examples above do.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. How does scikit-learn's semi-supervised API mark a sample whose label is unknown?

    You pass the full X and a y array in which unlabeled rows hold -1. Their features are still used during fitting.
  2. 2. What is the main risk of self-training with a low confidence threshold?

    A low threshold accepts many uncertain guesses. Wrong ones become training data and can reinforce the original error.
  3. 3. Which assumption makes label propagation work well?

    This is the cluster assumption. Labels spread along neighbors, so they are only correct if neighbors usually belong to the same class.
  4. 4. In uncertainty sampling for a classifier with predict_proba, which unlabeled row do you send to the annotator first?

    The lowest maximum probability means the model is least sure, so a label there is expected to teach it the most.
  5. 5. Why should you not measure accuracy on the rows an active learner asked to be labeled?

    Uncertainty sampling skews the selection toward confusing rows. Keep a separate randomly sampled, labeled test set for honest evaluation.

Summary

  • Semi-supervised learning uses a few labels plus many unlabeled rows; scikit-learn marks unlabeled targets with -1.
  • SelfTrainingClassifier adds confident pseudo-labels round by round; the threshold trades quantity against correctness.
  • LabelPropagation and LabelSpreading spread labels over a similarity graph and rely on the cluster assumption.
  • Active learning picks the rows worth labeling; uncertainty sampling chooses those with the lowest maximum predicted probability.
  • Compare against a supervised model on the same labels, average over several runs, and keep a random labeled test set.
  • Budget for batch diversity, annotator disagreement and a stopping rule.

Next up: put a trained model to work in Saving Models and Deploying.