Skip to content
Tutorials Center
My learning

Topics

Lesson 66 of 111 ยท Unsupervised Learning

Gaussian Mixture Models

Learn Gaussian mixture models: soft cluster membership, the EM algorithm in plain terms, covariance types, choosing components with BIC and AIC, and density estimation.

k-means gives every point a single, definite cluster. Real groups often overlap: a customer can behave partly like a bargain hunter and partly like a loyal regular. A Gaussian mixture model (GMM) captures this. It assumes the data was produced by several bell-shaped (Gaussian) distributions mixed together, and for each point it reports the probability of belonging to each one.

That gives you three upgrades over k-means. Clusters can be stretched ellipses instead of balls. Assignments are soft: probabilities instead of hard labels. And because a GMM is a full probability model, you can score how likely a point is, compare models with BIC, and even generate new data. This lesson covers all of that, plus the intuition behind the algorithm that fits it.

Soft clustering

A mixture with K components has, for each component k: a weight (how much of the data comes from it), a mean (its center) and a covariance (its shape and spread). The density of the data is the weighted sum of the component densities. Given a point, Bayes' rule turns this into the probability that the point came from each component. Those are called responsibilities, and in scikit-learn they are returned by predict_proba.

predict gives the component with the highest probability, which is the hard label you would compare with k-means.

A mixture of bell curves

Components Each cluster is one Gaussian.
Weights How much data each one explains.
Means Where each one is centered.
Covariances Each one's shape and spread.
1 / 2
Python Soft assignments with predict_proba

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In our run 103 of the 400 points have a top probability below 0.9, and the least certain ones are split almost exactly 50/50. In the right panel points in the overlap are colored in between, where a hard label would hide how uncertain they are. Use predict_proba when the decision is costly and you want to treat ambiguous rows differently, for example by sending them to a human.

How EM fits it: the intuition

A GMM has a chicken-and-egg problem. If you knew which component each point came from, you could estimate each component's mean and covariance by simple averaging. If you knew the components, you could compute each point's responsibilities. You know neither, so the expectation-maximization (EM) algorithm alternates:

  1. E-step (expectation): using the current component parameters, compute every point's responsibilities.
  2. M-step (maximization): update each component's weight, mean and covariance using the responsibility-weighted points.
  3. Repeat until the log-likelihood barely changes.

You can recognize k-means in this: k-means is the "hard" special case. Each EM round never lowers the likelihood, but, like k-means, it finds a local optimum, so n_init restarts and the initialization matter.

Python EM in one dimension, from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The estimates move quickly from the poor start to the truth (weights near 0.6 and 0.4, means near -2 and 3), and the log-likelihood rises at every step. scikit-learn lands on essentially the same answer.

Covariance types

The covariance controls each component's shape. covariance_type offers four options, from flexible to constrained:

Type Each component is... Parameters
"full" (default) any ellipse, any orientation most
"tied" an ellipse, with the same shape for all components fewer
"diag" an ellipse aligned with the axes fewer
"spherical" a ball, like k-means, but with its own radius fewest

More flexibility fits tilted ellipses but needs more data and can overfit with few points. Using "spherical" with equal sizes and spreads behaves much like k-means.

Python Covariance types on tilted ellipses

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two groups are long, tilted and stacked close together. k-means (ARI about 0.46) and the axis-aligned or spherical models (0.23 to 0.43) slice them wrongly; "full" and "tied" (which fits here because both clusters share one shape) recover them perfectly (ARI 1.0), and they also get the lowest (best) BIC, with "tied" slightly ahead because it needs fewer parameters. Next you will see why BIC is the right yardstick.

Choosing the number of components

Like inertia in k-means, the log-likelihood always improves with more components, so you cannot use it alone. GMMs offer principled penalties:

  • BIC (Bayesian information criterion) = -2 * log-likelihood + p * log(n), where p is the number of parameters.
  • AIC (Akaike information criterion) = -2 * log-likelihood + 2 * p.

Both reward fit and punish complexity; lower is better. BIC penalizes complexity harder, so it picks smaller models; AIC may pick more components. In GaussianMixture they are the methods bic(X) and aic(X).

Python BIC and AIC to choose the number of components

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both criteria are lowest at 4 components, the true number of groups (BIC about 4,428), and BIC then rises while AIC stays nearly flat. If AIC picks a larger K than BIC on your data, the extra components usually split a real cluster, so prefer BIC when you want a simpler explanation.

Density estimation and sampling

Because a GMM is a probability model, it is also a density estimator:

  • score_samples(X) returns the log-density at each point; low values mean the point is unusual.
  • sample(n) generates new points from the fitted mixture.

Low-density points are natural anomaly candidates, which Anomaly and Outlier Detection builds on.

Python Density scores and sampling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The probe at (20, 20) sits far from every component and gets an extremely low score, while a component's own center scores high. sample returns both the new points and the component each came from. Remember that a GMM is only as good as its Gaussian assumption: for data with strongly non-Gaussian clusters (crescents, rings), it still works as a density estimator only if you give it many components.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does GaussianMixture.predict_proba return?

    Each row has one probability per component, summing to 1 (the responsibilities). score_samples returns the log-density instead.
  2. 2. In the EM algorithm, what happens in the E-step?

    The E-step computes responsibilities from the current parameters. The M-step then re-estimates weights, means and covariances using them.
  3. 3. You compare GMMs with 1 to 8 components. Which statement is right about BIC?

    BIC adds a penalty of p * log(n), which is larger than AIC's 2p once n exceeds about 8. Lower values are better for both.
  4. 4. Which covariance_type lets each component be a tilted ellipse with its own shape?

    full gives every component its own unconstrained covariance. tied shares one covariance, diag allows axis-aligned ellipses and spherical gives round components.

Summary

  • A GMM models data as a weighted mix of Gaussians and gives soft cluster membership with predict_proba.
  • EM alternates computing responsibilities (E-step) and re-estimating parameters (M-step); k-means is its hard-assignment cousin.
  • covariance_type trades flexibility (full) against stability (diag, spherical); full fits tilted ellipses.
  • Choose the number of components by the lowest BIC (simpler) or AIC; the log-likelihood alone always improves.
  • score_samples and sample make a GMM a density estimator and generator; low density marks unusual points.

Next: how to judge any of these clusterings, with and without labels, in Evaluating Clusters.