Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 111 ยท Getting Started

Density Estimation and Maximum Likelihood

Estimate a probability distribution from data: maximum likelihood for parametric models, histograms and kernel density estimation for non-parametric ones.

Many machine learning methods rest on one question: how likely is this value, given the data I have seen? Naive Bayes needs the density of each feature in each class, Gaussian mixtures model data as a sum of bell curves, anomaly detectors flag points with a very low density, and logistic regression is trained by maximizing a likelihood. All of them are doing probability density estimation: they take a sample and recover the distribution it came from.

There are two broad ways to do it. A parametric method assumes a family of distributions (such as the normal distribution) and estimates its few parameters, usually by maximum likelihood. A non-parametric method makes no such assumption and lets the data draw the shape. This lesson builds on probability and statistics for ML.

Likelihood: how well does a model explain the data?

Suppose a model with parameters ฮธ says how probable each observation is. The likelihood of the whole dataset is the product of those probabilities, assuming the observations are independent:

TEXT
L(ฮธ) = p(x1 | ฮธ) ร— p(x2 | ฮธ) ร— ... ร— p(xn | ฮธ)

Products of many small numbers underflow quickly, and sums are easier to differentiate, so we work with the log-likelihood, which turns the product into a sum:

TEXT
log L(ฮธ) = log p(x1 | ฮธ) + log p(x2 | ฮธ) + ... + log p(xn | ฮธ)

Maximum likelihood estimation (MLE) chooses the parameters ฮธ that make the observed data most probable, that is, that maximize the log-likelihood. Because the logarithm increases with its argument, the best ฮธ is the same for the likelihood and the log-likelihood.

For two common distributions the answer has a closed form:

  • Bernoulli (a coin with heads probability p): the MLE is the observed fraction of heads. Seven heads in ten flips gives p = 0.7.
  • Normal: the MLE for the mean is the sample mean, and for the variance it is the average squared deviation from the mean, dividing by n (not n - 1).
  • Exponential (waiting times with rate ฮป): the MLE is 1 / mean.
Python Maximum likelihood by hand, by search and by formula

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The log-likelihood is a hill: it is lower for every mean away from 169.71 and highest at the sample mean. When no closed form exists (logistic regression, neural networks), training simply climbs this hill with gradient descent. Minimizing the negative log-likelihood is exactly the "log loss" or cross-entropy you meet in classification.

The parametric limit: the family must fit

MLE finds the best parameters within the family you chose. If the family is wrong, the result is the best of a bad lot. Data with two peaks (two groups of customers, say) cannot be described by one normal curve, however carefully you fit it. You can add flexibility with a mixture of several distributions (Gaussian mixtures), or you can drop the assumption altogether.

Non-parametric estimation: histograms and kernels

The simplest non-parametric estimate is a histogram: cut the range into bins, count the points per bin and rescale so the total area is 1. Its shape depends on the bin width and on where the bins start, and it is jagged.

Kernel density estimation (KDE) smooths this idea. Put a small bump (a kernel, usually a Gaussian) on top of every data point and add all bumps. The density is high where many points are close together. The only real choice is the bandwidth, the width of each bump:

  • A small bandwidth gives a spiky curve that memorizes the sample (high variance, like overfitting).
  • A large bandwidth gives a blurry curve that hides real structure (high bias, like underfitting).

This is the same bias-variance trade-off you know from overfitting and underfitting. You can choose the bandwidth by cross-validation, using the held-out log-likelihood as the score; KernelDensity already has a score method that does exactly that.

Python One Gaussian versus kernel density estimates on two-peaked data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The single Gaussian is wrong in a systematic way: its error (RMSE 0.083) is much larger than that of a well-chosen KDE (0.013 at bandwidth 0.3). A bandwidth that is too small (0.05) or too large (1.5) is worse again. The cross-validated KDE finds both peaks (at about -2.0 and 2.5, the true group centers) and gives test points a clearly higher likelihood than the single bell curve (-1.91 versus -2.27 per point).

Choosing between the two

Parametric (MLE) Non-parametric (KDE)
Assumption A distribution family Only smoothness
Parameters A few One bandwidth, but the whole sample is stored
Data needed Little More, and much more in high dimensions
Risk Wrong family (bias) Noisy or over-smoothed estimates
Speed at prediction Very fast Slower (scales with sample size)

KDE suffers from the curse of dimensionality: beyond a handful of features there is rarely enough data to fill the space. In high dimensions use a parametric model, a mixture, or a model that scores density in some other way (such as the detectors in anomaly detection).

Where you will use this

  • Naive Bayes estimates one density per feature and class (Naive Bayes).
  • Anomaly detection flags points where the estimated density is very low.
  • Generative models sample new data from an estimated density.
  • Model training is usually "minimize the negative log-likelihood".
  • Visualization: a smooth density often tells the story of a column better than a histogram.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does maximum likelihood estimation do?

    MLE searches for the parameter values under which the observed sample has the highest probability (equivalently, the highest log-likelihood).
  2. 2. Why do we usually maximize the log-likelihood instead of the likelihood?

    The logarithm is increasing, so it keeps the same maximizer, and it turns the product into a sum.
  3. 3. What is the MLE of the heads probability when a coin shows 7 heads in 10 flips?

    For a Bernoulli model the MLE is the observed fraction of successes: 7/10 = 0.7.
  4. 4. A kernel density estimate looks like many tiny spikes, one per data point. What is wrong?

    A tiny bandwidth puts a narrow bump on every point. Increase it, ideally by choosing it with cross-validation.

Summary

  • Density estimation recovers a probability distribution from a sample.
  • Maximum likelihood picks the parameters that make the data most probable; in practice we maximize the log-likelihood.
  • Closed forms: sample fraction (Bernoulli), sample mean and variance with n in the denominator (normal), 1 / mean (exponential).
  • Parametric models are compact but biased if the family is wrong; kernel density estimation makes no family assumption but needs a bandwidth and more data.
  • Choose the bandwidth by cross-validated log-likelihood.

Next: the calculus that powers optimization, in Multivariable Calculus for ML.