Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 36 · Bayes Theorem and Priors

The Likelihood Function

Learn what the likelihood function is, how it differs from probability, how to build one for binomial, Poisson and normal data, why we use logs, and how MLE relates to Bayes.

The likelihood is the part of a Bayesian model that talks to the data. It answers: "for each possible value of the parameter, how probable was the data I actually saw?" Get the likelihood right and half the modeling is done.

Probability vs likelihood

Take the Binomial model for k heads in n flips with head probability θ:

P(k | n, θ) = C(n, k) θ^k (1 - θ)^(n-k)

  • As a function of the data k (with θ fixed), this is a probability distribution. It sums to 1 over k = 0..n.
  • As a function of the parameter θ (with the data fixed), it is the likelihood L(θ). It does not integrate to 1. It simply ranks parameter values by how well they explain the data.
Python Same formula, two views

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The integral is about 1/11, not 1, confirming that a likelihood is not a probability distribution over θ.

Building a likelihood for independent data

When observations are independent given the parameter, the likelihood of the whole data set is the product of the individual probabilities:

L(θ) = Π p(yᵢ | θ)

For n flips with outcomes yᵢ ∈ {0, 1} this gives θ^k (1-θ)^(n-k). The binomial coefficient C(n, k) does not depend on θ, so it can be dropped without changing the shape.

Likelihood for other data types

Data Model Likelihood in θ
Successes out of trials Binomial(n, θ) θ^k (1-θ)^(n-k)
Event counts Poisson(λ) Π λ^yᵢ e^(-λ) / yᵢ!
Measurements Normal(μ, σ) Π (1/σ√2π) exp(-(yᵢ-μ)² / 2σ²)
Waiting times Exponential(rate) Π rate · e^(-rate yᵢ)

Choosing the likelihood is a modeling decision: ask what process generated the data. Counts of rare events: Poisson. Binary outcomes: Bernoulli or Binomial. Continuous with symmetric noise: Normal. Heavy tails: Student-t.

Work with logs

Multiplying hundreds of small probabilities underflows to 0 on a computer. Take logs: products become sums.

Python Why we use log-likelihoods

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The raw likelihood underflows to 0.0, while the log-likelihood works fine and clearly prefers μ = 3 over μ = 4. In practice you always compute log p(data | θ) + log p(θ), the log of the unnormalized posterior.

Maximum likelihood vs Bayes

The maximum likelihood estimate (MLE) is the parameter value where L(θ) peaks. For the Binomial it is k/n. A Bayesian does not stop at the peak: the posterior multiplies the likelihood by a prior and keeps the whole curve. With a flat prior the posterior has the same shape as the likelihood, and its mode equals the MLE.

Python Likelihood curve, MLE and posterior with a prior

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The Beta(5, 5) prior pulls the posterior mean toward 0.5, the pull being strongest when data is scarce.

Relative likelihood

Only ratios of likelihoods carry information. L(0.25)/L(0.5) tells you how many times better 0.25 explains the data than 0.5, and any constant factor cancels. This is why you may drop constants such as C(n, k): they change nothing about the posterior shape.

Common mistakes

  • Forgetting that the likelihood is not a distribution over the parameter, so you cannot say "the likelihood that θ is in this range".
  • Using a likelihood that ignores how the data was collected. For example, "flip until 3 heads" has a different likelihood from "flip 12 times", even though both could give 3 heads in 12 (the constants differ, but the shape in θ is the same here, which is the likelihood principle).
  • Assuming independence when observations are clustered or repeated measures. That needs hierarchical models later in the course.

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. What is the likelihood function?

    The likelihood fixes the data and varies the parameter; it ranks parameter values by how well they explain the data.
  2. 2. Why do we work with log-likelihoods?

    Independent observations multiply, which underflows; taking logs turns the product into a sum.
  3. 3. With a flat prior, where does the posterior peak?

    A flat prior is constant, so the posterior is proportional to the likelihood and has the same mode.
  4. 4. Can you drop the binomial coefficient C(n, k) when computing the posterior over theta?

    Constants that do not involve the parameter cancel when you normalize the posterior.

Summary

  • The likelihood is the data's probability as a function of the parameter; it is not a distribution over the parameter.
  • Independent data give a product of terms; use sums of logs in practice.
  • Pick the likelihood to match the data-generating process.
  • MLE is the likelihood's peak; the posterior combines the whole likelihood with a prior.
  • Constants not involving the parameter can be dropped.

Next: Grid Approximation