Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 36 ยท Bayes Theorem and Priors

Grid Approximation

Compute posteriors with grid approximation: one-parameter and two-parameter examples for coins and normal data, sampling from the grid, and why it fails in high dimensions.

Grid approximation is the most honest way to compute a posterior: try every plausible parameter value on a fine grid, multiply prior by likelihood at each point, and normalize. There is no theory to remember and no sampler to tune, so it is the perfect way to see Bayes in action. It breaks down when you have many parameters, which is the reason chapter 4 exists.

The recipe

  1. Choose a grid of parameter values covering the plausible range.
  2. Compute the prior at each grid point.
  3. Compute the likelihood of the data at each grid point.
  4. Multiply and divide by the sum.
flowchart LR
    G["Grid of ฮธ values"] --> P["Prior p(ฮธ)"]
    G --> L["Likelihood p(y | ฮธ)"]
    P --> M["Multiply"]
    L --> M
    M --> N["Divide by the sum"]
    N --> Z["Posterior on the grid"]

Four steps; the sum in the last one replaces the hard integral.

One parameter: a click rate

A new ad was shown 25 times and clicked 4 times. Our prior says typical click rates are low, so we use Beta(2, 10) (mean 1/6).

Python One-parameter grid posterior

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The grid mean matches the exact conjugate result Beta(6, 31), mean 6/37 = 0.1622. The finer the grid, the better the match.

Sampling from the grid

Once you have grid probabilities, you can draw samples with rng.choice(theta, p=post) and then compute any summary by counting, exactly as you would with MCMC draws.

Python Draw samples from the grid posterior

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Working with draws instead of grid weights makes later steps (predictions, differences between groups) easy.

Two parameters: mean and standard deviation

Now estimate both ฮผ and ฯƒ of normally distributed measurements. The grid becomes a 2-D table: one axis for ฮผ, one for ฯƒ. We use weak priors: ฮผ ~ Normal(0, 10) and ฯƒ ~ HalfNormal(5) (a normal restricted to positive values).

Python Two-parameter grid

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notes on the code: we work with log posteriors and subtract the maximum before exponentiating to avoid underflow. The marginal posterior of one parameter is obtained by summing the table over the other axis. That is the Bayesian way to "ignore" a nuisance parameter: integrate it out, rather than plugging in an estimate.

Why a grid does not scale

A grid with 100 points per parameter has 100^d cells for d parameters.

Python The curse of dimensionality

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

By 5 parameters you need ten billion evaluations, and models with dozens of parameters are common. Most of that space also has essentially zero posterior probability, so the grid wastes nearly all its work. MCMC samplers spend their effort where the probability is, which is why they take over.

When a grid is still the right tool

  • Teaching and checking: it exposes every step.
  • One or two parameters: fast and exact enough.
  • Verifying a fancy sampler against a trusted answer.
  • Models with no closed form, such as unusual priors.

Choosing the grid: cover the range where the posterior has mass (check that the posterior goes to ~0 at the edges), and use spacing fine enough that the posterior is smooth across neighboring points.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What are the four steps of grid approximation?

    Choose a grid, compute prior and likelihood on it, multiply them, and divide by the sum.
  2. 2. Why subtract the maximum log-posterior before exponentiating?

    Subtracting a constant does not change the normalized result but keeps exp() from underflowing to zero.
  3. 3. How do you get the marginal posterior of mu from a 2-D grid over (mu, sigma)?

    Summing over sigma integrates it out, leaving the distribution of mu.
  4. 4. Why does grid approximation fail for models with many parameters?

    A grid with m points per parameter has m^d cells, which quickly becomes impossible.

Summary

  • Grid approximation multiplies prior and likelihood on a grid of parameter values and normalizes.
  • Draw samples with rng.choice to compute any summary by counting.
  • Use log posteriors and subtract the maximum for stability.
  • Marginal posteriors come from summing over the other parameters' axes.
  • It scales exponentially with the number of parameters, which motivates MCMC.

Next: Choosing Priors