Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 36 Ā· Conjugate Models

The Normal-Normal Model

Estimate a mean from normal data with a normal prior: precision-weighted updates, shrinkage toward the prior, known versus unknown variance, and posterior predictive intervals in Python.

Measurements such as heights, test scores, reaction times and sensor readings are often modeled as normal. The first normal model to learn assumes the noise level σ is known and you want to infer the mean μ. With a normal prior on μ the posterior is again normal, and the update has an intuitive form: a precision-weighted average of the prior mean and the data mean.

The model

Text
yᵢ ~ Normal(μ, σ²)     σ known
μ  ~ Normal(μ₀, τ₀²)

Precision is the inverse of variance: 1/σ². Bigger precision means more certainty. Define:

  • prior precision pā‚€ = 1/τ₀²
  • data precision pd = n/σ² (the sample mean ȳ has variance σ²/n)

The update

posterior precision = pā‚€ + pd

posterior mean = (pā‚€ μ₀ + pd ȳ) / (pā‚€ + pd)

So the posterior is Normal(μₙ, τₙ²) with τₙ² = 1 / (pā‚€ + pd). Information adds: precisions simply sum, and the mean is weighted by precision. The more data you have, the more ȳ dominates, and the posterior is always narrower than both the prior and the data alone.

flowchart LR
    A["Prior mean μ₀<br/>precision pā‚€"] --> C["Weighted average"]
    B["Data mean ȳ<br/>precision n/σ²"] --> C
    C --> D["Posterior mean<br/>precision pā‚€ + n/σ²"]

Precisions add; means are averaged with precision as weight.

Worked example: battery life

A phone maker believes a new battery lasts 20 hours (prior Normal(20, 2²)). A lab test of 5 units gives mean 22.1 hours; the test noise is known to have σ = 3.

Python Normal-Normal update

Press Run (or Ctrl + Enter) to see the result here.

Program input Ā· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The data mean is 22.1, the prior says 20, and the posterior lands in between (about 21.4, with the data getting 69% of the weight), closer to whichever has the greater precision. This pull toward the prior is called shrinkage; we study it in depth in partial pooling (a later chapter).

Prior versus data: a weighting experiment

Watch how the posterior mean moves as the sample grows. The data mean stays at 22 but n increases.

Python More data, less prior

Press Run (or Ctrl + Enter) to see the result here.

Program input Ā· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

By n = 100 the prior weight is under 3%. This is the arithmetic behind "data overwhelms the prior".

Posterior predictive for a new measurement

A new observation includes two sources of uncertainty: the noise σ, and uncertainty in μ. Variances add:

ỹ ~ Normal(μₙ, σ² + τₙ²)

Python Predicting one new unit

Press Run (or Ctrl + Enter) to see the result here.

Program input Ā· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The prediction interval for a single unit is much wider than the interval for the mean. Confusing the two is a frequent error: the mean can be known precisely while individual outcomes remain noisy.

Unknown variance

Usually σ is unknown too. Two options:

  1. Conjugate Normal-Inverse-Gamma prior on (μ, σ²): closed-form posteriors, and the marginal posterior of μ is a Student-t.
  2. Grid or MCMC with separate priors like Normal for μ and HalfNormal for σ.

The grid from Grid Approximation is the simplest path. Here is a sketch of the two-parameter posterior using the closed-form t distribution under a vague prior (the "noninformative" reference analysis): μ | y ~ t(nāˆ’1, ȳ, s²/n).

Python Unknown sigma: Student-t posterior for the mean

Press Run (or Ctrl + Enter) to see the result here.

Program input Ā· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The interval differs from the known-σ one mainly because the sample sd (1.62) is smaller than the assumed 3, partly offset by the price of not knowing the noise level. It matches the classical t-interval numerically, a pleasing case where the two schools agree.

Common mistakes

  • Mixing up standard deviation, variance and precision. SciPy's scale is the standard deviation.
  • Using a posterior interval for the mean as if it were a range for individual outcomes.
  • Treating a tight prior as if it were data: check its precision weight against n/σ².

Check your understanding

Quick check Ā· 4 questions

of 4 answered You scored / 4 Ā· perfect!

  1. 1. In the Normal-Normal model with known sigma, how do precisions combine?

    Posterior precision equals prior precision plus data precision n/sigma^2.
  2. 2. The posterior mean is...

    Each source is weighted by its precision, so the more certain source counts more.
  3. 3. Why is the prediction interval for one new observation wider than the interval for the mean?

    A new observation has both parameter uncertainty and its own measurement noise.
  4. 4. With unknown variance and a vague prior, the posterior of mu is...

    Integrating out the unknown variance yields a t distribution with n-1 degrees of freedom.

Summary

  • Normal prior and normal likelihood with known σ give a normal posterior.
  • Precisions add; the posterior mean is the precision-weighted average of prior and data means.
  • More data reduces the prior's weight and shrinks the posterior sd.
  • A new observation's predictive variance is σ² + τₙ².
  • With unknown σ, use a Normal-Inverse-Gamma prior, a grid, or MCMC; a vague prior gives a Student-t posterior.

Next: The Dirichlet-Multinomial Model