Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 36 · Conjugate Models

The Dirichlet-Multinomial Model

Learn the Dirichlet-Multinomial conjugate model for categorical data: pseudo-counts, posterior updates, sampling proportions, comparing categories and posterior predictive probabilities.

The Beta-Binomial model handles two outcomes: success or failure. Many problems have more: which of five plans a customer picks, which of six faces a die shows, which of 20 topics a document covers. The Dirichlet-Multinomial model is the natural generalization, and it is still fully conjugate.

The model

You count K categories among n observations, giving counts c = (c₁, …, c_K) with Σcⱼ = n. The unknown proportions θ = (θ₁, …, θ_K) are non-negative and sum to 1.

Text
c ~ Multinomial(n, θ)
θ ~ Dirichlet(α₁, …, α_K)

The Dirichlet distribution is a distribution over proportion vectors. With K = 2 it is exactly a Beta. The parameters αⱼ act as pseudo-counts for each category:

  • Mean of θⱼ is αⱼ / Σα.
  • Σα is the prior strength.
  • All αⱼ = 1 is a uniform prior over all proportion vectors.

The update rule

Add the observed counts to the prior pseudo-counts:

posterior = Dirichlet(α₁ + c₁, …, α_K + c_K)

Beta grows up

Beta 2 categories
Dirichlet K categories

Same add-the-counts rule.

1 / 2

Example: is the die fair?

You roll a die 60 times and count each face: [6, 8, 10, 14, 11, 11]. A fair die has every proportion 1/6 ≈ 0.167.

Python Dirichlet posterior for a die

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each sampled row is a full set of proportions, so you can ask any question about the six numbers together. Face 4 appeared 14 times, and the data leans toward it being above 1/6, but not conclusively.

Marginals are Beta

Every single category of a Dirichlet is a Beta: θⱼ ~ Beta(αⱼ, Σα − αⱼ). That lets you use closed forms when you care about one category only.

Python Marginal of one category is Beta

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

These numbers match the sampling result above, as they should.

Comparing categories

Because draws are joint, you can compare two categories directly, for example whether face 4 is more likely than face 1.

Python Comparing two categories

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The "most likely face" distribution shows honest uncertainty: no single face is certain to be the biggest, because 60 rolls are few.

Posterior predictive

The probability that the next roll is face j is the posterior mean αⱼ' / Σα'. For several future rolls, the predictive distribution is Dirichlet-Multinomial (also called the multivariate Pólya distribution), which is more spread out than a Multinomial at a fixed θ.

Python Predicting the next 10 rolls

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Choosing the prior pseudo-counts

  • α = 1 per category: uniform; each category counts as one prior observation.
  • α = 0.5: Jeffreys-like; favors sparse proportions.
  • α < 1: encourages a few categories to dominate (sparse).
  • α = 10 per category: strong belief in balance (a fair die).
  • Set αⱼ = s · mⱼ with prior strength s and prior guess mⱼ (summing to 1) to encode a specific belief.

Where it shows up

  • Survey and polling shares (candidates, products).
  • Word or topic frequencies in text models (Latent Dirichlet Allocation uses Dirichlet priors).
  • Genetics (allele frequencies) and A/B/n tests with more than two variants.

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. What is the posterior when a Dirichlet(1, 1, 1) prior meets counts (4, 2, 0)?

    Add the counts category by category: (1+4, 1+2, 1+0) = (5, 3, 1).
  2. 2. What is the marginal distribution of a single category in a Dirichlet?

    Each coordinate is Beta(alpha_j, sum of the other alphas).
  3. 3. What does alpha_j / sum(alpha) represent?

    It is the Dirichlet mean of theta_j.
  4. 4. Why use joint Dirichlet draws to compare two categories?

    Because proportions are negatively dependent, comparing them needs joint draws.

Summary

  • Dirichlet is the multivariate Beta, and its parameters are pseudo-counts per category.
  • Posterior = Dirichlet(prior pseudo-counts + observed counts).
  • Each category's marginal is a Beta; joint draws allow comparisons between categories.
  • The predictive for future counts is Dirichlet-Multinomial, wider than Multinomial at a fixed θ.
  • Choose pseudo-counts to express strength and a prior guess; α < 1 favors sparse proportions.

Next: Why MCMC? Sampling from Posteriors