Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 36 ยท Projects and Practice

Project: Ranking Products with Hierarchical Ratings

Rank 40 products with very different numbers of reviews using a hierarchical beta-binomial model, a fully Bayesian grid over hyperparameters, and top-k probabilities.

An online shop shows "percent of customers who liked it" for each product. Product X has 5 out of 5 likes (100%). Product Y has 940 out of 1,000 (94%). Which belongs higher on the ranking page? Sorting by raw percentage puts X first, which nobody believes. In this project you build a ranking that treats few reviews with appropriate caution, using the hierarchical ideas from earlier in this chapter: the model, shrinkage and posterior draws.

The data

We simulate 40 products. Each has a true "like" rate drawn from Beta(16, 4) (average 80%), and the number of reviews varies hugely: some have 3, some 800. Because we know the truth, we can grade several ranking methods.

Python Three rankings against the truth

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both posterior-based scores should track the truth better than raw percentages, and the raw ranking pushes tiny-sample products to the extremes. The posterior mean ranks by "best guess of the true rate". The 5th percentile ranks by "a rate we are 95% sure it exceeds", which penalizes products with few reviews more and is a common choice on rating sites. Which is right depends on the business goal: expected quality, or a safe bet.

A fully Bayesian version

Empirical Bayes fixes (a, b) at one best guess. With only 40 products there is real uncertainty about the population prior. We can account for it using a grid over the hyperparameters: evaluate the marginal likelihood at many (a, b) pairs, sample pairs in proportion to it, and for each pair draw all the product rates.

We parameterize by the population mean m = a / (a + b) and the concentration kappa = a + b; both are easier to put priors on. We use a flat prior on m and a log-uniform prior on kappa over a wide range.

Python Top-5 probability with hyperparameter uncertainty

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Instead of a single rank, each product gets a probability of being in the top 5. Products with plenty of reviews and a high rate have a high probability; a product with 5 perfect reviews may appear with a modest probability, not 100%. This is the honest answer a ranking page can give, and you can display it, for example as "likely top performer".

From model to product decision

  • Display the posterior mean plus a "number of reviews" badge, or sort by a lower credible bound.
  • Explore vs exploit: products with few reviews have wide posteriors. To learn about them quickly, show them occasionally (the idea behind Thompson sampling: draw one value from each posterior and rank by the draws).
  • Review manipulation: a hierarchical model assumes honest reviews. Fake five-star reviews break exchangeability, so combine with fraud detection.
  • Time: ratings drift; weight recent reviews more or model rates as changing over time.
  • Categories: let categories form a second level (a hierarchy inside a hierarchy) so a new product in a strong category starts with a better prior.

Pitfalls

  • A single global prior treats a phone and a book as exchangeable; add a category level if they differ.
  • Do not report the empirical Bayes shrunk value as if it were exact; its uncertainty is real.
  • Never sort by raw percentage when review counts differ by orders of magnitude.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is sorting by raw percentage a poor ranking when review counts differ a lot?

    A product with 5 of 5 likes has a high raw rate but very little evidence.
  2. 2. What does ranking by the 5th percentile of the posterior do?

    A wide posterior (few reviews) has a lower 5th percentile, so uncertain products rank lower.
  3. 3. What is the benefit of the grid over hyperparameters compared with empirical Bayes?

    Empirical Bayes plugs in one best (a, b); the grid averages over plausible values.
  4. 4. How is P(product is in the top 5) computed?

    Draw all rates jointly, rank within each draw, and count.

Summary

  • Partial pooling with a beta-binomial hierarchy gives sensible estimates for products with very few reviews.
  • Rank by posterior mean for expected quality, or by a lower credible bound for a cautious ranking.
  • A grid over (mean, concentration) carries hyperparameter uncertainty into the rankings.
  • Top-k probabilities from joint posterior draws give an honest picture of who is really on top.

Next: Practice: Bayesian Problems