Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 37 ยท Sequential and Bayesian Tests

Multi-Armed Bandits and Thompson Sampling

Compare fixed A/B splits with epsilon-greedy and Thompson sampling bandits, measure regret in a runnable simulation, and learn when bandits are the wrong tool.

A classic A/B test splits traffic evenly for the whole test, even when one variant is obviously worse after the first day. That is the price of clean measurement: you explore the whole time. A multi-armed bandit shifts traffic toward the better variant as evidence builds, so you exploit what you learned while you are still learning.

The name comes from slot machines ("one-armed bandits"). You face several machines with unknown payout rates and want to collect as many rewards as you can. Each variant of your page is an arm; each visitor is a pull.

Explore versus exploit

If you always show the current best variant, you may lock onto a wrong favorite after a lucky start. If you keep splitting evenly, you waste traffic on a loser. Bandit algorithms are different recipes for balancing the two.

Three traffic policies

Fixed split 50/50 until the end
Epsilon-greedy Mostly best arm, a little random
Thompson sampling Pick by sampling each posterior

More exploit means less regret

1 / 2
  • Epsilon-greedy: with probability epsilon (say 10%) show a random arm, otherwise show the arm with the best observed rate.
  • Thompson sampling: keep a Beta posterior per arm (see Bayesian A/B testing), draw one sample from each, and show the arm with the highest draw. Uncertain arms get explored automatically; clear losers fade out.

Simulating regret

Regret is the reward you would have earned by always showing the best arm minus what you actually earned. We simulate three arms with true rates 4%, 5% and 6%.

Python Fixed split vs epsilon-greedy vs Thompson

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both bandits earn clearly more than the fixed split. Thompson sampling sends the largest share of traffic to the 6% arm and keeps shifting as evidence grows, while epsilon-greedy keeps spending a fixed 10% on random exploration. On a real site that difference is revenue gained during the test.

What you give up

Bandits are not a free upgrade. Because traffic is unbalanced and adaptive:

  • Estimates for the losing arms are poor (few samples), so you learn less about why things lost.
  • Classic p-values and confidence intervals are no longer valid without special corrections.
  • Time effects hurt: if the best arm changes by weekday, an early lead can mislead the algorithm.
  • Metrics that arrive late (refunds, 30-day retention) break the feedback loop.
  • It optimizes one metric; guardrail metrics need separate handling.
Python How well does Thompson sampling find the best arm?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When arms are nearly equal, the bandit cannot reliably tell which is better in a short time. That is fine for the business (little is lost either way) but poor for learning.

When to use which

  • Use a bandit for short-lived, low-stakes optimization: headlines on a news site, promotions during a one-week sale, ad creatives, recommendation slots. The goal is reward, not knowledge.
  • Use an A/B test when you need a trustworthy effect estimate for a decision that will last: a pricing page, a checkout redesign, an algorithm change, anything with long-term metrics or guardrails.
  • Hybrids: run an A/B test with a short initial burn-in, or use contextual bandits that pick an arm per user segment (more powerful and much harder to validate).

If you are unsure, start with the fixed A/B test. It is easier to defend in a review, and you can still estimate the loss from not using a bandit using the simulation above.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does Thompson sampling do on each visit?

    Uncertain arms sometimes draw high, so they are explored; clear losers are rarely chosen.
  2. 2. What is regret?

    Regret is the gap between the best possible total reward and the reward actually earned.
  3. 3. Which is a disadvantage of bandits versus a fixed A/B test?

    Unbalanced, adaptive sampling complicates inference and leaves little data on the losers.
  4. 4. Which case favors a bandit?

    Short-lived, low-stakes choices where reward matters more than precise knowledge suit bandits.

Summary

  • Bandits shift traffic toward better arms while the test runs, reducing regret compared with a fixed split.
  • Epsilon-greedy explores at random; Thompson sampling explores in proportion to uncertainty.
  • Costs: weak estimates for losers, invalid classic inference, trouble with delayed metrics and time effects.
  • Use bandits to optimize short-lived choices; use fixed or sequential A/B tests to learn lasting effects.

Next: Why Variance Reduction Matters