Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 38 ยท Getting Started

Confidence Intervals for Monte Carlo Estimates

Report a Monte Carlo answer with an honest error bar: central limit theorem, 95% intervals, coverage checks and choosing the sample size.

A Monte Carlo answer without an error bar is only half an answer. "The probability is 0.31" could mean anything from "very sure" to "wild guess". A confidence interval states the range in which the true value probably lies. In this lesson you will build one from a single run, check that it behaves as promised, and use it to choose how many samples to draw.

The central limit theorem

The law of large numbers says the sample mean approaches the true mean. The central limit theorem (CLT) says more: for large n, the sample mean is approximately normally distributed (the bell curve) around the true mean ฮผ, with standard deviation equal to the standard error ฯƒ/โˆšn. This holds even when single samples are far from bell-shaped, such as 0/1 outcomes.

For a normal distribution, about 68% of values lie within 1 standard deviation of the center, about 95% within 1.96, and about 99% within 2.576.

The 95% interval

Using the CLT, a 95% confidence interval for ฮผ is

Text
xฬ„ ยฑ 1.96 ร— s / โˆšn

where xฬ„ is the sample mean and s the sample standard deviation. The meaning is subtle: if you repeated the whole simulation many times and built an interval each time, about 95% of those intervals would contain the true value. It does not say "the true value is in this interval with probability 95%" (the true value is a fixed number; the interval is the random thing).

Confidence Multiplier z
90% 1.645
95% 1.960
99% 2.576

A reusable helper

Python Mean with a confidence interval

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check the exact answer by counting: the combinations of three dice with sum 15, 16, 17 or 18 number 10 + 6 + 3 + 1 = 20 out of 216. The interval should contain 0.0926 most of the time.

Does the interval really cover 95%?

You should never take a method on trust when you can test it. This experiment builds 1,000 intervals and counts how many contain the truth. That fraction is the coverage.

Python Check the coverage of the interval

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The coverage should land near 0.95, perhaps a little off. Coverage can fall below target for very rare events and small n, because the CLT needs enough data. A rule of thumb: you want at least about 10 events and 10 non-events in your sample.

Choosing the sample size

You can turn the formula around. Suppose you want a margin of error E (the half-width of the interval). Then

Text
n โ‰ˆ (z ร— ฯƒ / E)ยฒ

For a probability near p, ฯƒยฒ = p(1-p), which is at most 0.25 when p = 0.5. The conservative choice is n โ‰ˆ (1.96 ร— 0.5 / E)ยฒ.

Python How many samples for a given margin?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Before the main run you often do a small pilot run to estimate ฯƒ, then use this formula to pick n.

Relative error for rare events

For a rare event with probability p, an absolute margin such as ยฑ0.001 may be meaningless if p itself is 0.0005. Then think in relative error, SE / p. For a probability estimate, SE/p โ‰ˆ โˆš((1-p)/(np)), so rare events need n of order 1/p just to see the event happen at all. This is the motivation for the variance reduction chapter.

Always print three things: the estimate, its standard error, and the number of samples. Anyone reading your result can then judge its quality.

Common mistakes

  • Using ddof=0 instead of ddof=1 for the sample standard deviation. For large n it barely matters, but ddof=1 is the correct habit.
  • Applying the normal interval when the event happened only once or twice.
  • Interpreting "95%" as the probability that this specific interval is right.
  • Forgetting that samples must be independent. Correlated samples (for example consecutive steps of a chain) make the standard error larger than the formula says.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A simulation gives mean 0.40 and standard error 0.01. What is the approximate 95% confidence interval?

    The interval is mean ยฑ 1.96 ร— SE โ‰ˆ 0.40 ยฑ 0.02, which is [0.38, 0.42].
  2. 2. What does the central limit theorem tell us?

    For large n the distribution of the sample mean is close to normal, which justifies the interval formula.
  3. 3. You want a margin of error that is half as large. How must n change?

    Since n โ‰ˆ (zฯƒ/E)ยฒ, halving E multiplies n by 4.
  4. 4. Why is a pilot run useful?

    The pilot gives a rough sample standard deviation that feeds the formula for the required n.

Summary

  • The central limit theorem makes sample means approximately normal, which gives the interval xฬ„ ยฑ z ร— s/โˆšn (z = 1.96 for 95%).
  • You can test an interval's coverage by repeating the experiment many times.
  • Sample size for margin E is about (zฯƒ/E)ยฒ; use a pilot run to estimate ฯƒ.
  • Rare events need many samples for a useful relative error.
  • Samples must be independent for these formulas to hold.

Next: Simulating Dice, Cards and Games