Skip to content
Tutorials Center
My learning

Topics

Lesson 41 of 76 ยท Statistics for Data Science

Estimation and Confidence Intervals

Learn standard error, t-based confidence intervals and the bootstrap in Python, plus what a 95% interval really means, with live simulations you can run.

You almost never see the whole population. You see a sample: 40 support tickets, 2,000 survey answers, one month of orders. From that sample you compute a number, such as an average wait time, and you want to say something about the average wait time of all tickets, past and future. That number is an estimate, and a different sample would have given a slightly different one.

A confidence interval is the honest way to report an estimate. Instead of "the average wait is 8.4 minutes", you say "the average wait is probably between 6.9 and 9.9 minutes". The width of the interval tells the reader how much to trust the number. This lesson shows two ways to build one, the classic t-interval and the bootstrap, and then clears up what the "95%" really promises. It builds on sampling and bias: an interval only describes sampling noise, not a biased sample.

The standard error: how much an estimate wobbles

Imagine drawing many samples of the same size and computing the mean of each. Those means vary, and their spread is called the standard error (SE). For a mean it has a simple formula:

SE = s / sqrt(n)

where s is the sample standard deviation and n the sample size. Two things follow. A noisier variable (large s) gives a less precise mean. And precision improves with the square root of n: to cut the SE in half you need four times as much data, not twice.

Do not confuse the standard deviation with the standard error. The standard deviation describes how much individual values differ. The standard error describes how much your estimate would differ from sample to sample.

A t-based confidence interval

For a mean, the classic interval is the estimate plus and minus a multiple of the standard error:

mean ยฑ t * SE

The multiplier t comes from the Student t distribution, which is a slightly fatter version of the normal curve that accounts for the fact that we had to estimate s from a small sample. For a 95% interval with 40 observations, t is about 2.02; for large samples it approaches 1.96.

Python Standard error and a 95% t-interval

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Note ddof=1 in std: it divides by n - 1, which gives an unbiased estimate of the variance from a sample. The table at the bottom shows the square-root law: every fourfold increase in n halves the standard error.

flowchart LR
    A["Population<br/>(unknown mean)"] --> B["Draw a sample"]
    B --> C["Compute estimate<br/>and standard error"]
    C --> D["Interval:<br/>estimate ยฑ t ร— SE"]
    D -. "repeat with new samples" .-> B

Each new sample gives a new interval; the method is judged by how often those intervals catch the truth.

The t-interval assumes the sampling distribution of the mean is roughly normal. That holds for small samples if the data themselves are roughly symmetric, and for large samples almost always (the central limit theorem). With a small sample of heavily skewed data, such as the waits above, treat it as an approximation.

t or z? Finding the critical value

The multiplier in estimate ยฑ multiplier * SE is a critical value, and which distribution you take it from depends on what you know.

  • z (standard normal): use it when the population standard deviation is known, or the sample is large (roughly 30 or more) so the estimated standard deviation is reliable. For 95% the critical value is 1.96.
  • t (Student): use it when the standard deviation is estimated from a small sample. The t curve has heavier tails, controlled by the degrees of freedom n - 1, so the multiplier is larger and the interval wider.

The recipe has three steps. First compute the standard error s / sqrt(n). Second, look up the critical value: for a confidence level C, you need the point with (1 + C) / 2 of the area to its left. A printed t-table lists these by degrees of freedom; in Python, stats.t.ppf((1 + C) / 2, df) or stats.norm.ppf((1 + C) / 2) reads the same table. Third, apply the formula: mean ยฑ critical value ร— SE.

Some critical values worth knowing (two-sided):

Confidence z t, 10 df t, 30 df
90% 1.645 1.812 1.697
95% 1.960 2.228 2.042
99% 2.576 3.169 2.750
Python Critical values and a t versus z interval

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With 12 values the z interval is a little too narrow, so it promises 95% and delivers less. Use t for small samples; for large samples the two agree, so the choice stops mattering.

The bootstrap: an interval for any statistic

The formula s / sqrt(n) exists for the mean. What about a median, a 90th percentile, a ratio of two means, or the difference between two medians? Formulas get complicated or do not exist.

The bootstrap replaces algebra with computing. The idea: your sample is the best picture of the population you have, so pretend it is the population. Draw many new samples from your sample, with replacement, each the same size as the original. Compute your statistic on each resample. The spread of those values approximates the spread you would see across real samples.

The simplest interval takes the 2.5th and 97.5th percentiles of the bootstrap values, called the percentile interval.

Python Bootstrap intervals for the mean and the median

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The bootstrap needs no formula and works for almost any statistic, which makes it a favorite in practice. Its limits: it cannot invent information that is not in the sample (a sample of 8 is still a sample of 8), it struggles with extremes such as the maximum, and it assumes the observations are independent. For time series or grouped data you must resample whole blocks or groups. SciPy also ships scipy.stats.bootstrap, which adds refinements such as the BCa interval; see bootstrap and correlation.

What "95% confident" really means

This is the most misunderstood idea in statistics. The 95% describes the procedure, not the single interval you computed. If you could repeat the whole study many times and build an interval each time, about 95% of those intervals would contain the true value. Any one interval either contains it or does not, and you do not know which.

So the following statements are wrong, even though they are very common:

  • "There is a 95% probability that the true mean lies in (6.9, 9.9)." The true mean is a fixed number, not a random one. (The Bayesian lesson builds intervals that do allow this sentence.)
  • "95% of the data lie in this interval." The interval is about the mean, not about individuals.
  • "If I repeat the study, the new estimate will fall in this interval 95% of the time." Not exactly; that is a different quantity.

You can see the real meaning by simulation. Here the true mean is known (10), so we can check how many intervals catch it.

Python Coverage: how often do 95% intervals catch the truth?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Coverage comes out just under 95% (about 94%) because the data are skewed and the sample is small; the method is an approximation. With larger samples it gets closer to the promise. The chart shows the key picture: intervals jump around from study to study, and a few miss the truth by design.

Choosing the confidence level and reading width

A 99% interval is wider than a 95% interval, and a 90% interval is narrower. You pay for more certainty with less precision. The 95% level is a convention, not a law; pick the level that fits the cost of being wrong.

Interval width depends on three things: the noise in the data, the sample size, and the confidence level. If an interval is too wide to support a decision, the fix is usually more data or a less noisy measurement, not a different confidence level.

Warning: A confidence interval only covers random sampling error. It says nothing about a biased sample, a broken measurement or a wrong question. A tight interval around a biased estimate is precisely wrong.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A sample of 100 gives a standard error of 2.0. About how large must the sample be to halve the standard error?

    The standard error is s / sqrt(n). To halve it, sqrt(n) must double, so n must quadruple, from 100 to 400.
  2. 2. What does a 95% confidence interval promise?

    The 95% describes the long-run behavior of the procedure. A single computed interval either contains the fixed true value or does not.
  3. 3. Why use a bootstrap interval for a median?

    The bootstrap approximates the sampling distribution by recomputing the statistic on resamples. It cannot fix bias or create new information.
  4. 4. You change a 95% interval to a 99% interval using the same data. What happens?

    Greater confidence needs a larger multiplier, so the interval widens around the same estimate.

Summary

  • An estimate from a sample varies; the standard error s / sqrt(n) measures how much, and shrinks with the square root of n.
  • The t-interval mean ยฑ t * SE is the classic interval for a mean; it relies on a roughly normal sampling distribution.
  • The bootstrap resamples your data with replacement and works for medians, ratios and other statistics without formulas.
  • "95% confidence" is a property of the method over repeated studies, not a 95% probability for one interval.
  • Intervals cover sampling noise only; bias and bad measurement are not included.

Next: Hypothesis Testing Done Right, where intervals and decisions meet.