Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 57 ยท Describing Data

Variance and Standard Deviation

Learn how variance and standard deviation measure spread, why the sample version divides by n minus 1, how the empirical rule works, and how to compute them in Python.

Two classes can have the same average score and still be very different: in one, everyone scores near 70; in the other, half score 40 and half score 100. The mean cannot tell them apart. You need a measure of spread (also called dispersion or variability). The most important ones are the variance and its square root, the standard deviation.

Range: the simplest spread

The range is the maximum minus the minimum. It is easy but uses only two values, so a single outlier changes it completely. We need a measure that uses every observation.

Deviations and variance

A deviation is how far a value is from the mean: x - mean. Deviations add to zero, so averaging them tells you nothing. The fix is to square them first. The variance is the average squared deviation.

For a population of N values with mean ฮผ:

sigma^2 = sum((x - mu)^2) / N

Take the data 2, 4, 4, 4, 5, 5, 7, 9. The mean is 5. The squared deviations are 9, 1, 1, 1, 0, 0, 4, 16, which sum to 32. Dividing by 8 gives a population variance of 4.

Standard deviation

Variance is in squared units (dollars squared, cm squared), which is awkward. Taking the square root returns to the original units: sigma = sqrt(variance). For the data above, sigma = 2. The standard deviation reads as "a typical distance from the mean".

Sample variance: why n minus 1?

When your data are a sample, the true mean ฮผ is unknown and you use the sample mean xฬ„ instead. But xฬ„ is, by construction, the point that makes the squared deviations of this sample as small as possible. So the sum is a little too small, and dividing by n would underestimate the real variance. Dividing by n - 1 corrects it:

s^2 = sum((x - xbar)^2) / (n - 1)

This is called Bessel's correction, and n - 1 is the degrees of freedom: once the mean is fixed, only n - 1 of the deviations can vary freely because they must add to zero.

Python Population vs sample variance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The sample variance is 32 / 7 = 4.5714, slightly larger than the population value of 4.

Seeing the bias of dividing by n

A simulation makes the correction believable. Draw many small samples from a population whose variance is exactly 100, and average the two formulas.

Python Why n - 1 is fair

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Dividing by n lands near 80 (that is (n-1)/n of 100), while dividing by n - 1 lands close to 100. In the long run, the corrected formula hits the target, which is what "unbiased" means.

Computational formula and spread by hand

An equivalent form avoids computing deviations: variance = (sum of x^2)/N - mean^2 for a population. It is handy for hand calculation and for streaming data, though it can lose precision with huge numbers on a computer.

The empirical rule

For data that look bell-shaped, the 68-95-99.7 rule says roughly:

  • 68% of values lie within 1 standard deviation of the mean,
  • 95% within 2 standard deviations,
  • 99.7% within 3 standard deviations.

If exam scores have mean 70 and standard deviation 10, then about 95% of students scored between 50 and 90.

Python Checking the 68-95-99.7 rule

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For skewed data the rule fails. Chebyshev's inequality gives a guarantee for any distribution: at least 1 - 1/k^2 of values lie within k standard deviations. With k = 2 that is at least 75%.

Coefficient of variation

To compare spread across different units or scales, divide by the mean: CV = s / mean. A CV of 0.10 means the standard deviation is 10% of the mean. Comparing the variability of a mouse's weight (grams) with an elephant's (kilograms) only makes sense this way.

Standard deviation in action

Spread is not a nuisance, it often is the story: investors measure risk by the standard deviation of returns; factories track it for quality control; teachers use it to judge whether a test separated students well.

Python Same mean, different spread

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both teams average 70, but the standard deviation is about 1.6 for A and 26 for B. You will connect these ideas to the standard error in sampling distributions later, and to random variables, whose variance uses probabilities as weights.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. The sum of squared deviations of a sample of 6 values is 30. What is the sample variance?

    30 / (6 - 1) = 6.
  2. 2. Why is the standard deviation easier to interpret than the variance?

    Taking the square root undoes the squaring and returns to the original units.
  3. 3. Scores have mean 70 and sd 10 and look bell-shaped. About what share scored between 60 and 80?

    That is within one standard deviation of the mean.
  4. 4. Dividing a sample's squared deviations by n instead of n - 1 tends to...

    The sample mean is closer to the sample than the true mean is, so deviations are too small on average.

Summary

  • Spread tells you how much values differ; the range uses only two values.
  • Variance is the average squared deviation; the standard deviation is its square root.
  • Use n - 1 (Bessel's correction) for the sample variance to avoid underestimating.
  • The empirical rule (68-95-99.7) applies to bell-shaped data; Chebyshev applies to any data.
  • The coefficient of variation compares relative spread across scales.

Next: Quartiles, IQR and Outliers