Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 76 ยท Getting Data

Sampling and Collection Bias

Learn population vs sample, random, stratified and cluster sampling, and how selection, survivorship and response bias distort data, with live simulations.

Almost no dataset contains everything. A survey reaches a thousand people out of millions, a log covers one week, a table holds the customers who stayed long enough to be recorded. Data science is mostly the art of learning about a population (everything you care about) from a sample (the part you actually observed). Whether that works depends less on how much data you have than on how it was collected.

This lesson shows three things. First, how a random sample behaves and why sample size matters. Second, the common ways of sampling on purpose (stratified and cluster). Third, the biases that quietly break a sample, which no amount of clever modelling will repair afterwards. Every example is a simulation with a fixed seed, so you know the true answer and can see exactly how far the sample is from it.

Population, sample and the target

Three words are worth defining carefully:

  • The population is the group your question is about: all customers, all patients, all web visits next month.
  • The sampling frame is the list you can actually draw from: the customers in your CRM, the people with a phone number.
  • The sample is the subset you end up measuring.

The gap between the population and the frame is the first place bias enters. If you want to know what all adults think but your frame is "people who visited our website", you will learn about website visitors, however carefully you sample them.

flowchart LR
    A["Population<br/>(who you care about)"] --> B["Sampling frame<br/>(who you can reach)"]
    B --> C["Sample<br/>(who you measure)"]
    C --> D["Estimate"]
    D -.->|"generalize"| A

Every step loses something. Bias can enter at any arrow; random error enters only at the last one.

Random sampling and sample size

In a simple random sample every member of the frame has the same chance of being picked. The sample mean then lands near the population mean, sometimes a little above and sometimes a little below. The more you sample, the tighter it clusters. The simulation below builds a population of 100,000 people, then draws 500 samples at each of three sizes and measures how much the sample mean wobbles.

Python How sample size changes the wobble

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two things stand out. The typical sample mean is almost exactly the true mean at every size: random sampling is unbiased. But the spread shrinks as the sample grows. Going from 30 to 300 people (ten times as many) cuts the spread by about a factor of three, and going from 300 to 3,000 cuts it by about three again. The spread falls with the square root of the sample size, so to halve your uncertainty you need four times the data. That is why the first few hundred observations are valuable and the next million often are not.

Note: Sample size only controls random error. A biased sample of a million is still biased. We return to measuring that uncertainty in Estimation and Confidence Intervals.

Sampling on purpose: stratified and cluster

Pure random sampling is the baseline, but it is not always the best design.

Stratified sampling splits the population into groups (strata) that matter to your question, such as region or age band, and samples within each group. If you sample the same fraction from every stratum (proportional allocation), the sample mirrors the population on that variable exactly, instead of only on average. You can also oversample a small group so you have enough of it to analyze, and then weight the results back.

Cluster sampling picks whole groups (a school, a store, a village) at random and then measures everyone, or a random subset, inside them. It is cheaper when visiting a group has a fixed cost, but people in one cluster tend to resemble each other, so you learn less per person than with a random sample of the same size.

The next example compares a simple random sample with a stratified one, and also shows a convenience sample: only people who are reachable online. In this population, younger people are more likely to be online.

Python Random, stratified and online-only samples

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The random sample is close to the population on both rows. The online-only sample has the same size, yet its mean age is about seven years too low and its mean income about $4,000 too low, because income grows with age in this population. Notice that the sample is not small: 1,000 people is plenty. The problem is who got in.

The second table shows the benefit of stratifying: the stratified shares match the population exactly (0.501, 0.301, 0.198), while the random sample is off by up to a percentage point or so by chance.

Here is cluster sampling in a few lines. We pick 5 of 100 stores at random and measure all their customers:

Python Cluster sampling: pick whole stores

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both designs measure 1,000 customers, yet the cluster design is noticeably less precise, because customers of one store resemble each other. You accept that cost when visiting five stores is much cheaper than visiting a hundred.

The biases that sample size cannot fix

Bias is a systematic error: it pushes the estimate the same way every time. Three kinds appear constantly.

Selection bias means the way people entered the sample is related to what you measure. The online-only sample above is an example. Others: surveying shoppers at a luxury store, or analyzing only users who completed onboarding.

Survivorship bias means you only see what survived some filter. If you study the funds that still exist today, you miss the ones that closed after poor results, and the average looks better than reality. Famous examples include studying only successful companies, or only the planes that returned from missions.

Response (non-response) bias means the people who answer differ from those who do not. Customers who are very happy or very angry reply to satisfaction surveys more than those who feel neutral.

The next example simulates two of them with known truth.

Python Survivorship and response bias

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The survivors look about three percentage points better per year than the whole group, even though no fund was changed. The survey has thousands of replies, yet the replying customers are about 1.3 points happier than the customer base. In this simulation, customers far from the neutral score of 5 reply more often, and most customers sit above 5, so the replies lean toward the delighted end.

What to do about bias

You cannot test for bias using only the sample; you have to reason about how the data came to exist. Useful questions:

  1. Who or what could have been left out, and would they differ on the thing I measure?
  2. Did anything filter the records (a deleted account, a completed form, a surviving company)?
  3. Can I compare the sample with a known truth, such as census shares or total revenue?
  4. If the answer is "the sample is skewed", can I reweight, collect more from missing groups, or at least state the limit of what I can claim?

Reweighting helps when you know the population shares of a variable that explains the bias, but it only corrects what you measured. Honest wording matters too: "among customers who answered the survey" is a true statement; "among all customers" may not be.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A random sample of 100 people and a random sample of 10,000 people are both unbiased. What is the main difference between them?

    Random error shrinks with the square root of the sample size. Bias is a separate problem that more data does not fix.
  2. 2. You survey only people who visit your website to learn what all adults think. Which problem is this?

    The frame (website visitors) differs from the population (all adults), so the sample can be systematically different even if it is large.
  3. 3. Why use stratified sampling?

    Sampling the same fraction within each group guarantees that the group shares match the population, and lets you secure enough cases from small groups.
  4. 4. A study looks only at companies that are still in business today. Which bias is most likely?

    Companies that failed were filtered out, so the remaining ones look more successful than companies in general.
  5. 5. Why is cluster sampling usually less precise than a simple random sample of the same size?

    Similar members within a cluster repeat the same information, so the effective sample size is smaller than the head count.

Summary

  • A sample is only useful for what it can say about the population; check the gap between the population, the frame and the sample.
  • Random sampling is unbiased, and its random error shrinks with the square root of the sample size (four times the data halves the spread).
  • Stratified sampling matches the population on key groups; cluster sampling is cheaper but less precise per person.
  • Selection, survivorship and response bias are systematic: more data does not remove them.
  • Reason about how each row came to exist, compare with known totals, and word conclusions to match who was actually measured.

Next: Generating Synthetic Data for Practice, where you build your own data with a known truth.