Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 57 ยท Sampling and Experiments

Experiments and Sources of Bias

Learn observational studies versus randomized experiments, control groups, randomization, blinding, confounding and the common sources of bias, with a Python simulation.

Collecting data is not only about whom you pick; it is also about what you do with them. Does a new teaching method improve scores? Does a drug lower blood pressure? Does a new page layout increase sign-ups? To answer a cause-and-effect question you need the right study design. A badly designed study can produce a precise, confident and completely wrong conclusion. This lesson covers the two basic kinds of study and the traps to avoid.

Observational studies versus experiments

In an observational study you measure what happens naturally and do not interfere. You might compare people who already exercise with people who do not. You can find associations, but not prove causes.

In an experiment the researcher assigns the treatments. If the assignment is done by chance (randomization), the groups will be alike on average in everything except the treatment, so any consistent difference in the outcome can be attributed to the treatment.

Observational study Randomized experiment
Who chooses the group? The subjects themselves or nature The researcher, by chance
Can show causation? No, only association Yes, if well designed
Typical problem Confounding Cost, ethics, artificial settings

Confounding

A confounding variable is related to both the group and the outcome, mixing up their effects. People who drink coffee may have more heart disease, but coffee drinkers also smoke more often. Smoking is a confounder: it makes coffee look harmful even if coffee has no effect.

flowchart LR
    S["Smoking (confounder)"] --> C["Coffee drinking"]
    S --> H["Heart disease"]
    C -. "apparent link" .-> H

The confounder drives both variables, creating a link between them.

The elements of a good experiment

  1. Control group. A comparison group that does not receive the treatment (or receives the standard one). Without it you cannot tell treatment effects from natural change.
  2. Random assignment. Subjects are allocated to groups by chance, balancing known and unknown confounders.
  3. Replication. Enough subjects in each group that real effects stand out from random variation.
  4. Blinding. In a single-blind study subjects do not know which treatment they get; in a double-blind study neither subjects nor the people measuring outcomes know. This guards against the placebo effect (people improve because they expect to) and against observer bias.
  5. Blocking. If a variable is known to matter (age, sex, field), group subjects by it first, then randomize within each block. It is the experimental cousin of stratified sampling.

Randomization in action

Suppose older patients recover more slowly. If the doctor hands the new treatment mostly to younger patients, the treatment will look better than it is. Random assignment removes that choice.

Python Why randomization removes confounding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When doctors choose, the estimate is inflated because the treated group is younger. With random assignment the groups have similar ages and the estimate is close to the truth of 2.

Doing a random assignment

Random assignment is easy to program. Shuffle the participants and split them.

Python Randomly assigning participants

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common sources of bias

  • Selection bias: the sample or the groups are not comparable (self-selected volunteers).
  • Nonresponse bias: people who do not answer differ from those who do.
  • Response bias: answers are untruthful, or the question's wording steers respondents ("Don't you agree that...?").
  • Survivorship bias: only the successes are visible; failures dropped out of the data.
  • Measurement bias: a miscalibrated instrument or a poor definition.
  • Confirmation and publication bias: looking for, or only publishing, results that support a hypothesis.

Association is not causation, but association is not worthless either. Observational studies guide where to run experiments, and for some questions (smoking and cancer) experiments are unethical, so careful observational evidence is all we have.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main reason to randomly assign subjects to treatments?

    Chance assignment makes groups comparable on both known and unknown variables.
  2. 2. In a double-blind trial:

    Double-blinding hides the assignment from both participants and evaluators.
  3. 3. Ice-cream sales and drowning deaths rise together in summer. Temperature is a:

    Hot weather influences both variables, producing a misleading association.
  4. 4. Which design can show cause and effect?

    Random assignment with a control group lets you attribute differences to the treatment.

Summary

  • Observational studies find associations; randomized experiments can establish causation.
  • Confounders mix up effects; randomization, control groups, blocking and blinding defend against them.
  • Bias is systematic and cannot be cured by a larger sample.
  • Know the usual suspects: selection, nonresponse, response, survivorship and measurement bias.

Next: Point Estimation