Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 37 ยท Getting Started

Randomized Experiments: The Gold Standard

Learn why random assignment removes confounding, how to analyze a randomized experiment with a difference in means and confidence interval, and what can still go wrong.

A randomized controlled trial (RCT), or A/B test in technology, assigns the treatment by chance. It is called the gold standard because it solves the central problem of causal inference by design instead of by modelling. This lesson shows exactly why, how to analyze one, and the ways it can still fail.

Why randomization works

If a coin decides who is treated, then treatment is independent of everything else about a person: age, health, motivation, and also traits nobody measured. On average the two groups look alike before treatment, so any later difference must be due to the treatment (or to chance, which statistics quantifies).

flowchart LR
  C["coin flip"] --> T["treatment"]
  U["hidden traits"] --> Y["outcome"]
  T --> Y

No arrow runs from hidden traits to treatment, so there is no confounding path.

You can check this by looking at balance: pre-treatment variables should have similar averages in both groups.

Python Balance check after randomization

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Averages match closely; differences are small random noise. With observational data, the same table would often show large gaps.

Analyzing an RCT

The simplest estimator is the difference in means, with a standard error and a confidence interval:

TEXT
effect = mean(Y | treated) - mean(Y | control)
se     = sqrt(var1/n1 + var0/n0)
95% CI = effect +/- 1.96 * se
Python Difference in means with a confidence interval

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The interval should usually cover the true effect (2.5) and is wide enough to remind you that a single experiment is noisy. For more on tests, see the statistics course and A/B testing.

Adding covariates for precision

Adjusting for pre-treatment variables in a regression does not fix bias in an RCT (there is none to fix) but it can shrink the standard error, because it removes outcome variation explained by the covariates.

Python Covariate adjustment narrows the interval

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both estimates are unbiased; the adjusted one is more precise. Use only variables measured before treatment.

What can still go wrong?

An RCT protects against confounding of assignment but not against everything:

  • Non-compliance: people assigned to treatment do not take it. The assignment effect is the intention to treat; recovering the effect of actually taking it needs instrumental variables.
  • Attrition: people drop out, and the dropouts differ between groups.
  • Spillovers (interference): treating one person affects another, such as a vaccine or a social feature.
  • Small samples: randomization balances groups only on average.
  • Limited generalization: volunteers in a trial may not resemble everyone.
  • Ethics and cost: you often cannot randomize smoking or poverty.

When an experiment is impossible, the rest of this course teaches how to use observational data to approximate one, by finding the assumptions under which a comparison is as good as random.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does randomization remove confounding?

    Assignment by chance cannot be related to hidden traits.
  2. 2. Adjusting for a pre-treatment covariate in an RCT mainly:

    There is no confounding bias to remove, but explained variation is subtracted from the noise.
  3. 3. Which problem is NOT solved by randomization?

    Attrition after assignment can reintroduce bias.
  4. 4. People assigned to treatment who do not take it make the difference in means an estimate of:

    Comparing by assignment estimates the effect of being offered the treatment.

Summary

  • Random assignment makes groups comparable on average, observed or not.
  • A difference in means with a confidence interval is a sound RCT estimate.
  • Pre-treatment covariates can improve precision but are not needed for validity.
  • Non-compliance, attrition, spillovers and generalization remain concerns.

Next: Confounding