Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 37 ยท Practice and Pitfalls

Common Causal Pitfalls

A field guide to causal analysis mistakes: bad controls, collider bias, adjusting for mediators, immortal time, regression to the mean, interference and p-hacking, with simulations.

Most failed causal analyses do not fail because of a hard method. They fail because of a quiet mistake in the setup: adjusting for the wrong variable, defining groups using the future, or treating a convenient sample as if it were the population. This lesson is a checklist of the mistakes seen most often, with small simulations so you can see the damage.

1. "More controls is always better"

Adjusting for a variable is only good if it blocks a confounding path. Some variables make things worse.

  • Confounders (cause both treatment and outcome): adjust.
  • Mediators (caused by treatment, cause the outcome): adjusting removes part of the effect you want.
  • Colliders (caused by both treatment and outcome, or their causes): adjusting creates a spurious association.
  • Descendants of the outcome: adjusting is usually harmful.
Python Adjusting for a mediator hides the effect

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Controlling for the mediator answers a different question (the direct effect), and even that is biased if the mediator and outcome share an unmeasured cause. If you want the total effect, leave mediators out.

2. Collider bias and selection

If you condition on a variable that both treatment and outcome influence, you open a path between them. Selecting a sample (only hospitalized patients, only admitted students, only people who responded) is conditioning on a collider.

Python Collider bias: two unrelated traits become negatively correlated

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Among the famous, talent and looks look negatively related: someone who got in with little talent must have had a lot of looks. The same logic explains "paradoxes" like the apparent protection of a risk factor among hospital patients (Berkson's bias). See selection and collider bias.

3. A quick catalog of other traps

Pitfall What goes wrong Remedy
Post-treatment variables Adjusting for or conditioning on something treatment changed (for example "users who clicked") Define groups and covariates before treatment
Immortal time bias Treated group must survive long enough to get treated, so their early survival is guaranteed Align time zero, use landmark or time-varying methods
Regression to the mean Units picked for extreme values drift back toward average, which looks like an effect Use a control group selected the same way
Simpson's paradox The trend reverses when data is aggregated Decide from the causal diagram whether to stratify; see Simpson's paradox
Interference (SUTVA violation) One unit's treatment affects another unit's outcome (network effects, marketplaces) Randomize clusters, model spillovers
Survivorship bias Only successes are observed Include the full starting population
Positivity violations No treated (or control) units in some covariate region Check overlap, restrict the population
Measurement error in confounders Adjustment is only partial Better measurement, sensitivity analysis
P-hacking and forking paths Trying many specifications until one "works" Pre-register, report all analyses
Extrapolating a LATE or local effect RDD and IV effects hold only for some units State exactly who the estimate applies to

4. Regression to the mean in a simulation

A very common business mistake: select the worst-performing stores, give them a training program, and celebrate when they improve. Without a control group, much of the improvement is just noise reverting.

Python Selecting extreme units creates a fake improvement

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The worst stores "improve" by more than a full unit with no program at all. A randomized or matched control group chosen with the same rule would improve by the same amount, and the program's effect is the difference between them.

A pre-analysis checklist

flowchart TD
    A["State the causal question and the estimand"] --> B["Draw the causal diagram"]
    B --> C["Choose adjustment set: confounders only"]
    C --> D["Define treatment time and groups before looking at outcomes"]
    D --> E["Check overlap and balance"]
    E --> F["Estimate with a method suited to the design"]
    F --> G["Sensitivity and falsification checks"]
    G --> H["Report assumptions and limits plainly"]

Writing this down before analysis protects you from your own flexibility.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You want the total effect of a treatment. Should you control for a mediator?

    Mediators carry part of the effect; adjusting for them removes it.
  2. 2. What is collider bias?

    Conditioning on a variable caused by both opens a non-causal path between them.
  3. 3. Why is picking the worst performers and giving them an intervention problematic without a control group?

    Part of an extreme observation is noise, which does not repeat.
  4. 4. What does a SUTVA/interference violation mean?

    Standard methods assume each unit's outcome depends only on its own treatment.

Summary

  • Adjust for confounders; do not adjust for mediators, colliders or other post-treatment variables.
  • Selecting or conditioning on a common effect creates spurious associations.
  • Watch for regression to the mean, immortal time, survivorship and interference.
  • Fix the question, diagram and analysis plan before touching outcome data, and report sensitivity checks.

Next: Project: Coupon Targeting with Uplift