Skip to content
Tutorials Center
My learning

Topics

Lesson 43 of 49 ยท Causal Methods and Practice

Chapter 8 review: causal designs

Review chapter 8: potential outcomes, randomized trials, difference-in-differences and regression discontinuity in one picture, a table, two checks and a quiz.

The thread running through chapter 8 is simple to say and hard to practise: a causal estimate is only as good as the comparison behind it. You learned to start from the question, to name the counterfactual you need, and then to choose a design that supplies a credible stand-in for it. Each design has a signature assumption, and each assumption has checks that can embarrass it.

Three columns: define the effect, pick a design, and test what you can.
Chapter 8 in one picture: name the effect, pick the design, test what you can.

What you will learn

  • A compact comparison of RCTs, DiD and RD: what each assumes and how to check it.
  • Two code checks: a placebo test for DiD, and a comparison of local linear with global polynomial RD.
  • Ten mixed questions to test the chapter.

Key terms

ATE / ATT / ITT / LATE: average effects for the population, the treated, those assigned, and compliers. Parallel trends: the DiD identifying assumption. Continuity: the RD identifying assumption. Placebo test: applying the method where the truth is known to be zero. Internal validity: is the estimate right for the studied units? External validity: does it generalize?

Recap table

Design Comparison Key assumption Typical checks Estimand
RCT Treated vs control by coin flip Random assignment; no spillovers (SUTVA); little attrition Balance table, attrition by arm, power ATE (ITT with noncompliance)
IV (chapter 6) Variation induced by an instrument Relevance, exclusion, monotonicity First-stage F, argument, J test LATE
Difference-in-differences Change over time, treated vs control Parallel trends Pre-trends, event study, placebo date ATT
Regression discontinuity Just above vs just below a cutoff Continuity of potential outcomes at c Density, covariate jumps, placebo cutoffs, bandwidths Local effect at c
Controls / matching Units with similar covariates No unobserved confounders (conditional independence) Sensitivity analysis, balance ATE or ATT

Four lessons to take away:

  1. Start with the counterfactual. Write down who is being compared with whom, and why they would have had the same outcome without treatment.
  2. Local versus global. RD is local to the cutoff, IV and fuzzy RD to compliers, DiD to the treated. Say whose effect you have.
  3. Checks can fail you, but cannot prove you right. Balanced covariates and flat pre-trends make the story consistent; they do not guarantee the unobservable part.
  4. Design beats estimator. A fancier estimator never rescues a weak comparison.

Check 1: a placebo test for DiD

A convincing DiD analysis estimates a "treatment effect" at a fake date before the real policy; if the method reports an effect there, parallel trends fail. The program below creates two panels: one where trends are parallel before treatment and one where the treated group was already trending upward. It runs the placebo test in both.

Python Placebo DiD on pre-treatment periods

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With parallel pre-trends the placebo estimate is near zero. When the treated group was already trending, the placebo estimate is clearly positive, and the real DiD is inflated too: the placebo test caught the problem that the headline estimate hid.

Check 2: local linear versus a global polynomial in RD

Gelman and Imbens (2019) warned against fitting high-order polynomials across the whole range of the running variable. Here is why: the true function is smooth and curved with no jump at all, and we ask each method to estimate a jump at the cutoff.

Python No true jump: which method is fooled?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The straight global line is biased because it cannot follow the curve; the quartic is nearly unbiased here but much noisier; the local linear fit is close to zero with moderate variance. The lesson: judge an RD method by both bias and variance, and show robustness to specification.

Common misconceptions

"A significant coefficient with many controls is causal." Without a credible design the controls only describe the conditional association.

"Every design estimates the same effect." RD speaks about the cutoff, IV about compliers, DiD about the treated. Different estimates need not conflict.

"Passing placebo tests proves validity." They fail to refute; they cannot confirm unobservable assumptions.

Check your understanding

Quick check ยท 10 questions

of 10 answered You scored / 10 ยท perfect!

  1. 1. Which design relies on parallel trends?

    DiD requires that without treatment, the treated and control groups would have changed by the same amount.
  2. 2. The effect estimated by a sharp RD applies to...

    RD is local: the jump is measured at the threshold.
  3. 3. In a trial where 70% of those offered the program take it up and nobody else does, ITT = 2.1. The LATE is...

    LATE = 2.1 / 0.7 = 3.0.
  4. 4. A naive treated-versus-untreated comparison equals...

    The decomposition: E[Y|D=1] - E[Y|D=0] = ATT + (E[Y(0)|D=1] - E[Y(0)|D=0]).
  5. 5. A placebo DiD at a fake pre-treatment date finds a large effect. This suggests...

    An effect where none should exist signals pre-existing differential trends.
  6. 6. Which is NOT a standard RD validity check?

    A sharp RD has no first stage; the F rule applies to fuzzy designs and IV.
  7. 7. Why should you not adjust for post-treatment variables in an RCT?

    Controlling for an outcome of the treatment removes part of the effect or creates selection.
  8. 8. Staggered adoption with heterogeneous effects can make two-way fixed effects misleading because...

    Using treated units as controls with changing effects can produce negative weights.
  9. 9. In RD, a smaller bandwidth usually...

    Units near the cutoff are more comparable, but there are fewer of them.
  10. 10. The ATE differs from the ATT when...

    If the treated have different average effects than the population, the two averages differ.

Summary

  • A causal estimate rests on a credible comparison: randomization, parallel trends, continuity or an instrument.
  • RCTs give the ATE (ITT with noncompliance); DiD the effect on the treated; RD a local effect at the cutoff; IV the effect for compliers.
  • Run the checks each design invites (balance, event studies, placebo tests, density and bandwidth sensitivity) and report them.
  • Prefer local, transparent specifications; design matters more than estimator sophistication.

Next: Chapter 9 project: diagnosing a wage regression