Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 37 ยท Uplift and Heterogeneous Effects

Heterogeneous Treatment Effects

Go beyond the average effect: define the CATE, estimate subgroup effects and interactions in a randomized experiment, and avoid the false discoveries that come from slicing data too many ways.

An average treatment effect of +2 can hide a lot. Maybe the treatment gives +6 to half of the people and -2 to the other half. Maybe it only helps beginners and does nothing for experts. For decisions such as "who should get the discount?" or "which patients should get the drug?", the average is not enough. You want to know how the effect varies.

This is the study of heterogeneous treatment effects (HTE). The central quantity is the conditional average treatment effect (CATE):

tau(x) = E[Y(1) - Y(0) | X = x]

It is the average effect for units whose covariates equal x. The ATE is just the average of tau(x) over the population.

Start with a randomized experiment

HTE is easiest to learn when treatment is randomized, because then confounding is not a worry and you can concentrate on variation. We simulate an experiment where the effect depends on a customer's tenure: new customers benefit, long-time customers do not.

Python Subgroup effects in a randomized experiment

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The overall ATE is near 0, which would suggest the campaign does nothing. The bands tell a very different story: a clearly positive effect for new customers shrinking to a negative one for the oldest. That is the value of looking at heterogeneity.

Interaction regression

A compact way to estimate a trend in the effect is to include an interaction between treatment and the covariate:

Y = a + b*X + c*T + d*(T x X) + error

The effect at x is c + d*x. The coefficient d says how much the effect changes per unit of x, and its standard error tells you whether the heterogeneity is real. A common practical tip: center X first, so c is the effect at the average customer.

Python Linear effect modification with an interaction term

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The interaction coefficient is close to the true slope of -0.8, with a small standard error. Interactions are easy to interpret, but they assume the effect is linear in x. Meta-learners relax that using flexible machine learning models.

The danger: slicing until something is significant

If you test 20 subgroups at the 5% level, you expect one "significant" result even when the true effect is the same for everyone. Reporting that subgroup as a finding is the forking paths problem. The simulation below has no heterogeneity at all.

Python False discoveries from many subgroups

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In a single experiment you may see none, one or two such findings, but across many experiments the average is one, and about two experiments in three produce at least one "discovery" even though the effect is identical for everyone. Defenses:

  1. Pre-register the subgroups you will examine, with a reason for each.
  2. Correct for multiplicity (Bonferroni, Holm, false discovery rate).
  3. Test the heterogeneity itself, for example with an interaction test, instead of testing each subgroup against zero.
  4. Validate on held-out data, or in a follow-up experiment.
  5. Prefer a few theory-driven effect modifiers over a data dredge.

Observational data

When treatment is not randomized, CATE estimation needs the same assumptions as before (no unmeasured confounding and overlap) at every x of interest, and the estimates are harder because you must model both the confounding and the effect variation. The tools that handle this, such as meta-learners, causal forests and double machine learning, are the subject of the next lessons.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the CATE tau(x) represent?

    CATE = E[Y(1) - Y(0) | X = x]; the ATE averages it over the population.
  2. 2. In Y = a + bX + cT + d*(T x X), what does d tell you?

    The treatment effect at x equals c + d*x, so d is the change in effect per unit of X.
  3. 3. An overall ATE of zero means:

    An average can hide opposing effects in different groups.
  4. 4. Why is testing many subgroups risky?

    At the 5% level you expect about one false positive per 20 independent tests.

Summary

  • The CATE is the effect for units with covariates x; the ATE hides variation.
  • In experiments, estimate heterogeneity with subgroup effects or treatment-by-covariate interactions.
  • Center covariates so main effects are interpretable.
  • Beware forking paths: pre-specify, correct for multiplicity, validate.
  • Observational CATEs need unconfoundedness and overlap plus flexible modeling.

Next: Meta-Learners