Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 37 ยท Matching and Weighting

Balance Diagnostics and Overlap

Check whether matching or weighting worked: standardized mean differences, love plots as tables, propensity score overlap, effective sample size and what to do when overlap fails.

Fitting a propensity model is not the finish line. The model is only a means to an end: covariates that look the same in the treated and control groups. Balance diagnostics tell you whether you got there, and overlap diagnostics tell you whether it was even possible. Both are checked before you look at any outcome.

Standardized mean difference

The most common balance measure is the standardized mean difference (SMD) for each covariate:

SMD = (mean_treated - mean_control) / sqrt((var_treated + var_control) / 2)

Dividing by the pooled standard deviation makes it unit-free, so you can compare age in years with income in dollars. A frequent rule of thumb: an absolute SMD below 0.1 is good balance, above 0.25 is a problem. Do not use p-values for balance: they depend on sample size, and matching shrinks the sample, so a p-value can rise just because you threw data away.

A Love plot shows SMDs before and after adjustment as dots on a vertical axis. Here we print the same information as a table.

Python SMD before and after weighting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The "before" column should show clear imbalance (age and income especially) and the "after" column values near zero. If a covariate stays above 0.1, improve the model: add interactions, squared terms or splines, then check again. Also compare variances (ratio between 0.5 and 2) and, for important variables, the whole distribution.

Overlap (positivity)

Causal estimates need comparable units in both groups: for each covariate pattern, the probability of treatment must lie strictly between 0 and 1. Without it, you extrapolate from a model, and the model decides the answer rather than the data.

Check overlap by comparing the estimated score distribution by group. A numeric summary is enough for a quick look.

Python Checking overlap on the propensity score

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the extreme rows of the table: bins at the two ends contain almost only one group. Units there have no real counterparts. Options when overlap is weak:

  1. Trim units outside the common support or outside [0.05, 0.95] and say clearly that the effect applies to the trimmed population.
  2. Change the estimand, for example the ATT, which only needs controls for treated-like units.
  3. Redefine the population with a rule that makes sense (for example only eligible patients).
  4. Collect more data or accept that the question cannot be answered from this dataset.

Effective sample size

Weighting reduces the amount of information in your data. Kish's effective sample size summarizes this:

ESS = (sum of weights)^2 / (sum of squared weights)

With equal weights, ESS equals n. If a few huge weights dominate, ESS collapses; an ESS far below the raw count of a group is a warning sign.

Python
def ess(w):
    return w.sum() ** 2 / (w ** 2).sum()

# df.w holds the IPW weights, as in the first example above
print("ESS treated:", round(ess(df.w[df.t == 1]), 0))
print("ESS control:", round(ess(df.w[df.t == 0]), 0))

A diagnostic checklist

flowchart TD
    A["Fit score"] --> B["Overlap OK?"]
    B -->|"No"| C["Trim or change estimand"]
    C --> A
    B -->|"Yes"| D["Weight or match"]
    D --> E["SMD below 0.1?"]
    E -->|"No"| F["Refine model"]
    F --> A
    E -->|"Yes"| G["Estimate effect"]

Only estimate the effect once both checks pass.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is the standardized mean difference preferred over a t-test p-value for balance?

    p-values depend on n, so throwing away data in matching can make imbalance look fine. The SMD measures the size of the difference.
  2. 2. What does a lack of overlap mean?

    Without comparable units, estimates depend on extrapolation by the model.
  3. 3. A covariate's SMD is 0.4 after weighting. What is a sensible next step?

    A large leftover imbalance leaves confounding. Refine the model; do not drop the variable.
  4. 4. What happens to the effective sample size when a few weights are very large?

    Kish's formula shrinks when weights are very unequal, because a few units carry the estimate.

Summary

  • Check covariate balance with the SMD (target below 0.1), not p-values, and do it before looking at outcomes.
  • Overlap means both groups appear across the score range; without it, extrapolation drives the answer.
  • Fix problems by trimming, changing the estimand or the population, or refining the score model.
  • Effective sample size shows how much information remains after weighting.

Next: Instrumental Variables