Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 37 ยท Projects and Practice

Project: A Policy Change with Difference-in-Differences

Evaluate a regional policy with a panel: build the data, run two-way fixed effects with clustered errors, draw an event study, add placebo and permutation checks, and write the conclusion.

A government introduced a free-transit pass in 12 of 40 regions in year 2020. The outcome is the share of residents commuting by public transport, observed every year from 2014 to 2024. The treated regions were not chosen at random, since larger and denser regions were more likely to adopt the pass. A straight comparison of treated and control regions after the policy would therefore be unfair. Your job: estimate the policy's effect using difference-in-differences and defend the result.

The plan

  1. Build and inspect the panel.
  2. Estimate a two-way fixed-effects DiD with clustered standard errors.
  3. Draw an event study to check pre-trends and see the dynamics.
  4. Run falsification checks: a placebo date and a permutation test.
  5. Write the conclusion with limits.

Step 1 and 2: data and the DiD estimate

Because all treated regions start in the same year and we have a balanced panel, two-way fixed effects can be computed by double demeaning: subtract region means and year means (and add back the grand mean). The coefficient on the treatment indicator after double demeaning equals the TWFE estimate.

Python Build the panel and estimate the effect

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The true effect is 3 in 2020 and grows by 10% per year, so its average over 2020 to 2024 is 3.6. The estimate (a bit above 4) overshoots because only about a dozen regions were treated, so chance differences between those regions matter. Look at the interval: it is narrow and does not quite cover 3.6. With so few treated clusters, cluster-robust standard errors tend to be too small, which is why the permutation test below is the more trustworthy check. Note also that treated regions have higher levels before the policy (they are larger), which DiD handles with region fixed effects.

Step 3: event study

Estimate the gap for every year relative to 2019. Leads should hover near zero; lags show the build-up.

Python Event-study coefficients by year

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Before 2019 the estimates are small and within about one to one and a half standard errors of zero; from 2020 they jump to 3.5 and above. That supports parallel trends and shows the effect developing over time. If the leads had trended instead, you would have to question the whole design (for instance, by matching regions on size, or adding region-specific trends).

Step 4: falsification checks

Placebo date. Use only pre-policy data and pretend the pass started in 2017. The "effect" should be about zero.

Permutation test. Randomly reassign the treated label among regions (keeping the number of treated regions) many times, recompute the DiD, and see how extreme the real estimate is among these fake ones. This does not rely on normal approximations and is useful with few clusters.

Python Placebo date and permutation test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The placebo estimate is close to zero, and the real estimate (about 4) lies far outside the range of roughly -1.3 to 1.3 generated by randomly shuffled treatment labels. Both support a causal reading. The permutation range also shows how much a "fake" effect can wobble with 40 regions.

Step 5: the write-up

A short conclusion could read:

Regions with the free pass saw their public transport share rise by roughly 4 percentage points relative to the trend in other regions (the cluster-robust interval is about 3.8 to 4.5, probably optimistic with only 14 treated regions). Effects appear in the first year and keep growing. Pre-policy trends were parallel, a placebo date shows no effect, and the result is far outside a permutation distribution. Limits: adoption was not random, so a regional shock that coincided with the policy would bias the estimate; the effect applies to regions like those that adopted.

flowchart TD
    A["Panel data"] --> B["TWFE DiD with clustered SE"]
    A --> C["Event study"]
    B --> D["Placebo date"]
    B --> E["Permutation test"]
    C --> F["Parallel trends evidence"]
    D --> G["Conclusion with limits"]
    E --> G
    F --> G

Estimate, then try hard to break your own result.

Extensions

  • Make adoption staggered (each region starts in a different year) and see how plain TWFE behaves compared with a "clean controls only" comparison.
  • Add region-specific linear trends and check how much the estimate moves.
  • Use a synthetic control for a single treated region.
  • Bring in covariates (population growth) and compare.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Treated regions had higher transport shares before the policy. Why is that not a problem for DiD?

    DiD compares changes over time, so fixed gaps in levels cancel.
  2. 2. Why cluster the standard errors by region?

    Treatment varies at the region level, and repeated observations of a region are not independent.
  3. 3. What does a permutation test do in this project?

    It shows how unusual the real estimate is under no-effect, without normal approximations.
  4. 4. Which finding would most threaten the design?

    Pre-existing different trends violate parallel trends.

Summary

  • Build the panel, estimate TWFE DiD with region-clustered errors and read the effect size with an interval.
  • Draw an event study to check pre-trends and see the dynamics.
  • Add placebo dates and permutation tests to try to break the result.
  • State the assumptions and limits plainly in the write-up.

Next: Practice Problems