Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 47 ยท Getting Started

Methods and evidence

How behavioral economists gather and judge evidence: lab and field experiments, natural data, randomization, real incentives, effect sizes and replication.

Behavioral economics makes claims about what people do, so the quality of its evidence matters more than the elegance of any theory. A famous effect that appears in one classroom and vanishes elsewhere is a story, not a fact. This lesson shows how researchers collect evidence, what each method can and cannot tell you, and how to read a claim critically.

You do not need statistics software. You need a few ideas: randomization, validity, effect size and replication. With them you can read the rest of this course like a scientist rather than a spectator.

Three methods compared: lab experiment, field experiment and natural data
Caption: each method trades control against realism, so strong claims use more than one.

What you will learn

  • The difference between lab experiments, field experiments and natural data.
  • Why randomization lets us talk about cause and effect.
  • Why economists pay real incentives and rarely deceive participants.
  • How to think about effect size, publication bias and replication.

Key terms

Randomized controlled trial (RCT): participants are assigned by chance to a treatment or control group, so groups are alike on average.

Internal validity: confidence that the treatment caused the difference you see.

External validity: confidence that the result applies outside the study setting.

Effect size: how big a difference is, not just whether it exists.

Replication: repeating a study to check whether the result holds.

Why experiments?

Suppose a company sends reminder letters to some customers and then sees that those customers pay faster. Did the letters cause faster payment, or did the company send letters to customers it already expected to pay? Without a fair comparison you cannot tell.

Randomization solves this. If a coin flip decides who gets the letter, the two groups are the same on average in everything, including things you never measured (age, income, mood, conscientiousness). Any later difference beyond chance can then be put down to the letter. This is the logic of an RCT, and it is why experiments are so central to the field.

Three families of method

Lab experiments

Participants, often students or paid volunteers, make choices in a controlled setting: choose between lotteries, split money, decide how much to contribute to a group fund. The researcher sets the rules and changes one thing at a time. Strengths: tight control and high internal validity, cheap repetition. Weaknesses: artificial situations, small and unrepresentative samples, and uncertainty about whether behavior for a $10 stake predicts behavior with a mortgage.

A widely discussed sampling concern is that psychology and economics studies draw heavily from people in WEIRD societies (Western, Educated, Industrialized, Rich, Democratic), as Joseph Henrich, Steven Heine and Ara Norenzayan pointed out in 2010. Results from such groups may not be universal.

Field experiments

Researchers randomize in the real world: change the wording of a tax letter, the default on a form, the layout of a cafeteria. People often do not know they are in a study, so behavior is natural. Strengths: realism and relevance for policy. Weaknesses: less control, higher cost, ethics to manage and results tied to context.

Natural data and quasi-experiments

Sometimes the world runs the experiment. A rule changes in one region but not a neighboring one; a lottery assigns draft numbers; a company changes a default at a known date. Comparing before and after, or across groups, can reveal causal effects, but only if the groups would have followed the same trend without the change. Strengths: large, real datasets. Weaknesses: researchers cannot control assignment, so hidden confounds are a constant worry.

Rule of thumb. Ask two questions of every study: Is the comparison fair? (internal validity) and Would this hold in my situation? (external validity).

Real incentives and honesty

Experimental economists typically pay participants according to their choices, so decisions have real consequences, and they avoid deceiving participants, because deception can make people suspicious in later studies. Psychologists often use hypothetical questions and sometimes deception, which is one source of differences between the two fields. Incentives matter most when a task requires effort and care; for some judgments (for example the Linda problem in Chapter 3) the main findings appear with or without payment.

Reading effect sizes and p-values

A p-value tells you how surprising your data would be if there were really no effect. It does not tell you how large or important the effect is. A huge study can find a tiny, unimportant difference with a tiny p-value. A small study can miss a real effect.

Two habits help.

  1. Look for the effect size and its uncertainty. "Raised participation by 5 percentage points (95% confidence interval 3 to 7)" is far more informative than "p < 0.05."
  2. Beware of the winner's curse. If many small studies chase a modest true effect, the ones that cross the significance threshold are the ones that happened to overshoot. Published estimates from small studies tend to be too large.

Replication and the nudge debate

The replication crisis in psychology (see Lesson 1.3) led to large coordinated projects that re-ran studies. Several effects held; others shrank or failed. Behavioral economics has its own examples. A frequently cited study comparing academic nudge papers with large trials run by government "nudge units" (DellaVigna and Linos, 2022) found that average effects were much smaller in the unit trials, roughly 1.4 percentage points versus about 8.7 in the journal papers. Part of the gap is publication bias, and part reflects trials at scale. The practical lesson is to treat headline numbers from small, early studies with caution and to ask whether results appear in large, preregistered trials.

Common misconceptions

  • "A significant result proves an effect is real and large." Significance depends on sample size and is not a measure of importance.
  • "Lab results never apply to the real world." Many do, and some do not. The way to find out is to test them in the field.
  • "If an effect failed to replicate once, it is false." A failure is evidence, not proof. Conditions, samples and statistical power all matter. Look at the full pattern of studies.

Try it

The first demonstration shows why randomization works. We randomly assign 40 people with random ages to two groups and compare the average age. The groups differ a little by chance, and the difference shrinks as the sample grows.

JS Randomization balances groups (a simulation)

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The second demonstration simulates the winner's curse. The true effect is small (0.2 standard deviations). We run many small studies and compare all estimates with only the "significant" ones.

Python The winner's curse (a simulation)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does randomization help establish cause and effect?

    Chance assignment balances both observed and unobserved characteristics on average, so later differences can be put down to the treatment.
  2. 2. Which is a typical weakness of a lab experiment?

    Labs offer control and internal validity, but external validity is a concern.
  3. 3. What does a p-value tell you?

    A p-value speaks to surprise under the no-effect assumption. It says nothing about effect size or importance.
  4. 4. What is the winner's curse in published studies?

    When power is low, the studies that happen to reach significance are the ones that overshoot, so published effects are inflated.
  5. 5. Why do experimental economists usually avoid deceiving participants?

    Trust in the instructions is a shared resource for experimenters, so economists protect it.

Summary

  • Randomization makes groups comparable and supports causal claims.
  • Lab experiments favor control, field experiments favor realism, natural data offers scale with less control.
  • Economists pay real incentives and avoid deception; psychologists sometimes do not.
  • Judge claims by effect size, uncertainty and replication, not by significance alone.
  • Small, early studies tend to overstate effects (the winner's curse).

Next: Chapter 1 review