Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 44 ยท Ethics and Replication

The Replication Crisis and Open Science

Understand why many psychology findings failed to replicate, how p-hacking, publication bias and low power contributed, and how preregistration and open data help.

Science advances by checking. If a finding is real, other labs should be able to find it again, and it should not matter whose hands run the study. Around 2011 psychology began to ask, uncomfortably, how often that was true. The answer turned out to be "less often than we hoped", and the reasons are an excellent tour of everything that can go wrong between a good idea and a trustworthy result. The remedies, now widely adopted, are among the most practical skills in this course.

Two bar charts: about 97 percent of original studies but only about 36 percent of replications were significant, and the average effect size fell from about r 0.40 to about r 0.20.
The Open Science Collaboration (2015) repeated 100 published studies; fewer replications were significant and their effects were about half as large.

What you will learn

  • The events that triggered the replication debate.
  • What the Open Science Collaboration found, and what it does not show.
  • How p-hacking, HARKing, publication bias and low power distort literatures.
  • What preregistration, registered reports, open data and multi-lab replications are.
  • How to judge how much to trust a single study.

Key terms

Replication: repeating a study to see whether the finding appears again; a direct replication follows the original procedure closely, a conceptual one tests the same idea differently.

p-hacking: trying different analyses, exclusions or outcomes until p falls below .05, then reporting only that analysis.

HARKing: Hypothesizing After the Results are Known: presenting a post-hoc hypothesis as if it had been planned.

Publication bias (file-drawer problem): the tendency for significant results to be published and null results to stay unpublished.

Preregistration: a time-stamped public record of hypotheses, design and analysis plan made before data collection.

Registered report: a journal format in which the plan is peer-reviewed and accepted before results exist.

How the debate started

A cluster of events around 2011 made the problem impossible to ignore.

  • Daryl Bem's "Feeling the Future." A respected social psychologist published nine experiments in a top journal suggesting that people could anticipate random future events. Almost nobody believed it, yet the methods were standard. If standard methods could yield evidence for precognition, something was wrong with the standard methods.
  • Fraud. Diederik Stapel, a prominent Dutch social psychologist, was found to have fabricated data in dozens of papers. Fraud is rare, but the case showed how easily trust could be exploited.
  • "False-positive psychology." Joseph Simmons, Leif Nelson and Uri Simonsohn (2011) showed by simulation and by a deliberately absurd demonstration that a few ordinary, undisclosed analysis choices could make almost anything "significant". With several such choices combined, the false-positive rate rose far above the nominal 5%.
  • A survey of questionable practices. Leslie John, George Loewenstein and Drazen Prelec (2012) found that substantial shares of surveyed psychologists admitted to practices such as not reporting all measures or deciding whether to collect more data after peeking at the results.

The big replication projects

In 2015 the Open Science Collaboration, a group of 270 researchers, attempted to replicate 100 studies from three leading psychology journals, each as faithfully as possible and with high power. About 97% of the original studies had reported significant results; about 36% of the replications did. The average effect size in the replications was roughly half that of the originals. Other projects (the Many Labs series, in which dozens of labs run the same protocol) found a more varied picture: many classic effects replicated well, while some, notably certain social-priming and other eye-catching findings, did not. A large multi-lab project on ego depletion and a larger replication of a "power posing" study failed to find the effects originally reported.

Careful interpretation matters. A failed replication does not automatically mean the original was false; the replication could have differed in population or procedure, or could itself be a chance result. And replication rates vary by subfield. But the pattern, smaller effects and fewer significant results on retest, was clear enough to change practice.

What went wrong?

No single villain. A set of understandable incentives and habits combined.

  1. Low power. Small studies find only large observed effects, so the significant ones are inflated (the winner's curse).
  2. Flexible analysis (p-hacking). With many outcomes, covariates, exclusions and test choices, a "significant" path almost always exists. Gelman and Loken called this the garden of forking paths: even honest researchers, choosing analysis after seeing data, make choices that follow the data.
  3. HARKing. A surprising result gets a neat story written afterward, which readers then take as a confirmed prediction.
  4. Publication bias. Journals preferred novel, significant findings, so the literature contained the lucky winners and few of the nulls (Rosenthal named this the "file-drawer problem" in 1979).
  5. Careers and incentives. Publish-or-perish pressure rewarded quantity and surprising claims.

The fixes

Researchers responded with a set of practices designed so that you can no longer fool yourself, or readers, by accident.

  • Preregistration. Before data collection, record your hypotheses, sample size (with the power analysis), exclusion rules and primary analysis on a platform such as the Open Science Framework or AsPredicted. Anything you do beyond the plan is labelled exploratory, which is perfectly legitimate when disclosed.
  • Registered reports. A journal peer-reviews the question and method first and promises to publish regardless of the result. This removes the incentive to chase significance.
  • Open materials, data and code. Sharing stimuli, anonymized data and analysis scripts lets others check and reuse the work. Many journals award open-practice badges and follow the TOP (Transparency and Openness Promotion) guidelines.
  • Bigger samples and power analysis. The previous chapter's lessons, put to use.
  • Multi-lab collaborations such as Many Labs and the Psychological Science Accelerator, which pool resources to test an effect across sites and countries.
  • Reporting effect sizes and intervals, not just p. Chapter 6.
  • Robustness checks and multiverse analysis. Show whether the result survives reasonable alternative analyses.

Not every change is settled. Preregistration can be done well or poorly, and critics worry about the rigidity of some rules and about extra work. Still, evidence suggests that registered reports report many more null results than the traditional literature, which is just what a less-biased record should show.

How to read a single study

You now have a quick set of questions: Was the sample size justified? Was the plan preregistered? Are all outcomes reported? Is the effect large enough to matter, and is its interval narrow? Has it been replicated in another lab? A single significant result from a small, unregistered study deserves interest, not belief.

Common misconceptions

"The replication crisis means psychology is not a science." It means psychology found, by using the scientific method on itself, that some of its habits were unreliable, and it has been repairing them. Other fields, including medicine and economics, report similar issues.

"A failed replication proves the original was wrong." Both studies are estimates with error. One failure is a reason to look harder, not a verdict.

"p-hacking is fraud." It is usually an honest, unconscious habit, but it can still produce false findings. Fraud is deliberate fabrication.

"Preregistration forbids exploratory analysis." No: exploration is welcome as long as it is labelled as such.

Try it

The first demonstration is a simulation of publication bias. We run 2,000 made-up studies of a small true effect (d = 0.2) with different sample sizes, then "publish" only the significant ones. The funnel plot shows all studies (light) and the published ones (bold). Watch how the published average drifts above the truth.

Python Publication bias and the funnel plot

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second demonstration is the simplest open-science habit: a codebook. It builds a small data dictionary and a fingerprint (hash) of the dataset, so anyone who receives your file can confirm they have the same one.

Python A codebook and a data fingerprint

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 6 questions

of 6 answered You scored / 6 ยท perfect!

  1. 1. What does preregistration do?

    A time-stamped plan prevents hypotheses and analyses from being quietly adjusted to fit the data.
  2. 2. HARKing means...

    It presents a post-hoc explanation as if it were a prediction made in advance.
  3. 3. In the Open Science Collaboration (2015), about what share of replications were significant?

    Roughly 36% of the replications reached significance, compared with about 97% of the originals.
  4. 4. Why does publication bias inflate effect estimates in the literature?

    If mostly the winners are published, the average published effect exceeds the true effect.
  5. 5. A registered report is reviewed...

    That format removes the incentive to chase significant results.
  6. 6. Which interpretation of one failed replication is most reasonable?

    One failure is information, not a verdict. Pooling evidence across studies is better.

Summary

  • Fraud, Bem's precognition studies, flexible-analysis demonstrations and large replication projects exposed reliability problems in psychology.
  • Low power, p-hacking, HARKing and publication bias together inflate the literature.
  • The Open Science Collaboration found far fewer significant replications and roughly halved effect sizes.
  • Remedies include preregistration, registered reports, open data and code, larger samples and multi-lab replications.
  • Read single studies with healthy skepticism: sample size, preregistration, completeness and replication matter.

Next: Chapter review: ethics and replication