Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 44 ยท Writing and Evaluating Research

Evaluating Studies Critically

Learn to judge a psychology study with the four validities, spot overclaiming and sampling problems, and read news about research without being fooled.

"Scientists discover that chocolate makes you smarter." Headlines like this appear every week, and most of them describe a study that does not support the claim. Learning to read research critically is not about cynicism. It is a skill: asking a small set of good questions in a fixed order, so that you can tell a solid result from a shaky one, and a modest finding from the inflated version that reaches the news. This lesson gives you those questions.

Four cards for construct, internal, external and statistical validity, each with the question it asks and a short example.
Four questions to ask about any study: Did it measure the right thing, did the manipulation cause the change, will it generalize, and can the numbers be trusted?

What you will learn

  • The four kinds of validity and the question each asks.
  • Why samples can limit generalization (WEIRD samples and convenience samples).
  • How to spot overclaiming, especially causal language from correlational data.
  • A step-by-step reading routine for a research article.
  • How to assess news reports of research.

Key terms

Construct validity: how well the measures and manipulations capture the concepts they are meant to represent.

Internal validity: the degree to which a study supports the conclusion that the IV caused changes in the DV.

External validity: the degree to which findings generalize to other people, settings and times.

Statistical conclusion validity: whether the statistical analysis and its conclusions are appropriate and trustworthy.

WEIRD: Western, Educated, Industrialized, Rich and Democratic, a label for the populations that dominate psychological samples.

Ecological validity: how closely a study's setting and tasks resemble real life.

The four validities

The framework below goes back to Cook and Campbell (1979) and Shadish, Cook and Campbell (2002). Each validity is a different way a study can succeed or fail.

Construct validity: did you measure what you meant to?

Psychology studies abstractions (stress, intelligence, prejudice, happiness) through concrete measures. The link between the two can be weak. If "intelligence" is measured by a ten-item quiz that mostly tests vocabulary, a claim about "intelligence" stretches too far. Ask: is the measure reliable (consistent)? Does it correlate with other measures of the same concept (convergent validity) and not with unrelated ones (discriminant validity)? Is the manipulation a good representation of the idea? A stress manipulation that was a boring task is not stress.

Internal validity: did the manipulation cause the change?

This is the question of confounds (Chapter 5). Strong internal validity needs random assignment, a proper comparison group, control of expectation effects and low attrition (participants who drop out unevenly across groups can bias results). For non-experimental designs, internal validity is weaker by design; the right claim is association, not cause.

External validity: will it hold elsewhere?

Does the finding hold for other people, places, tasks and times? Three questions help: Population (who was in the sample?), Setting (a lab, an online survey, the real world?), and Operationalization (would different measures of the same idea give the same answer?). External validity is established by replication in different contexts, not by a single study.

A well-known problem: Joseph Henrich, Steven Heine and Ara Norenzayan (2010) pointed out that most psychology research draws on WEIRD participants, who are an unusual slice of humanity, and that some findings that looked universal differ in other populations. Many classic studies used undergraduates in introductory courses, a convenient sample of mostly young, educated people. That does not make the results wrong, but it limits what we can say.

Statistical conclusion validity: can you trust the numbers?

Here you ask the questions of Chapter 6: Was the sample large enough for the claim (power)? Were the tests appropriate and assumptions checked? Were many tests run, so that some "significant" results are expected by chance? Is an effect size reported, and with an interval? Was anything reported selectively?

The trade-off

No study maximizes everything. A tightly controlled lab experiment gains internal validity but may lose ecological validity. A field study may look like real life but has more confounds. Good research is open about the trade, and good readers expect it.

A reading routine

You do not need to read an article from front to back like a novel. A useful order:

  1. Title and abstract. What is the claim?
  2. Method. Who took part, and how many? What was manipulated or measured? Was assignment random? This is where most problems hide.
  3. Results. Look at the figures and tables before the text. Check effect sizes and intervals, not just p-values.
  4. Discussion. Does the interpretation match the results, or has it grown? Are the limitations real or token?
  5. Introduction and references. Is the context fair? Is the evidence cited balanced?

Write down the claim in one sentence and the evidence in one sentence. If you cannot, the paper is probably unclear, or you need to re-read it.

Red flags

  • Causal language without a causal design. "Leads to", "boosts", "makes", "protects against" after a correlational study.
  • Tiny samples with large, surprising effects. Suspect the winner's curse and flexible analysis.
  • Many measures, few reported. Where are the other outcomes?
  • Only a p-value. No effect size, no interval.
  • A conclusion that goes beyond the sample. Results from 40 students generalized to "people".
  • Reverse causation or a third variable not discussed (a correlation between exercise and happiness may run either way, or both may follow from good health).
  • No limitations section, or only polite ones.
  • Conflicts of interest or a sponsor with a stake in the result.
  • A single study presented as settled. Look for replications and meta-analyses.

Reading research in the news

Press releases and news stories compress a paper into a sentence. Check: was this a human or animal study? A correlation or an experiment? How many participants? Is the headline's verb justified? Does the story say who funded the work? When possible, find the paper (or at least the abstract), and search for whether others have replicated the result. Keep in mind that a preprint has not yet been peer reviewed, and that peer review is a useful filter but not a guarantee of correctness.

Common misconceptions

"Peer reviewed means true." It means experts read it and found no fatal flaw. Reviewers often cannot see the raw data and can miss problems.

"A big sample guarantees validity." A large sample reduces random error. It does nothing for a biased sample, a poor measure or a confounded design.

"External validity requires a random sample of the whole world." It is built up by replication across populations and settings; most useful studies are not worldwide random samples.

"Criticism means the study is bad." Every study has limits. Good critical reading identifies what the evidence does and does not support.

Try it

First, a study scorecard. It rates a (made-up) study on a few yes/no questions drawn from the four validities and prints a cautious summary. The weights are illustrative; the point is the habit of checking systematically.

Python A study scorecard (a learning aid)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Second, a causal-language detector. It scans a sentence for verbs that imply causation and suggests more cautious wording for correlational studies.

JS Causal language detector

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A stress study uses a quiz about favorite colors as its measure of 'stress'. Which validity is weakest?

    The measure does not capture the concept it claims to represent.
  2. 2. A survey shows that people who exercise are happier. A headline says 'Exercise makes you happy'. What is the problem?

    Without manipulation and random assignment, reverse causation or third variables are possible.
  3. 3. WEIRD refers to...

    Henrich, Heine and Norenzayan (2010) used it to highlight the narrowness of typical samples.
  4. 4. Which is the best way to build external validity?

    Generalization is established through replication across contexts.
  5. 5. Which part of a paper deserves the most scrutiny first?

    The method shows who was studied and how, which determines how far the results can be trusted.

Summary

  • Four validities, construct, internal, external and statistical conclusion, structure critical reading.
  • Strong internal validity usually costs some ecological validity; good studies acknowledge the trade.
  • Read the method first, look at effect sizes and intervals, and compare the discussion with the results.
  • Be alert to causal language, tiny samples, selective reporting and unsupported generalization.
  • Treat news and single studies as tentative until replicated.

Next: Chapter review: writing and evaluating research