Skip to content
Tutorials Center
My learning

Topics

Lesson 41 of 44 ยท Projects and Practice

Practice: Critiquing Study Designs

Practice classifying and critiquing five realistic study scenarios with a decision tree, from correlational surveys and quasi-experiments to a true experiment.

The best way to learn research methods is to critique designs, again and again, until the questions become automatic. This practice lesson gives you a decision tree, a critique template and five short scenarios. Try to answer each before reading the model answers.

A decision tree: if a variable was manipulated, ask whether people were randomly assigned (yes gives a true experiment, no gives a quasi-experiment); if no variable was manipulated, the study is correlational or observational.
Start with two questions: was something manipulated, and was assignment random? The answers tell you what kind of claim the design can support.

What you will learn

  • To classify a study quickly as experimental, quasi-experimental or correlational.
  • To name the confounds and validity problems in a short description.
  • To propose a better design for the same question.
  • To write a calm, specific critique.

Key terms

Quasi-experiment: a study that compares conditions without random assignment, for example existing groups.

Selection effect: groups differ at the start because of how people ended up in them.

Regression to the mean: extreme scores tend to move toward the average when measured again, with no treatment at all.

Practice (testing) effect: people improve on a test simply because they have taken it before.

Reverse causation: the supposed effect actually causes the supposed cause.

A critique template

For any scenario, answer these in order.

  1. Design. Was a variable manipulated? Was assignment random? (Use the tree.)
  2. Claim. What does the report or headline claim? Is it causal?
  3. Threats. Which of selection, confounds, expectancy, practice, regression, attrition or sampling apply?
  4. Statistics. Is the sample size justified? Are effect sizes and intervals reported? Were multiple outcomes tested?
  5. Improvement. What is the best feasible change?

The scenarios

Scenario 1: Breakfast. A survey of 800 pupils finds that those who eat breakfast have higher grades. A newspaper reports "Free breakfast will raise grades", and a school board considers the idea.

Scenario 2: Therapy choice. A clinic offers a new anxiety program. Patients choose either the program or the waitlist. After eight weeks, the program group shows a larger drop in anxiety scores.

Scenario 3: Noise in the lab. Forty undergraduates are randomly assigned to read a passage in quiet or with traffic noise. The noise group answers fewer questions (d = 0.5, p = .04). The hypothesis was stated in advance. There were 20 people per group.

Scenario 4: Social media and loneliness. An online survey of 500 adults finds that heavy social-media users report more loneliness. The authors conclude that social media makes people lonely.

Scenario 5: Brain training. Thirty adults play a brain-training app for a month. Their scores on a memory test improve from pretest to posttest. A news story says "Brain training makes you smarter." There was no comparison group.

Model answers

Cover these with your hand first.

Scenario 1. Correlational (no manipulation). The causal headline does not follow. Possible third variables: family income, parental involvement, sleep, school attendance. Reverse causation is less plausible here, but confounding is serious. A better design: randomly assign classrooms or schools to a breakfast program or a comparison, then measure grades (a cluster randomized trial). Note that the unit of assignment (class or school) matters for the analysis.

Scenario 2. Quasi-experiment (manipulated by the clinic, but self-selected). Selection effect: people who choose treatment may be more motivated or less severe, and they may expect to improve (expectancy). Waitlist controls also show how much people improve with time alone. Better: random assignment, an active control (a comparable attention condition), and blind assessors of outcomes.

Scenario 3. True experiment (manipulation plus random assignment), with a stated hypothesis, which is good for internal validity. Concerns: n = 20 per group has power of only about one in three for d = 0.5, so a significant result is likely to overestimate the true effect (winner's curse); p = .04 is only modestly surprising; a single lab and a single type of noise limit external validity. Better: preregistered power analysis (about 64 per group for d = 0.5), a manipulation check, and replication with other tasks and noises.

Scenario 4. Correlational. Causal language is unjustified. Reverse causation: lonely people may use social media more. Third variables: age, personality, life events. Self-report measures may be imperfect (construct validity), and a self-selected online sample limits generalization. Better: longitudinal designs that measure both over time, or experiments that reduce usage and measure loneliness, along with a reliable loneliness measure. Calibrated wording: "heavy use was associated with more loneliness."

Scenario 5. No control group, so not an experiment at all. The improvement could be due to practice effects on the memory test, regression to the mean (people may have joined when their scores were low), motivation, or maturation. It says nothing about whether the app is better than doing something else, or whether gains transfer to everyday memory. Better: randomize people to the app or an active control (such as a general video game), blind testers, and test for transfer.

Try it

The first demonstration is a classifier based on the decision tree. Give it two facts and it names the design and the strongest claim it supports.

Python Design classifier from the decision tree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second shows regression to the mean, one of the sneakiest threats. We simulate a stable ability measured twice with noise, pick the people who scored lowest the first time, and watch them "improve" at the retest with no intervention at all.

Python Improvement from nothing: regression to the mean

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common misconceptions

"Any study with a treatment and a before-after comparison is an experiment." Without a comparison group and random assignment, before-after change cannot be separated from practice, time and regression.

"A significant p-value in a small experiment settles the question." Underpowered studies are prone to inflated and unstable effects.

"Correlational studies are useless." They are valuable for description, prediction and generating hypotheses; they just cannot support causal claims alone.

Check your understanding

Quick check ยท 6 questions

of 6 answered You scored / 6 ยท perfect!

  1. 1. A school compares test scores of pupils who chose to join a chess club with those who did not and claims chess improves math. The key flaw is...

    Without random assignment, preexisting differences between groups explain the result.
  2. 2. Patients are selected for a study because their symptoms are currently at their worst. Later their symptoms are lower even with no treatment. This is...

    Extreme scores tend to move toward the average on remeasurement.
  3. 3. Which design can support a causal claim?

    Manipulation plus random assignment is the core of a causal design.
  4. 4. Which improvement most directly addresses Scenario 5 (brain training)?

    A randomized comparison group separates the app's effect from practice and time.
  5. 5. A heavy-use group reports more loneliness. Reverse causation means...

    The direction of the arrow cannot be determined from a single correlation.
  6. 6. With 20 per group, what is the main statistical concern in Scenario 3?

    Small samples have low power and exaggerate significant effects.

Summary

  • Ask two questions first: was a variable manipulated, and was assignment random?
  • Name the threats: selection, confounds, expectancy, practice, regression to the mean, attrition, small or narrow samples.
  • Calibrate claims to the design: causal only with manipulation and random assignment.
  • Propose a concrete improvement for every flawed study.

Next: Glossary of research methods