Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 44 ยท The Logic of Research

Validity and Confounds

Learn internal, external, construct and statistical conclusion validity, and the common threats to them: confounds, regression to the mean, selection, attrition and demand effects.

Imagine a program that gives low-scoring students extra tutoring. Their scores rise at the next test. Did the tutoring work? Maybe. Or maybe low scores on one day tend to be followed by better scores on the next, whatever happens in between. Without asking the right questions, you cannot tell.

Validity is the word researchers use for "how much should I trust the conclusion?" There is more than one kind, because a study can fail in several ways. This lesson explains the four main types and the usual threats to each.

Four panels showing internal, external, construct and statistical conclusion validity, each with the question it asks.
Four questions every study must face: did the cause really work here, does it apply elsewhere, did we measure the idea, and were the numbers sound?

What you will learn

  • The four types of validity in the tradition of Campbell, Cook and Shadish.
  • The major threats to internal validity, including confounds and regression to the mean.
  • Common threats to external and construct validity.
  • How researchers reduce these threats.

Key terms

Internal validity: the degree to which a study shows that the IV caused the change in the DV.

External validity: the degree to which results generalize to other people, settings and times.

Construct validity: whether the measures and manipulations really reflect the intended concepts.

Statistical conclusion validity: whether the statistical analysis and its conclusions are sound.

Confound: a variable that changes along with the IV and offers a rival explanation.

The four kinds of validity

Donald Campbell and Julian Stanley introduced internal and external validity in 1963, and later work by Thomas Cook, William Shadish and colleagues added construct and statistical conclusion validity. Together they ask four questions.

  1. Internal: within this study, did X really cause Y?
  2. External: will the result hold for other people, places and times?
  3. Construct: did we manipulate and measure the ideas we say we did?
  4. Statistical conclusion: did we analyze the data properly, and is the pattern bigger than chance?

A study can be strong on one and weak on another. A tightly controlled laboratory experiment may be high in internal validity and low in external validity. A large survey in the real world may reverse that.

Threats to internal validity

Most of these are confounds, which are explanations other than the IV.

Threat What happens Example
History An outside event changes outcomes during the study A school-wide anti-bullying campaign starts mid-study
Maturation Participants naturally change with time Children get better at reading anyway
Testing Taking a test once changes performance on the next Practice effects
Instrumentation The measure changes over time Raters become stricter
Regression to the mean Extreme scores drift toward average on retest Lowest scorers improve without any help
Selection Groups differ from the start Volunteers are more motivated than non-volunteers
Attrition Participants drop out unevenly Those finding the program too hard quit

Regression to the mean

If you pick the people with the lowest scores on a test that is partly luck, many were unlucky that day. On retest their luck averages out and their scores rise, even if you did nothing. This was described by Francis Galton in the 1880s. It is one of the most common reasons that "we helped the worst-off, and they improved" is not good evidence. A comparison group chosen in the same way is the cure.

Selection and attrition

If participants choose their own condition, differences between conditions may reflect who chose what. If people drop out more from one group, the remaining members are no longer comparable. Random assignment protects against selection, and tracking and reporting dropouts protects against attrition.

Threats to external validity

  • Sample: Joseph Henrich, Steven Heine and Ara Norenzayan pointed out in 2010 that much behavioral research uses people from Western, educated, industrialized, rich and democratic societies (nicknamed WEIRD), and that these people can differ from others on many measures. Results from them may not generalize.
  • Setting: laboratory tasks may not resemble daily life.
  • Time: findings from one era may change.
  • Reactivity: people behave differently when they know they are observed. The Hawthorne studies of the 1920s and 1930s are often cited for this, though later reanalyses suggest the original data support less than the legend claims.

Threats to construct validity

  • Demand characteristics: participants guess the purpose and try to help or hinder.
  • Experimenter expectancy: researchers who know the hypothesis may unintentionally influence participants. Robert Rosenthal's research on this effect in the 1960s led to the use of blind procedures.
  • Poor operational definitions: the measure captures only part of the idea, or a different idea.

Threats to statistical conclusion validity

Too small a sample to detect a real effect, using a test whose assumptions are violated, running many tests and reporting only the significant ones, or measuring unreliably. Chapter 6 deals with these.

Reducing threats

Threat Typical remedy
Selection Random assignment
Expectancy and demand Blinding, standardized scripts, deception where ethical
Regression, history, maturation A comparison group studied at the same time
Attrition Track and compare dropouts
Poor generalization Diverse samples, replication, field studies

Common misconceptions

"High internal validity means the result is true everywhere." It means the cause-and-effect conclusion is sound within the study. Generalization is a separate question.

"A bigger sample fixes everything." A large sample reduces random error but not confounds or biased sampling.

"A pretest-posttest improvement shows the program worked." Without a comparison group, history, maturation and regression could explain the change.

Try it

Python Regression to the mean (simulation)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Attrition can fake a success (simulation)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A lab experiment is tightly controlled but uses an unusual task. It probably has

    Control supports causal conclusions within the lab, but an artificial task may not generalize to daily life.
  2. 2. Students with the lowest scores improve at the next test even without help. This is best explained by

    Extreme scores contain luck, which averages out on retest, moving them toward the mean.
  3. 3. Random assignment mainly protects against

    It makes groups similar at the start so pre-existing differences do not explain the result.
  4. 4. Participants guess the purpose of a study and change their behavior to match. This is a threat to

    Demand characteristics change what the measure reflects, so it no longer captures the intended construct.
  5. 5. WEIRD samples mainly concern

    If most participants come from a narrow slice of the world, results may not generalize.

Summary

  • Four validities ask: did X cause Y here, does it generalize, did we measure the idea, and were the statistics sound?
  • Internal threats include history, maturation, testing, instrumentation, regression, selection and attrition.
  • External threats include narrow samples, artificial settings and reactivity.
  • Random assignment, comparison groups, blinding, tracking dropouts and replication reduce the threats.

Next: Chapter 2 Review: The Logic of Research