Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 50 ยท Methods and Cognitive Architecture

Experimental Design and Statistics

Learn independent and dependent variables, controls and confounds, within- and between-subject designs, sample size, p-values and replication in cognitive experiments.

An experiment is a question asked in a way that can be answered honestly. The idea is simple: change one thing on purpose, hold everything else steady, and see whether another thing changes. The difficulty is in the details. People differ, trials are noisy, stimuli differ in ways you did not plan, and a researcher who wants a result can, without meaning to, find one. Good design is the set of habits that keeps the answer honest.

Variable you set, variable you read, and controls, joined by arrows
The skeleton of an experiment: you set the independent variable, measure the dependent variable and control the rest.

What you will learn

  • The meaning of independent, dependent and control variables.
  • Within-subject and between-subject designs and when to use each.
  • Common threats: confounds, order effects and demand characteristics.
  • What a p-value, an effect size and a confidence interval say, and what they do not.
  • Why sample size, preregistration and replication matter.

Key terms

Independent variable (IV): the factor the researcher manipulates (for example, whether a word is common or rare).

Dependent variable (DV): the outcome that is measured (for example, reaction time).

Confound: a variable that changes along with the IV and could explain the result.

Random assignment: using chance to decide who gets which condition.

Counterbalancing: varying the order of conditions across participants.

Effect size: how big a difference is, in units that can be compared across studies.

A worked example

Question. Does a quiet background help people read faster than a noisy one?

  • IV: background (quiet vs. cafรฉ noise).
  • DV: time to read a passage and the score on a comprehension quiz (always measure accuracy too, because of the speed-accuracy trade-off).
  • Controls: the same passages in both conditions (or matched in length and difficulty), the same room and screen, the same instructions.
  • Possible confounds: if the noisy condition always happens second, tiredness is confounded with noise. If the noisy passages happen to be harder, passage difficulty is confounded with noise.

The fixes are the design tools below.

Between-subjects and within-subjects designs

Between-subjects Within-subjects
Who experiences what Each person gets one condition Each person gets all conditions
Strength No carry-over between conditions Each person is their own control, so far fewer participants are needed
Weakness Groups may differ by chance, so more people are needed Order effects: practice, fatigue, carry-over
Typical fix Random assignment, matching Counterbalancing, breaks between conditions

In cognitive psychology the within-subjects design is very common, because people differ in baseline speed far more than conditions differ. Comparing each person with themselves removes that person-level variation.

Counterbalancing

With two conditions, A and B, give half the participants the order AB and half BA. With more conditions, a Latin square makes every condition appear in every position equally often. Counterbalancing does not remove order effects, but it spreads them evenly so they do not masquerade as a condition effect.

Threats to a fair test

  • Confounds. Words that differ in frequency may also differ in length. The standard remedy is to match stimuli on every other variable you can measure.
  • Demand characteristics. Participants guess the purpose and adjust their behavior to please you. Use neutral instructions, filler trials and, if needed, a cover story that is explained afterward.
  • Experimenter expectations. A researcher who knows the hypothesis can influence results unintentionally. Automating procedures and blinding (the person interacting with participants does not know the condition) reduces this.
  • Selection and attrition. If particular kinds of people drop out of one condition, the groups stop being comparable.
  • Ceiling and floor effects. If a task is so easy everyone is near 100%, differences vanish, even if they exist.

Statistics: what they do and do not say

Because every measurement is noisy, we ask whether an observed difference is bigger than what chance alone would produce. In a typical significance test, such as a t-test, you compute a p-value: the probability of seeing a difference at least this large if there were no real effect. A small p-value (often below 0.05 by convention) means the data would be unusual under "no effect".

It does not mean:

  • that there is a 95% chance the hypothesis is true;
  • that the effect is large or important;
  • that the result will replicate.

Better practice reports three things together:

  1. The effect size (for example, a difference of 28 ms, or Cohen's d = 0.5). A tiny effect can be "significant" with a huge sample.
  2. A confidence interval showing the plausible range for the effect.
  3. The sample size and how it was chosen.

Sample size and power

Statistical power is the chance of detecting a real effect of a given size. Small studies have low power: they often miss real effects and, when they do find one, they tend to overestimate it. A power analysis before the study estimates how many participants are needed. In the 2010s the field learned that many published studies were underpowered, one source of findings that failed to replicate.

Preregistration and replication

Researchers have many small choices: which trials to exclude, which test to run, when to stop collecting data. If they are made after seeing results, it is easy to find something "significant" by accident. Preregistration means writing down the hypothesis, design and analysis plan before collecting data. Replication, repeating a study with new participants, shows whether a result is dependable. Large collaborative replication projects have shown that many well-established effects in cognitive psychology (such as the Stroop effect) replicate very well, while others from social and cognitive priming research have not, which is a reason to treat any single study with care.

Ethics

Research with people follows widely accepted principles, set out in documents such as the Belmont Report and the Declaration of Helsinki. In practice:

  • Informed consent: participants know what will happen and can say no.
  • Right to withdraw at any time without penalty.
  • Confidentiality of data.
  • Minimizing harm and debriefing after any deception.
  • Review by an independent ethics committee (called an IRB in the US) before the study begins.

Common misconceptions

"A significant result proves the hypothesis." It shows the data are unlikely under one specific alternative, "no effect". It does not prove your explanation.

"A non-significant result means there is no effect." It may mean the study was too small to detect one.

"Correlation implies causation." Only a design with a manipulated IV and controlled confounds supports a causal claim. Observational patterns, such as "people who sleep less have worse memory", could be caused by a third variable.

Try it

Python How sample size changes what you can trust (simulation)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

JS A Latin square for counterbalancing

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. The independent variable is

    The IV is the factor the researcher changes; the DV is what is measured.
  2. 2. A key reason many cognitive experiments use within-subjects designs is that

    Baseline differences between people are large, so comparing each person with themselves is efficient.
  3. 3. A p-value of 0.03 means

    A p-value is a statement about the data under the assumption of no effect, not about the hypothesis's probability.
  4. 4. Why is a small study likely to overestimate a real effect when it finds one?

    Under low power, the studies that pass the significance threshold tend to be those with exaggerated estimates.
  5. 5. Preregistration helps because it

    Writing the plan in advance limits choices made after seeing the results.

Summary

  • An experiment changes an independent variable, measures a dependent variable and controls everything else.
  • Within-subjects designs are efficient but need counterbalancing; between-subjects designs avoid carry-over but need more people.
  • Watch for confounds, demand characteristics, expectations, attrition and floor or ceiling effects.
  • Report effect sizes and confidence intervals with p-values; plan sample size with a power analysis.
  • Preregistration, replication and ethical practice protect the quality of research.

Next: Cognitive Neuroscience Methods