Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 39 ยท Projects and Practice

Project: Analysing an Experiment with ANOVA and Regression in SPSS

A second SPSS project: a three-group learning experiment with a pre-test and post-test. Check data, run paired tests, ANOVA with post hoc, ANCOVA-style regression and write it up.

This project walks through a small randomized experiment, the kind psychology, education and marketing teams run daily. Three teaching methods are compared, with a pre-test and a post-test for every student. You will combine several lessons: data checking, a paired t test, a one-way ANOVA with post hoc tests, and a multiple regression that adjusts for the pre-test.

The design

Thirty students were randomly assigned to one of three methods (method: 1 = Lecture, 2 = Practice, 3 = Online), ten per group. Each took a test before (pre) and after (post) the course.

Questions:

  1. Did scores rise overall from pre to post?
  2. Does the gain (post minus pre) differ by method?
  3. Does the method still matter after adjusting for the starting level?

Random assignment is what allows a causal reading of group differences, if the experiment was run properly. Without random assignment the same statistics only show association.

Python Stand-in: the experiment data and group summaries

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 1: set up and check

Variable View: define method with value labels, pre, post as Scale. Create the gain score:

Text
COMPUTE gain = post - pre.
EXECUTE.
FREQUENCIES VARIABLES=method.
EXAMINE VARIABLES=pre post gain BY method
  /PLOT=BOXPLOT NPPLOT /STATISTICS=DESCRIPTIVES.

Check for outliers, equal group sizes (10 each), and that pre-test means are similar across groups, which random assignment should produce.

Step 2: did scores rise overall?

All students, paired data:

Text
T-TEST PAIRS=post WITH pre (PAIRED).

Step 3: does the gain differ by method?

A one-way ANOVA on the gain score, with homogeneity and Welch options and a post hoc test:

Text
ONEWAY gain BY method
  /STATISTICS DESCRIPTIVES HOMOGENEITY WELCH
  /POSTHOC=TUKEY GH ALPHA(0.05).
Python Stand-in: paired test, ANOVA on gains and Tukey

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the Multiple Comparisons table and the Homogeneous Subsets table in SPSS the same way: significant pairs have Sig. below .05, and groups sharing a column in the subsets table do not differ.

Step 4: adjust for the pre-test

Gain scores can be criticised (they depend on the starting level). An alternative is to predict the post-test from the pre-test and the method, so the method effect is estimated at equal starting levels. This is ANCOVA, available as Analyze > General Linear Model > Univariate with post as Dependent, method as Fixed Factor and pre as Covariate. You can get the same adjusted comparison from multiple regression with dummy variables:

Text
RECODE method (2=1) (ELSE=0) INTO practice.
RECODE method (3=1) (ELSE=0) INTO online.
EXECUTE.
REGRESSION
  /STATISTICS COEFF OUTS CI(95) R ANOVA
  /DEPENDENT post
  /METHOD=ENTER pre practice online.

Lecture is the reference group; the B for practice is the adjusted difference between Practice and Lecture.

Python Stand-in: regression with dummy variables

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 5: assumptions and diagnostics

  • ANOVA: equal variances (Levene), roughly normal gains, independent students.
  • ANCOVA / regression: the slope of post on pre should be similar in each method (no method-by-pre interaction); residual plots should look random.
  • Ethics and design: report randomization, any dropouts and whether the testers knew the groups.

Step 6: report

Describe the design, give means and SDs per group for pre, post and gain, then the tests: "Scores rose overall (paired t test), and gains differed by method (one-way ANOVA with Tukey post hoc tests), the Practice group gaining most." Give statistics and effect sizes copied from your SPSS tables, and note that with 10 students per group the confidence intervals are wide.

Common mistakes

  • Running the ANOVA on post-test scores only and ignoring the pre-test.
  • Treating the experiment's three groups with three t tests.
  • Reporting means without the post hoc results.
  • Claiming the experiment proves the cause without describing the randomization.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which test checks whether the average pre-to-post change differs between three methods?

    The gain is a scale outcome and method has three independent groups.
  2. 2. Why include the pre-test as a covariate?

    Adjusting for the baseline makes group comparisons fairer and often more precise.
  3. 3. In the dummy-coded regression, Lecture is the reference group. What does B for 'practice' mean?

    Each dummy coefficient compares its category with the reference, holding other predictors constant.
  4. 4. What allows a causal interpretation of group differences?

    Randomization balances other factors on average; the statistics alone cannot do that.

Summary

  • Check data and group balance, then test overall change with a paired t test.
  • Compare gains across three methods with one-way ANOVA and a post hoc test.
  • Adjust for the baseline with ANCOVA or regression using dummy variables.
  • Report design, descriptives, tests, effect sizes and limitations.

Next: Practice Exercises and Test Chooser Drill