Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 39 ยท Hypothesis Tests

Hypothesis Testing and p Values in SPSS

Understand null and alternative hypotheses, the Sig. column, alpha, confidence intervals and Type I and II errors so you can read any SPSS test table correctly.

Almost every SPSS test table ends in one column named Sig. (in newer versions, Significance with One-Sided p and Two-Sided p). Before you run a single t test you need to know what that number means, and what it does not mean. Most misreadings of SPSS output come from here.

This lesson explains the logic shared by all the tests in this chapter. The later lessons then only have to say which menu to open and which rows of output to read.

The two hypotheses

A hypothesis test starts with a pair of competing statements about a population.

  • The null hypothesis (H0) says nothing is going on: no difference between groups, no relationship, no effect.
  • The alternative hypothesis (H1) says something is going on: the means differ, the variables are related.

You never prove H0 true. You ask: if H0 were true, how surprising would my data be? The answer to that question is the p value.

What the p value is

The p value is the probability of getting a result at least as extreme as yours if the null hypothesis were true. A small p means your data would be rare in a world where nothing is going on, so you doubt that world.

By convention you compare p with a pre-chosen alpha, usually 0.05:

  • p < alpha: reject H0 (the result is "statistically significant").
  • p >= alpha: fail to reject H0 (you have not shown an effect; that is not the same as showing there is none).

The p value is not the probability that H0 is true, and not the probability your result happened "by chance". It is a statement about the data given H0. A tiny p can also come from a tiny, useless effect in a huge sample. Always report an effect size and a confidence interval next to it.

Reading Sig. in SPSS

SPSS prints a decimal such as .032. Rules for reading it:

  1. If it shows .000, it is not zero. Report it as p < .001.
  2. Sig. (2-tailed) tests "different in either direction". 1-tailed tests only one direction and must be justified before you look at data. When in doubt use two-tailed.
  3. In version 27 and later, t test tables show One-Sided p and Two-Sided p columns; the two-sided value is double the one-sided one for symmetric tests.

Test statistic, degrees of freedom and p

Each test turns your data into a test statistic (t, F, chi-square, U) whose distribution is known under H0. The p value is the tail area beyond your statistic. Degrees of freedom (df) select which version of the distribution applies; they are printed in the table because you must report them, as in t(28) = 2.41.

Python Stand-in: from a t statistic to a p value

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is scipy doing the arithmetic SPSS does internally. Bigger |t| always means smaller p.

Type I and Type II errors

Tests can go wrong in two ways.

H0 really true H0 really false
Reject H0 Type I error (false alarm), probability = alpha Correct (this probability is power)
Do not reject H0 Correct Type II error (a miss), probability = beta

Alpha is your tolerated false-alarm rate. Power (1 - beta) is the chance to detect a real effect, and it grows with sample size, effect size and alpha. A non-significant result from a tiny sample often just means low power.

Python Stand-in: alpha is the false-alarm rate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

About 5% of tests on pure noise come out significant. That is why running dozens of tests and reporting only the significant ones is dangerous: the more tests, the more false alarms. Corrections such as Bonferroni (built into post hoc dialogs) exist for this reason.

Confidence intervals

SPSS prints a 95% Confidence Interval of the Difference with most tests. It is a range of plausible values for the true effect. If the interval for a difference excludes 0, the two-sided p is below .05. The interval is more informative than p because it also shows how large the effect might be.

flowchart LR
    Q["Research question"] --> H["State H0 and H1"]
    H --> T["Pick a test (next lessons)"]
    T --> C["Check assumptions"]
    C --> R["Run it, read Sig., CI, effect size"]
    R --> D["Decide and report in words"]

The same loop applies to every test in this chapter.

A reporting template

Write results the same way each time: the test, the statistic with df, p, an effect size and what it means in plain language. For example: "The groups differed, t(28) = 2.41, p = .023, d = 0.88: the treated group scored higher." The details of each test follow in the next lessons, starting with choosing the right test.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a p value of .03 tell you?

    A p value is the probability of data at least this extreme given H0. It says nothing direct about the size of the effect or the truth of H0.
  2. 2. SPSS shows Sig. = .000. How should you report it?

    The value is rounded to three decimals; the true p is small but never exactly zero.
  3. 3. Which change increases the power of a test?

    Power grows with sample size and effect size. Lowering alpha actually reduces power.
  4. 4. A 95% CI for a mean difference is [-0.4, 2.1]. What can you say about the two-sided p?

    An interval containing 0 means the difference is not significant at the matching alpha of .05.

Summary

  • H0 states no effect; the p value measures how surprising your data are if H0 were true.
  • Compare Sig. with alpha (usually .05); report .000 as p < .001.
  • A significant p is not a large effect: always add an effect size and a confidence interval.
  • Alpha is the false-alarm rate; power is the chance of finding a real effect.
  • Many tests mean many false alarms, so correct for multiple comparisons.

Next: Choosing the Right Test