Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 57 ยท Hypothesis Testing

The Logic of Hypothesis Testing

Learn how hypothesis tests work: null and alternative hypotheses, test statistics, p-values, significance levels, Type I and II errors and power, with simulations in Python.

A confidence interval says which values of a parameter are plausible. A hypothesis test asks a sharper question: is the data compatible with a specific claim? Does the new drug lower blood pressure, or could the improvement be luck? Has the website's conversion rate really changed? Hypothesis testing gives a disciplined way to decide when an observed difference is too large to blame on chance.

The two hypotheses

Every test sets up two competing statements about a population parameter:

  • The null hypothesis H0 is the default, "nothing is going on" claim: no effect, no difference, parameter equals a specific value. Example: H0: mu = 100.
  • The alternative hypothesis H1 (or Ha) is what you suspect or want to show: mu > 100 (right-tailed), mu < 100 (left-tailed) or mu != 100 (two-tailed).

The alternative is chosen before looking at the data. We never "prove" H0; we either find enough evidence against it or we do not. It resembles a court trial: the defendant is presumed innocent (H0), and we convict only if the evidence is strong.

The steps

  1. State H0 and H1.
  2. Choose a significance level alpha (commonly 0.05).
  3. Compute a test statistic, which measures how far the sample result is from what H0 predicts, in standard-error units.
  4. Find the p-value.
  5. Decide: if p <= alpha, reject H0; otherwise, fail to reject it.
  6. State the conclusion in plain words about the context.
flowchart TD
    A["State H0 and H1"] --> B["Pick alpha"]
    B --> C["Compute test statistic from data"]
    C --> D["Find p-value assuming H0 is true"]
    D --> E{"p <= alpha?"}
    E -- "Yes" --> F["Reject H0"]
    E -- "No" --> G["Fail to reject H0"]

The standard decision procedure.

The p-value

The p-value is the probability, if H0 were true, of getting a result at least as extreme as the one observed. A small p-value means the data would be surprising under H0, which counts as evidence against it.

It is not the probability that H0 is true, and it is not the chance the result happened "by luck alone" in any general sense. It only measures how compatible the data are with H0.

Python A p-value from a coin simulation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Roughly 2% of fair-coin experiments produce a result at least this lopsided. At alpha = 0.05 we reject fairness, though not at 0.01.

Errors and power

Because a test uses a sample, it can be wrong in two ways:

H0 is true H0 is false
Reject H0 Type I error (false positive), probability alpha Correct (probability = power)
Fail to reject Correct Type II error (false negative), probability beta

The significance level alpha is the Type I error rate you are willing to accept. Power is 1 - beta, the chance of detecting a real effect. Power increases with a larger effect, a bigger sample, a larger alpha and less noisy data. A study with low power often misses real effects and, when it does find one, tends to overstate it.

Python Simulating Type I error and power

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When H0 is true the rejection rate stays near the chosen 5%. As the real mean moves away from 100, power rises.

Statistical versus practical significance

With an enormous sample, a trivial difference (say 0.01 points) can have a tiny p-value. "Statistically significant" only says the effect is probably not zero, not that it is important. Always report the effect size and a confidence interval alongside the p-value, and judge whether the size matters in context.

Python Big samples find tiny effects

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common mistakes

  • p-hacking: trying many tests and reporting only the significant one. With 20 tests at alpha = 0.05, you expect one false positive by chance. Plan analyses in advance or adjust for multiple comparisons (for instance a Bonferroni correction: use alpha / m).
  • Reading "fail to reject" as "H0 is true".
  • Switching to a one-sided test after seeing the direction of the data.
  • Treating p = 0.049 and p = 0.051 as completely different findings.

Confidence intervals and tests agree: a two-sided test at level alpha rejects H0: mu = m0 exactly when m0 lies outside the (1 - alpha) confidence interval.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which is a valid null hypothesis?

    H0 states a specific value for a population parameter (not a sample statistic).
  2. 2. A p-value of 0.03 means:

    The p-value is a probability about the data under H0, not about H0 itself.
  3. 3. A Type I error is:

    A Type I error is a false positive: rejecting H0 when it is actually true.
  4. 4. Which change increases the power of a test?

    More data reduce the standard error and make real effects easier to detect.

Summary

  • Tests compare data with a null hypothesis; the alternative is fixed in advance.
  • The p-value is the probability of data this extreme if H0 were true; reject if p <= alpha.
  • Type I error probability is alpha; power is 1 - beta.
  • Significance is not importance: report effect sizes and intervals.
  • Beware p-hacking and multiple comparisons.

Next: The z-Test