Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 37 · Sampling and Inference Basics

Hypothesis Testing Basics

Learn the logic of a significance test: null and alternative hypotheses, test statistic, p-value, significance level, and Type I and II errors, with runnable examples.

A friend gives you a coin and claims it is fair. You flip it 100 times and see 60 heads. Is that convincing evidence of a biased coin, or could a fair coin easily do this by luck? A hypothesis test answers that kind of question in a systematic way.

It builds on sampling distributions and on binomial probabilities.

The steps

  1. State the null hypothesis H₀: the "nothing special" claim (the coin is fair, p = 0.5).
  2. State the alternative hypothesis H₁: what you suspect instead (p ≠ 0.5, two-sided; or p > 0.5, one-sided).
  3. Choose a significance level α, commonly 0.05, before looking at the data.
  4. Compute a test statistic and the p-value.
  5. If p-value ≤ α, reject H₀; otherwise, fail to reject it. State the conclusion in context.

What is a p-value?

The p-value is the probability, assuming H₀ is true, of getting a result at least as extreme as the one observed. A small p-value means the result would be surprising if H₀ were true, which is evidence against H₀.

The p-value is not the probability that H₀ is true. And "fail to reject" is not the same as "H₀ is proven".

Example 1: the coin

If the coin is fair, the number of heads in 100 flips is B(100, 0.5). For a two-sided test with 60 heads, the p-value is the probability of 60 or more heads, or 40 or fewer, which is twice P(X ≥ 60).

Python Exact binomial test for 60 heads in 100 flips

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two-sided p-value comes out slightly above 0.05, so 60 heads in 100 is not quite strong enough evidence to call the coin biased at the 5% level. Compare that with 62 heads or 65 heads by changing observed.

Example 2: a z-test for a mean

A company claims its batteries last 500 hours. You test 40 batteries and get a mean of 510 hours. Suppose the standard deviation is known to be 30 hours. Test H₀: μ = 500 against H₁: μ ≠ 500.

Text
SE = 30 / √40 ≈ 4.743
z = (510 - 500) / 4.743 ≈ 2.108
p-value (two-sided) = 2 × (1 - Φ(2.108)) ≈ 0.035

Since 0.035 < 0.05, reject H₀: there is evidence that the mean differs from 500 hours.

Python One-sample z-test

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The conclusion depends on α. At 1% the same data would not be enough. That is why α is chosen before collecting data.

Two kinds of error

H₀ is true H₀ is false
Reject H₀ Type I error (false alarm) Correct
Fail to reject Correct Type II error (missed effect)

The probability of a Type I error is α. The probability of a Type II error is often called β, and power is 1 - β, the chance of detecting a real effect. Larger samples increase power.

Python Simulate the Type I error rate

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When H₀ is actually true, about 5% of tests wrongly reject it. That is precisely what α = 0.05 means.

Statistical vs practical significance

With a huge sample, even a tiny, useless difference can give a small p-value. Always look at the size of the effect (and a confidence interval) alongside the p-value, and ask whether the difference matters in real life.

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. A test gives p-value = 0.03 with α = 0.05. What is the decision?

    0.03 ≤ 0.05, so the result is statistically significant at the 5% level. It is evidence, not certainty.
  2. 2. Which best describes a p-value?

    It is calculated assuming H₀ is true.
  3. 3. Rejecting a true null hypothesis is called...

    A false alarm is a Type I error, and its probability is α.
  4. 4. How can you increase the power of a test?

    Larger samples reduce the standard error, making real effects easier to detect.

Summary

  • State H₀ and H₁, choose α, compute a test statistic and p-value, then decide.
  • A small p-value is evidence against H₀; it is not the probability that H₀ is true.
  • Type I error = rejecting a true H₀ (probability α); Type II = missing a real effect.
  • Report effect sizes and intervals, not just "significant".

Next: Practice Set: Probability and Distributions