Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 39 ยท Exploration and Visualization

Hypothesis Testing for Data Miners

Learn null and alternative hypotheses, p-values, t-tests, z-tests, chi-square tests, Type I and II errors and confidence intervals, with runnable scipy examples.

Data mining hands you patterns: group A spends more than group B, this rule has high confidence, this feature looks predictive. Is the pattern real, or could it have appeared by luck in this sample? Hypothesis testing gives a disciplined answer. It is part of the statistical toolkit every miner needs, especially when you test hundreds of candidate patterns.

The logic

  1. State the null hypothesis H0: there is no effect or difference (the default, boring claim).
  2. State the alternative hypothesis H1: there is an effect.
  3. Pick a significance level alpha, commonly 0.05.
  4. Compute a test statistic from the sample and its p-value: the probability of seeing a result at least this extreme if H0 were true.
  5. If p < alpha, reject H0 (the result is statistically significant). Otherwise you fail to reject it. That does not prove H0.
flowchart LR
    A["Claim: no difference (H0)"] --> B["Collect sample"]
    B --> C["Compute statistic and p-value"]
    C --> D{"p < alpha?"}
    D -- yes --> E["Reject H0: evidence of an effect"]
    D -- no --> F["Fail to reject: not enough evidence"]

A p-value is not the probability that H0 is true, and a small p-value does not mean a big or important effect. With a huge sample, tiny, useless differences become significant.

t-tests: comparing means

Use a t-test when the population standard deviation is unknown (almost always). Three flavors:

  • One-sample: is the mean different from a reference value?
  • Two-sample (independent): do two groups have different means?
  • Paired: do the same subjects differ before and after?
Python Two-sample and paired t-tests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

equal_var=False (Welch) is a safe default because it does not assume equal spreads. T-tests assume roughly normal data or a reasonable sample size; for strongly skewed small samples use a rank test such as Mann-Whitney.

z-test

A z-test compares a mean (or proportion) when the population standard deviation is known or the sample is large (n > 30 is the folk rule). The statistic is z = (x_bar - mu0) / (sigma / sqrt(n)), compared with the standard normal distribution. For proportions, it is the usual test for A/B conversion rates.

Python z-test by hand: two proportions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Chi-square test

For counts in categories, the chi-square test compares observed counts with expected ones. There are two uses: goodness of fit (does a die look fair?) and independence (are two categorical attributes related?, see Correlation and Relationships). Expected counts should be at least 5 or so in most cells.

Python Goodness of fit: is the die fair?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Type I and Type II errors

H0 true H0 false
Reject H0 Type I error (false positive), probability alpha Correct: power = 1 - beta
Fail to reject Correct Type II error (false negative), probability beta

Lowering alpha reduces false alarms but increases misses. Power is the chance of detecting a real effect; it grows with sample size, effect size and alpha.

Python Simulate Type I error and power

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Confidence intervals

A confidence interval (CI) gives a plausible range for the unknown true value. A 95 percent CI procedure captures the true value in 95 percent of repeated samples. It shows both the size and the uncertainty of an effect, which a p-value alone cannot.

Python 95 percent CI for a mean

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The multiple comparisons trap

This is the miner's special danger. Test 1,000 meaningless patterns at alpha = 0.05 and about 50 will look significant by chance. Fixes: the Bonferroni correction (use alpha / m), the Benjamini-Hochberg false discovery rate procedure, and validating findings on held-out data.

Python Fifty false discoveries from pure noise

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A p-value of 0.03 means:

    A p-value is a probability about the data under H0, not about H0 itself.
  2. 2. A Type I error is:

    Type I is a false positive; its rate is alpha.
  3. 3. Which test compares the means of the same subjects before and after a treatment?

    The paired t-test uses the within-subject differences.
  4. 4. You test 500 unrelated patterns at alpha 0.05. Roughly how many false 'discoveries' do you expect?

    5 percent of 500 is 25. Correct for multiple comparisons.
  5. 5. What does a wider confidence interval indicate?

    Wide intervals come from small samples or high variability.

Summary

  • State H0 and H1, pick alpha, compute a statistic and p-value, then reject or fail to reject H0.
  • t-tests compare means (one-sample, independent, paired); z-tests suit known sigma, large samples and proportions.
  • Chi-square tests goodness of fit and independence for counts.
  • Type I error = false positive (alpha); Type II = false negative; power = 1 - beta.
  • Report confidence intervals along with p-values.
  • When mining many patterns, correct for multiple comparisons and validate on new data.

Next: Probability Distributions for Data Mining