Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 28 ยท Hypothesis Testing

Classical Tests and Power

Run t-tests, z-tests, proportion tests and confidence intervals with statsmodels.stats, then plan sample size with power analysis for t-tests and proportions.

Statsmodels is not only regression. statsmodels.stats holds the classic hypothesis tests, confidence intervals and power analysis. Unlike a bare p-value from a calculator, these functions give you the test statistic, p-value and the degrees of freedom, and they share a consistent interface.

A hypothesis test starts with a null hypothesis (no difference, no effect), computes how surprising the data would be if it were true, and reports that surprise as a p-value. A small p-value is evidence against the null. It is not the probability that the null is true, and it says nothing about how large the effect is.

Comparing means

ttest_ind compares two independent groups, ttest_1samp-style tests use DescrStatsW, and ttest_ind accepts usevar="unequal" for Welch's version.

Python Two-sample t-test and confidence interval

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Always report the difference and its interval, not only the p-value: the interval shows how big the effect could be, which is what readers care about.

Comparing proportions

Conversion rates, defect rates and survey shares are proportions. proportions_ztest takes counts of successes and sample sizes; proportion_confint gives an interval for a single proportion (Wilson is a good default).

Python A/B test on proportions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Here the difference (6.0% vs 7.5%) is on the edge of significance. This is exactly where power matters.

Statistical power

Power is the chance a test detects an effect that is really there. Underpowered studies miss real effects and exaggerate the ones they do find. Power depends on four connected quantities; fix three and solve for the fourth:

  1. the effect size (standardized, for example Cohen's d),
  2. the sample size,
  3. the significance level alpha (often 0.05),
  4. the power itself (often 0.8 is the target).
Python Plan a sample size

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Small effects need big samples. That A/B test with 2000 visitors per group had far too little power for a 1.5 point lift, which is why the p-value hovered near 0.05.

Python Power curve

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Compute the sample size before collecting data, using the smallest effect that would matter in practice. Computing "observed power" from an experiment that is already finished adds no information beyond the p-value.

For equivalence ("is the difference small enough to ignore?") use ttost_ind. For paired data, test the differences with DescrStatsW(a - b). Distribution-free alternatives and ANOVA are in ANOVA and Multiple Testing; SciPy has similar tests in Hypothesis Tests.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a p-value of 0.02 mean?

    A p-value assumes the null is true and measures how surprising the data would be.
  2. 2. Which argument gives Welch's t-test (unequal variances) in ttest_ind?

    statsmodels.stats.weightstats.ttest_ind uses usevar='pooled' or 'unequal'.
  3. 3. Which function compares conversion counts of two versions?

    proportions_ztest takes successes and totals for the groups.
  4. 4. To detect a smaller effect with the same power you need what?

    Power rises with sample size and effect size, so smaller effects require more data.

Summary

  • statsmodels.stats offers t-tests, proportion tests and intervals with consistent outputs.
  • Report effect sizes and confidence intervals, not just p-values.
  • Power analysis (TTestIndPower, NormalIndPower) solves for sample size, effect size or power.
  • Plan sample size before collecting data.

Next: ANOVA and Multiple Testing