Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 37 ยท Hypothesis Tests and Metrics

Chi-Square and Fisher's Exact Test

Test conversion differences with the chi-square test of independence and Fisher's exact test. Learn contingency tables, expected counts, and when to use each.

The z-test is one way to compare two conversion rates. Two other classic tools work from a contingency table of counts: the chi-square test of independence for large samples and Fisher's exact test for small ones. They answer the same question as the z-test but extend to more than two variants and handle tiny counts safely.

The contingency table

Arrange the data as counts of outcomes by variant:

Converted Did not convert
A 480 4,520
B 560 4,440

The null hypothesis is that conversion is independent of the variant: knowing which arm a user was in tells you nothing about whether they converted.

Chi-square test

For each cell, compute the expected count under independence: expected = row total * column total / grand total. The chi-square statistic sums how far observed counts are from expected:

chi2 = sum( (observed - expected)^2 / expected )

A large value means the pattern is unlikely under independence.

Python Chi-square test of independence

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For a 2 x 2 table, chi-square and the two-sided z-test are the same test: chi2 = z^2 and the p-values match. SciPy applies Yates' continuity correction by default (correction=True), which makes it slightly more conservative; for A/B tests with thousands of users the difference is negligible.

More than two variants

The chi-square test generalizes naturally to A/B/C/n tests. It gives one omnibus p-value for "is conversion the same across all variants?" If significant, you follow up with pairwise comparisons (adjusted for multiple testing, see Multiple Comparisons).

Python Three variants at once

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The omnibus test says something differs; it does not say which pair. Do not skip the follow-up, and do not run it without a correction.

Fisher's exact test

Chi-square relies on a large-sample approximation. When counts are small (a rule of thumb: any expected cell count below 5), it is inaccurate. Fisher's exact test computes the exact probability of seeing a table at least as extreme, given the row and column totals, using the hypergeometric distribution. No approximation is needed.

Python Fisher's exact test for small samples

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two p-values differ, and the exact one is the trustworthy answer for small data. (SciPy's odds ratio is for the first row relative to the second, so we invert it to read B versus A.)

Which test should you use?

flowchart TD
    Q["Binary metric, compare arms"] --> S{"Expected counts<br/>all at least 5?"}
    S -- yes --> L["Large sample:<br/>z-test / chi-square<br/>(CI via unpooled SE)"]
    S -- no --> F["Fisher's exact test"]
    L --> M{"More than<br/>two variants?"}
    M -- yes --> C["Chi-square omnibus<br/>+ adjusted pairwise tests"]
    M -- no --> Z["Two-proportion z-test"]

For ordinary A/B traffic volumes, z-test and chi-square give the same answer.

Odds ratios and relative risk

Contingency tables also invite two other effect-size measures:

  • Relative risk (risk ratio): p_B / p_A, the same as 1 + relative lift. Easy to explain.
  • Odds ratio: (p_B/(1-p_B)) / (p_A/(1-p_A)). Common in medicine and logistic regression; it looks bigger than the relative risk when rates are high.

For product experiments, absolute lift and relative lift communicate best. The odds ratio is mostly a modeling convenience.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. For a 2 x 2 table, how does the chi-square test relate to the two-sided two-proportion z-test?

    The two tests are algebraically equivalent for a 2 x 2 table without continuity correction.
  2. 2. When should you prefer Fisher's exact test?

    Fisher's exact test does not rely on a large-sample approximation, so it is reliable for small counts.
  3. 3. What does a significant chi-square omnibus test across three variants tell you?

    The omnibus test is a global check. Follow-up pairwise comparisons, with multiple-testing correction, identify the differences.
  4. 4. What is the null hypothesis of the chi-square test of independence in an A/B test?

    Independence means the variant has no relationship to whether the user converted.

Summary

  • Count data fit in a contingency table; the chi-square test compares observed with expected counts under independence.
  • For 2 x 2 tables it equals the two-sided z-test; with three or more variants it gives an omnibus test.
  • Use Fisher's exact test when expected counts are small.
  • Follow a significant omnibus result with corrected pairwise comparisons.

Next: Bootstrap and Mann-Whitney Tests