Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 76 Β· Statistics for Data Science

Chi-Square Tests - Goodness of Fit and Independence

Learn the chi-square goodness-of-fit and independence tests: expected counts, degrees of freedom, assumptions, Cramer's V effect size and checking model fit, with Python.

Means and standard deviations suit numbers. When your data are counts in categories (how many users picked each plan, how many orders arrived each weekday, how many patients improved under each treatment), the right tool is the chi-square test. It compares the counts you observed with the counts you would expect if some assumption were true, and turns the gap into one number.

There are two main versions. The goodness-of-fit test asks whether one categorical variable follows a claimed distribution. The test of independence asks whether two categorical variables are related. Both use the chi-squared distribution to turn the statistic into a p-value, and both are introduced briefly in Hypothesis Testing. Here we go through them in detail.

The statistic

For every category (or table cell), take the difference between observed (O) and expected (E) counts, square it, divide by E, and add up:

chiΒ² = sum of (O - E)Β² / E

Squaring makes every gap positive, and dividing by E means a gap of 10 matters more when you expected 20 than when you expected 2,000. If observed and expected agree, chiΒ² is near 0. The larger it is, the worse the fit. Under the null hypothesis the statistic follows a chi-squared distribution, and the test is always upper-tailed: only large values are evidence against the null.

Goodness of fit: does one variable follow a claimed distribution?

Suppose a support team believes tickets arrive evenly across the five working days. You count 200 tickets over a week. If the claim were true, you would expect 40 per day. The null hypothesis is "the distribution is uniform"; the degrees of freedom are categories - 1 = 4.

You can also test against uneven expectations, such as "plan sizes should follow a 50/30/20 split" or "defects follow the proportions in last year's report". Give the expected proportions and multiply by the total to get expected counts. When the expected distribution has parameters estimated from the same data (for instance a Poisson with the sample mean as its rate), subtract one more degree of freedom per estimated parameter (ddof).

Python Goodness-of-fit test with SciPy

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Residuals tell you where the misfit is. A residual beyond about +/-2 marks a category that differs noticeably from expectation. Here Monday is high and Wednesday through Friday are low, so the claim of an even spread is rejected (p is about 0.005).

Now there are two categorical variables, shown as a contingency table (a cross-tab, built in pandas with pd.crosstab). The null hypothesis says the variables are independent: knowing a row tells you nothing about the column. Under independence, the expected count in each cell is

E = (row total Γ— column total) / grand total

The degrees of freedom are (rows - 1) Γ— (columns - 1). scipy.stats.chi2_contingency returns the statistic, the p-value, the degrees of freedom and the full table of expected counts, so you can check assumptions easily.

Assumptions

  • Observations are independent (each person or event counted once).
  • The data are counts, not percentages or averages.
  • Expected counts are large enough: a common rule says every expected count at least 5 (or at least 80% of cells at 5 and none below 1). For 2Γ—2 tables with small counts use Fisher's exact test (stats.fisher_exact). Merge sparse categories if that makes sense.
  • By default SciPy applies Yates' continuity correction to 2Γ—2 tables; leave it on unless you have a reason.

A significant result is not a strong result: CramΓ©r's V

With enough data, even a trivial association is "significant". Measure the size of the association with CramΓ©r's V, which runs from 0 (none) to 1 (perfect):

V = sqrt(chiΒ² / (N Γ— (min(rows, columns) - 1)))

Rough labels for small tables: 0.1 small, 0.3 medium, 0.5 large.

Python Independence test, expected counts and Cramer's V

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If the test is significant, look at the table and the residuals before writing the conclusion. The chi-square test says "related", never "causes"; see Causal Thinking.

Assessing model fit with chi-square

Goodness-of-fit logic is also a quick model check: does a model's predicted distribution match the data? Bin the observations, compute the model's expected counts per bin, and compare. Two details matter. First, subtract the number of fitted parameters from the degrees of freedom (ddof). Second, a non-significant result means "no evidence of misfit", not "the model is right". With big samples, tiny misfits become significant, so look at the residuals too.

Python Does a Poisson model fit daily order counts?

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check Β· 5 questions

of 5 answered You scored / 5 Β· perfect!

  1. 1. A chi-square goodness-of-fit test has 6 categories. How many degrees of freedom does it have if no parameters are estimated?

    Degrees of freedom are categories minus 1, so 5.
  2. 2. In a 3 x 4 contingency table, how many degrees of freedom does the independence test have?

    (3 - 1) x (4 - 1) = 2 x 3 = 6.
  3. 3. How is the expected count for a cell computed under independence?

    If the variables are independent, the cell probability is the product of the row and column probabilities.
  4. 4. A chi-square test on 50,000 rows gives p < 0.001 but Cramer's V is 0.02. What should you say?

    Large samples make trivial effects significant. V measures the size of the association.
  5. 5. Which situation calls for Fisher's exact test instead of chi-square?

    The chi-square approximation is unreliable when expected counts are small; Fisher's test is exact.

Summary

  • Chi-square tests compare observed counts with expected counts: chiΒ² = sum of (O - E)Β² / E, always judged in the upper tail.
  • Goodness of fit: one variable against a claimed distribution, df = categories - 1 (minus estimated parameters).
  • Independence: two variables in a contingency table, E = row Γ— column / total, df = (r - 1)(c - 1).
  • Check that expected counts are at least about 5; otherwise merge categories or use Fisher's exact test.
  • Report an effect size (CramΓ©r's V) and look at standardized residuals to see which cells matter.

Next: A/B Testing and Experiments, where counts like these decide real product questions.