Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 39 ยท Exploration and Visualization

Correlation and Relationships

Measure how attributes relate: Pearson and Spearman correlation, covariance, chi-square for categories, mutual information, and why correlation is not causation.

Which attributes move together? Which ones predict the target? Which are redundant copies of each other? Answers come from measures of association. Which measure to use depends on the attribute types: two numbers, two categories, or one of each. This lesson covers the standard measures and the traps around them.

Covariance and Pearson correlation

Covariance says whether two numeric attributes tend to be above their means together (positive) or in opposite directions (negative). Its size depends on units, so it is hard to read. Pearson's correlation coefficient r divides by both standard deviations and lies between -1 and +1:

  • r = +1: a perfect rising straight line. r = -1: a perfect falling line. r = 0: no linear relation.
  • Rough strength labels: |r| < 0.3 weak, 0.3 to 0.7 moderate, > 0.7 strong. Context matters.
Python Covariance versus correlation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Covariance exploded when we changed the unit; correlation did not.

Pearson only sees straight lines

A strong relation can have r near zero if it is curved. Always plot.

Python Zero correlation, perfect relation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Spearman and Kendall: rank correlation

Spearman's rho is the Pearson correlation of the ranks. It detects any consistently rising or falling (monotonic) relation, ignores outliers' size, and works for ordinal data. Kendall's tau compares pairs for concordance; it is preferred for small samples with ties.

Python Pearson versus Spearman on a curved monotonic relation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A correlation matrix with pandas

Python Which attributes relate to the target?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Use it to find predictors (high correlation with the target) and redundancy (high correlation among inputs, see Data Integration and Reduction). Note that age correlates with spend only through income here.

Categorical attributes: chi-square and Cramer's V

For two nominal attributes, build a contingency table of counts and test independence with the chi-square test. It compares observed counts with the counts expected if the attributes were unrelated. A small p-value means the attributes are related. Cramer's V rescales chi-square to 0 to 1 so strengths can be compared.

Python Chi-square test on a contingency table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The test details are explained in Hypothesis Testing for Data Miners.

Mixed types and nonlinear relations

  • Numeric versus categorical: compare group means with a box plot, a t-test (two groups) or ANOVA (more), or use the correlation ratio.
  • Mutual information measures any dependence, linear or not, between two attributes. Zero means independent. It works for categories and numbers and is the basis of many feature-selection filters.
Python Mutual information finds the curve Pearson missed

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Correlation is not causation

Three ingredients routinely fool analysts:

  1. Confounders: a third variable drives both. Ice cream sales and drowning both rise in summer; heat is the confounder.
  2. Reverse causation: A may follow B, not the other way around.
  3. Coincidence: with hundreds of attributes, some pairs correlate by pure chance. Test many pairs and some will look significant.
Python Spurious correlations from chance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When a correlation matters for a decision, support it with more data, a controlled experiment, and a causal story that makes sense.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Pearson's r between x and y = x squared on symmetric data around 0 is close to:

    The relation is perfect but not linear, so Pearson's r is about zero. Always plot.
  2. 2. Which correlation measure suits ordinal data and monotonic curved relations?

    Spearman works on ranks, so it captures monotonic relations and ignores outlier magnitude.
  3. 3. Which test checks independence between two categorical attributes?

    Chi-square compares observed and expected counts in a contingency table.
  4. 4. Ice cream sales and drownings correlate. The best explanation is likely:

    A hidden third variable influences both attributes.
  5. 5. Why does testing 200 attributes produce 'strong' correlations among pure noise?

    With thousands of pairs, some will be large by chance, especially with few rows.

Summary

  • Pearson r measures linear association; Spearman and Kendall measure monotonic association on ranks.
  • Covariance depends on units; correlation does not.
  • Chi-square and Cramer's V relate categorical attributes; mutual information finds nonlinear dependence.
  • Use correlation to find predictors and redundant attributes.
  • Correlation is not causation: beware confounders, reverse causation and chance.

Next: Hypothesis Testing for Data Miners