Skip to content
Tutorials Center
My learning

Topics

Lesson 42 of 64 ยท Statistics and Analysis

Finding Correlation

Measure how columns move together with pandas corr, corrwith and rank-based Spearman correlation, draw a correlation heatmap, and avoid the classic traps.

Do tall people weigh more? Do customers who visit more often spend more? Does the temperature move with ice cream sales? Questions like these ask whether two columns move together, and the standard answer is a correlation coefficient: a single number between -1 and 1.

  • +1 means the columns rise together in a perfectly straight-line pattern.
  • 0 means no straight-line relationship at all.
  • -1 means that when one rises, the other falls in a perfectly straight-line pattern.

This lesson goes deeper than the quick corr() call in Descriptive Statistics. You will build a correlation matrix, read it, draw it as a heatmap, compare Pearson with rank-based correlation, and learn the traps that make correlations lie.

One pair: Series.corr

s1.corr(s2) returns the Pearson correlation between two Series. pandas lines the values up by index and ignores pairs where either value is missing:

Python Pearson correlation of two columns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first pair is clearly related (a value near 0.8), the second is not (near 0). Pearson correlation is covariance divided by the product of the two standard deviations. Dividing removes the units, which is why the result is always between -1 and 1 and does not change if you convert hours to minutes.

The whole table: df.corr()

Called on a DataFrame, corr() returns a correlation matrix: every numeric column against every other. The diagonal is always 1 (a column correlates perfectly with itself) and the matrix is symmetric.

Python A correlation matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two tricks make the matrix usable. select_dtypes("number") keeps only numeric columns (text columns would raise an error; numeric_only=True does the same job). And the upper-triangle mask with stack() turns the matrix into a ranked list (dropna() removes the masked cells) of pairs without the diagonal and the mirrored duplicates, so you can see the strongest relationships first.

One column against many: corrwith

If you have a target (such as spend) and want its correlation with every other column, use corrwith instead of calculating the whole matrix:

Python Correlation with one target

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is a quick way to see which columns could help to predict a target. It is only a first look, because it measures straight-line relationships one column at a time.

A heatmap makes the matrix readable

Numbers in a 10 by 10 matrix are hard to scan. A heatmap colours each cell by its value. Use a diverging colour map centred on zero (blue for negative, red for positive) and fix the colour range to -1 to 1 so that colours mean the same thing on every chart:

Python A correlation heatmap with Matplotlib

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The warm block of age, income and spend shows that these move together; savings is negatively related to spending (cool colours), and height is a pale stripe, unrelated to everything. If you have seaborn, sns.heatmap(corr, annot=True, cmap="coolwarm", vmin=-1, vmax=1) does the same in one line. For the details of colour maps and annotations, see the Matplotlib heatmaps lesson.

Pearson versus Spearman

Pearson measures straight-line relationships. Many real relationships are curved but consistently rising or falling, for example "more practice, higher score, with diminishing returns". Spearman correlation replaces each value by its rank (1st, 2nd, 3rd, ...) and then computes the Pearson correlation of the ranks. It measures whether the columns rise and fall together in any monotonic way, and a single outlier has far less influence.

You can calculate it yourself without any extra library:

Python Spearman is Pearson on ranks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

df.corr(method="spearman") and method="kendall" exist as built-in options; they use SciPy, which pandas does not install for you (pip install scipy). In the browser runner you would load it with await micropip.install("scipy") first, as shown in the descriptive statistics lesson.

When correlations lie

A coefficient is a summary, and summaries hide things. Four traps cause most mistakes.

1. Different pictures, same number. Data sets with very different shapes can have the same correlation. Always plot before you believe the number:

Python The same r for very different data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The curve is a perfect, deterministic relationship, yet its Pearson correlation is near 0: the left half falls while the right half rises, and the two cancel. The third chart gets a high correlation only from two outliers.

2. Correlation is not causation. Ice cream sales and sunburn correlate, because both rise with the temperature. A third factor, called a confounder, drives both. See Causal Thinking for ways to reason about this.

3. Hidden groups. A positive correlation can turn negative inside each subgroup (Simpson's paradox). Compare the overall correlation with df.groupby("group")[["x", "y"]].corr().

4. Chance and small samples. With 10 rows you can see a correlation of 0.5 by luck. With 1,000 columns, some pairs will correlate by pure chance. The more pairs you test, the more you must expect false alarms. Large samples shrink this problem, but they never fix the first three.

Warning: A correlation near zero does not mean "unrelated". It means "no straight-line relationship". Always look at a scatter plot for the pairs that matter.

A practical routine

  1. Clean the data first: outliers and sentinel values such as 999 can create or hide a correlation.
  2. Compute df.corr() for numeric columns and look at it as a heatmap.
  3. List the strongest pairs, then plot each of them as a scatter plot.
  4. Compare Pearson with Spearman. A big difference means curvature or outliers.
  5. Ask whether a third variable could explain the link, and write down your caution in the report.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does a Pearson correlation of -0.9 tell you?

    The sign gives the direction and the size gives the strength of the linear pattern. Nothing about causation follows from it.
  2. 2. What is Spearman correlation, computed by hand?

    Spearman replaces values by ranks and then computes Pearson on those ranks, so it measures any consistently rising or falling pattern.
  3. 3. A perfect parabola y = x^2 over x from -3 to 3 has a Pearson correlation near 0. Why?

    The relationship is perfect but curved. Pearson sees no consistent straight-line direction, which is why you should plot the data too.
  4. 4. Which call shows the correlation of every column with a column named 'target'?

    corrwith computes the correlation of each column of a DataFrame with a given Series.
  5. 5. Why use vmin=-1 and vmax=1 in a correlation heatmap?

    A fixed range stops the colour map from stretching to the data, so a pale cell always means a weak correlation.

Summary

  • A correlation coefficient between -1 and 1 summarises a straight-line relationship; s1.corr(s2) compares two Series and df.corr() builds a matrix.
  • Use select_dtypes("number"), an upper-triangle mask and stack() to list the strongest pairs; use corrwith for one target.
  • Draw the matrix as a heatmap with a diverging colour map and fixed limits of -1 to 1.
  • Spearman correlation (Pearson on ranks) catches monotonic curves and resists outliers.
  • Plot before you trust a number; watch for curves, outliers, confounders, hidden groups and chance findings.
  • Correlation is a starting point for questions, not proof of cause.

Next: Visualisation with pandas shows more ways to draw relationships.