Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 76 ยท Exploratory Data Analysis

Relationships Between Variables

Explore how columns relate: scatter plots, correlation matrices, grouped comparisons, cross-tabs, Simpson's paradox, and why correlation is not causation.

Single columns tell you what the data looks like; relationships tell you what it might mean. Does revenue rise with advertising? Do customers on the premium plan stay longer? Is the lower failure rate of one machine real, or does that machine just handle easier jobs? This is the bivariate and multivariate part of the EDA process, and it builds on the one-variable checks.

The right tool depends on the types of the two columns:

Column A Column B Typical tools
Numeric Numeric Scatter plot, correlation
Numeric Category Grouped summaries, boxplots by group
Category Category Cross-tab (contingency table), stacked or grouped bars
flowchart LR
    A["Two variables"] --> B{"Types?"}
    B -->|"num + num"| C["Scatter plot<br/>correlation"]
    B -->|"num + cat"| D["Group means or medians<br/>boxplot per group"]
    B -->|"cat + cat"| E["Cross-tab<br/>row or column shares"]

Pick the plot or table from the types of the two columns.

Numeric against numeric: scatter plots and correlation

A scatter plot puts one variable on each axis, one dot per row. Look at the pattern before computing anything: is it a line, a curve, a cloud, or several clusters? With thousands of points, use small markers and transparency (alpha) so overlapping dots do not hide the density.

The Pearson correlation coefficient r summarizes the strength of a straight-line relationship on a scale from -1 to +1. Values near +1 mean both rise together, near -1 one rises as the other falls, near 0 no linear relationship. Two cautions:

  • Pearson measures only linear association. A perfect U-shaped relationship can have r near 0.
  • It is sensitive to outliers. Spearman correlation works on ranks, so it captures any steadily rising or falling relationship and resists extreme values.
Python Linear, curved and outlier-driven relationships

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The U-shape has a clear relationship and a Pearson r of about 0.04. The curved rising case has a Spearman of 0.98 against a Pearson of 0.79, because ranks ignore the curvature. In the last panel, 60 unrelated points plus one extreme point give a Pearson r of 0.82, while Spearman is only about 0.25. Always plot first; a correlation number alone can mislead.

Correlation matrices

With many numeric columns, a correlation matrix (df.corr()) shows every pairwise r at once, and a heat map makes it readable. Use it to spot strongly related pairs (candidates for redundancy), and the columns most related to your outcome. Treat it as a map of leads, not a conclusion: correlations are only linear, and with dozens of columns some large values appear by chance (see the section on fishing in the EDA process).

Numeric against category: grouped comparisons

To compare a number across groups use groupby with several summaries (count, mean, median, spread). Always include the count: a mean over four rows is not comparable with a mean over four thousand. Show the distributions too (boxplots or overlaid histograms), because two groups can have the same mean and very different spreads.

Category against category: cross-tabs

pd.crosstab counts rows for each combination of two categories. Raw counts are dominated by group sizes, so convert to shares with normalize="index" (each row sums to 1) or normalize="columns". Decide which question you are asking: "of the customers on each plan, what share churned?" is a row share.

Python Grouped comparison and cross-tab

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The counts table is hard to compare because the plans have different sizes; the row shares make the story obvious: churn falls as the plan level rises. Whether the plan causes that is a separate question (see below).

Simpson's paradox: when groups reverse the story

A trend that appears in the combined data can reverse when you split by a third variable. This is Simpson's paradox. It happens when groups differ in size and in baseline rate, so the mix of groups, not the treatment, drives the overall number.

Here is a made-up example. Two tutoring programs, X and Y, are compared by the share of students passing a test. Students are either "advanced" or "basic" learners, and the two programs enrolled very different mixes.

Python Simpson's paradox with tutoring programs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Overall, program Y looks much better (80% against 59%). Within each level, program X does better (95% against 90%, and 50% against 40%). The overall result is misleading because program Y mostly enrolled advanced students, who pass more often whatever the program. Which table is "right" depends on the causal story: if level influences who enrolls, compare within levels. This is why the multivariate pass matters, and why Causal Thinking and Confounding exists.

flowchart LR
    L["Student level<br/>(advanced or basic)"] --> P["Program chosen"]
    L --> R["Pass rate"]
    P --> R

Level influences both which program a student joins and whether they pass; ignoring it distorts the comparison of programs.

Correlation is not causation

Two variables moving together does not mean one causes the other. Three common reasons for association:

  1. Causation: A really affects B (maybe).
  2. Reverse causation: B affects A.
  3. A confounder: a third variable C drives both. Hot weather raises both ice-cream sales and swimming accidents, so they correlate although neither causes the other.
Python A confounder creates a correlation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The overall correlation is strong. Once the part explained by temperature is removed from each column, the leftover correlation is close to zero: nothing links the two columns except the shared driver. Comparing within groups of similar temperature, or adjusting like this, is the basic way to check whether a third variable explains an association (see Causal Thinking and Confounding).

A checklist for any relationship

  • Did I plot it, not only compute a number?
  • What are the sample sizes behind each group?
  • Could a third variable (time, region, customer type) explain it? Did I split by it?
  • Could the direction be reversed?
  • How many relationships did I look at to find this one?

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A scatter plot shows a clear U shape, but the Pearson correlation is close to 0. Why?

    A symmetric U has no net linear trend, so Pearson r can be near zero even though the relationship is strong. Always plot first.
  2. 2. Why should grouped comparisons include the count per group?

    A mean from four rows is noisy. Showing counts tells the reader how much to trust each number.
  3. 3. What does normalize="index" do in pd.crosstab?

    Row shares answer "of the customers in each row group, what fraction fall in each column category", independent of group size.
  4. 4. In Simpson's paradox, what happens?

    Groups with different sizes and baselines can make the overall comparison point the opposite way from the comparison inside each group.
  5. 5. Ice-cream sales and swimming accidents are strongly correlated. The best explanation is:

    Temperature is a confounder, since it raises both quantities, producing a correlation without any direct link between them.

Summary

  • Choose the tool from the column types: scatter and correlation, grouped summaries, or cross-tabs.
  • Plot before trusting a correlation; Pearson sees only linear association and is sensitive to outliers, Spearman uses ranks.
  • Show group sizes next to group statistics; use row or column shares in cross-tabs.
  • Simpson's paradox shows that a third variable can reverse a trend, so check the main relationships within subgroups.
  • Correlation does not prove causation: consider reverse causation and confounders.

Next: Trends, Seasonality and Time Patterns, where one of the variables is time.