Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 57 ยท Random Variables

Joint Distributions and Covariance

Learn joint, marginal and conditional distributions of two random variables, test independence, and compute covariance and correlation by hand and with NumPy.

Most interesting questions involve more than one variable: height and weight, study time and score, two stocks. A joint distribution describes how two random variables behave together. From it you can recover each variable on its own, ask conditional questions and measure how strongly the two move in step. This lesson closes the chapter on random variables.

Joint PMF

For two discrete variables X and Y, the joint PMF gives the probability of each pair:

p(x, y) = P(X = x and Y = y)

All entries are non-negative and add up to 1. A small example: X is the number of items in a cart (1 or 2) and Y is whether the customer used a coupon (0 or 1).

Y = 0 Y = 1 Row total
X = 1 0.30 0.10 0.40
X = 2 0.20 0.40 0.60
Column total 0.50 0.50 1.00

Marginal distributions

The marginal distribution of one variable is found by summing the joint over the other. The totals in the margins of the table are exactly these: P(X = 1) = 0.40, P(X = 2) = 0.60, P(Y = 1) = 0.50. The name comes from writing them in the margins.

Conditional distributions

Fix one variable and renormalize, using the conditional probability rule:

p(y | x) = p(x, y) / p(x)

Given the customer bought 2 items, P(Y = 1 | X = 2) = 0.40 / 0.60 = 0.667. Given 1 item, P(Y = 1 | X = 1) = 0.10 / 0.40 = 0.25. Customers with bigger carts use coupons more often.

Independence of random variables

X and Y are independent if the joint PMF factors for every pair: p(x, y) = p(x) * p(y). Here p(1, 1) = 0.10 but p(1) * p(1) = 0.4 * 0.5 = 0.20, so the variables are dependent.

Python Marginals, conditionals and independence

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Covariance

To measure how two variables vary together, use their covariance:

Cov(X, Y) = E[(X - mu_X)(Y - mu_Y)] = E[XY] - E[X] E[Y]

  • Positive: when X is above its mean, Y tends to be above its mean too.
  • Negative: they move in opposite directions.
  • Zero: no linear relationship.

Independent variables have covariance 0, but the reverse is false: X uniform on -1, 0, 1 and Y = X^2 are strongly dependent, yet their covariance is 0 because the relationship is not linear.

For the coupon table (taking X as 1 or 2 and Y as 0 or 1): E[X] = 1.6, E[Y] = 0.5, E[XY] = 1 * 1 * 0.10 + 2 * 1 * 0.40 = 0.90. So Cov = 0.90 - 1.6 * 0.5 = 0.10.

Python Covariance from a joint table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Correlation

Covariance depends on units (centimeters times kilograms), so its size is hard to judge. The correlation coefficient rescales it to a unit-free number between -1 and 1:

rho = Cov(X, Y) / (sd_X * sd_Y)

  • rho = 1: a perfect increasing straight line.
  • rho = -1: a perfect decreasing line.
  • rho = 0: no linear association.

Correlation measures linear association only and says nothing about causation.

Python Correlation of simulated pairs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Variance of a sum

For any two random variables:

Var(X + Y) = Var(X) + Var(Y) + 2 * Cov(X, Y)

When variables are independent the covariance term vanishes (the rule you used in expected value and variance). When they are positively correlated, the sum fluctuates more than independence would suggest; when negatively correlated, less. This is the mathematical reason investors diversify across assets that do not move together.

Python Why diversification works

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With rho = 1 there is no benefit (sd stays 0.20), and as correlation falls the portfolio risk drops.

flowchart TD
    J["Joint distribution<br/>p(x, y)"] --> M["Marginals<br/>sum over the other variable"]
    J --> C["Conditionals<br/>p(y | x) = p(x,y) / p(x)"]
    J --> V["Covariance and correlation<br/>how they move together"]
    M --> I["Independent if<br/>p(x,y) = p(x) p(y)"]

One joint table contains the marginals, the conditionals and the dependence structure.

Continuous joint distributions

For continuous variables a joint density f(x, y) replaces the table, with probabilities as volumes under a surface, and marginals found by integrating out the other variable. The bivariate normal, parametrized by two means, two standard deviations and a correlation, is the standard example. The ideas are identical, and they return in regression and in multivariate analysis.

Common mistakes

  • Equating zero covariance with independence.
  • Interpreting correlation as causation, or as the slope of a line.
  • Comparing covariances across variables with different units.
  • Computing a sum's variance without the covariance term.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How do you get the marginal distribution of X from a joint PMF?

    Summing out Y leaves the distribution of X alone.
  2. 2. If X and Y are independent, then Cov(X, Y) is...

    Independence implies zero covariance (though not the other way round).
  3. 3. Correlation of 0.8 between height and weight means...

    It describes the direction and strength of linear association, not causation.
  4. 4. Var(X) = 4, Var(Y) = 9, Cov(X, Y) = 3. What is Var(X + Y)?

    4 + 9 + 2 * 3 = 19.

Summary

  • A joint distribution gives probabilities of pairs; marginals come from summing out the other variable.
  • Conditional distributions renormalize a row or column: p(y | x) = p(x, y) / p(x).
  • Variables are independent when the joint equals the product of the marginals.
  • Covariance measures joint linear variation; correlation is its unit-free version between -1 and 1.
  • Var(X + Y) = Var(X) + Var(Y) + 2 Cov(X, Y); zero covariance does not prove independence.

Next: Common Distributions