Skip to content
Tutorials Center
My learning

Topics

Lesson 54 of 68 ยท Probability and Statistics for Engineers

Random Variables, Covariance and Correlation

Learn discrete and continuous random variables, joint and marginal distributions, covariance and correlation, with worked tables and numpy simulations.

One number rarely tells the whole story. A bridge carries a load and has a strength; a signal has an amplitude and a noise level; a motor draws current while delivering torque. These quantities are random, and they are usually related: when one is large, the other tends to be large (or small). To describe that relationship you need to treat two or more random variables together.

This lesson sets up the vocabulary for random variables, then builds the joint distribution, the marginals, covariance and correlation. It continues from Probability and Random Variables, and the ideas of conditioning come from Conditional Probability and Bayes' Theorem.

Discrete and continuous random variables

A random variable is a number produced by a random experiment. It comes in two flavours:

  • A discrete random variable takes values you can list: the number of defects on a board (0, 1, 2, ...), the number of packets lost. It is described by a probability mass function (PMF) p(x) = P(X = x), with p(x) >= 0 and the sum of all p(x) equal to 1.
  • A continuous random variable takes any value in an interval: a voltage, a length, a time. It is described by a density f(x), and probabilities are areas, as in the previous lesson.

For both, the mean is a weighted average: E[X] = sum of x p(x) (discrete) or integral of x f(x) dx (continuous). The variance is Var[X] = E[X**2] - E[X]**2 in both cases. Every formula in this lesson has a sum version and an integral version that look alike.

Joint, marginal and conditional distributions

Two discrete variables X and Y have a joint PMF p(x, y) = P(X = x and Y = y). The entries of the table add to 1. The marginal distribution of one variable adds across the other:

pX(x) = sum over y of p(x, y), and pY(y) = sum over x of p(x, y).

(For continuous variables, a joint density f(x, y) has volume 1 under its surface, and the marginal is fX(x) = integral of f(x, y) dy; see Multiple Integrals.) The conditional PMF is p(y | x) = p(x, y) / pX(x).

The variables are independent when the joint is the product of the marginals, p(x, y) = pX(x) pY(y), for every cell. One cell that disagrees is enough to make them dependent.

Here is a small table. X is the number of defects found on a circuit board (0, 1 or 2) and Y is whether the board needed a second test run (0 = no, 1 = yes).

Python A joint PMF: marginals, covariance and independence

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The marginals are X: 0.5, 0.3, 0.2 and Y: 0.6, 0.4. If the two were independent, the top-left cell would be 0.5 x 0.6 = 0.30, but the table says 0.40, so they are dependent. The conditional probability of a retest given two defects is 0.75, against 0.40 overall: more defects, more retests.

Covariance: do they move together?

The covariance measures how two variables vary together:

Cov(X, Y) = E[(X - muX)(Y - muY)] = E[XY] - E[X] E[Y]

When X is above its mean at the same time Y is above its mean (and below together), the products are positive on average, and the covariance is positive. If one tends to be high when the other is low, it is negative. In the table, Cov = 0.45 - 0.7 x 0.4 = 0.17.

Covariance has three useful rules:

  • Cov(X, X) = Var[X].
  • Cov(aX + b, cY + d) = a c Cov(X, Y): it scales with the units of both variables.
  • Var[X + Y] = Var[X] + Var[Y] + 2 Cov(X, Y). The "variances add" rule from the previous lesson is the special case of zero covariance.

The last rule is the heart of engineering design with random quantities. If two tolerances move together (Cov > 0) the total spread is larger than the independent case, and if they cancel (Cov < 0) it is smaller.

Correlation: covariance without units

A covariance of 200 means nothing without the units. The correlation coefficient divides them out:

rho = Cov(X, Y) / (sigma_X sigma_Y)

It always lies between -1 and +1. The extremes mean a perfect straight-line relationship (Y = aX + b with a > 0 gives +1, with a < 0 gives -1), and 0 means no linear relationship. For the table above, rho is about 0.444. Changing the units of X (volts to millivolts) changes the covariance but not the correlation.

Independence implies zero covariance, but zero covariance does not imply independence. Correlation only sees straight-line patterns. Take X symmetric around zero and Y = X**2: Y is completely determined by X, yet the covariance is zero because positive and negative x cancel.

Python Covariance versus correlation, and a nonlinear dependence

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The linear case has covariance near 2 and correlation about 0.895 (exactly 2 / sqrt(1 x 5)). The square case has correlation near zero even though y_sq is a function of x. Multiplying x by 100 makes the covariance 100 times larger (about 199.6) and leaves the correlation at 0.895. The last line confirms the variance-of-a-sum rule, with both sides near 9.98 (theory: 1 + 5 + 2 x 2 = 10).

The bivariate normal

The most common joint model in practice is the bivariate normal, specified by two means, two standard deviations and one correlation rho. Contours of its density are ellipses, tilted when rho is not zero. A special feature: for normal variables, zero correlation does imply independence. Another: given X = x (with unit variances and zero means), Y is normal with mean rho x.

Python Sampling a correlated normal pair

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With rho = 0.8, both variables exceed 1 together about 9.7% of the time, nearly four times the 2.5% you would expect if they were independent. This is why correlated loads matter in structural design: two moderately rare events become much more likely to happen together. The conditional mean 0.798 matches the theoretical rho x 1.

Common slips

  • Reading correlation as causation. Two quantities can move together because a third drives both.
  • Concluding "no relationship" from rho = 0. Plot the data: the dependence may be curved.
  • Comparing covariances of variables with different units; compare correlations instead.
  • Adding variances without checking whether the variables are independent.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. How do you get the marginal PMF of X from a joint table?

    The marginal of X adds the joint probability over every value of Y, which collapses the table onto the X axis.
  2. 2. If X and Y are independent, what is Cov(X, Y)?

    Independence gives E[XY] = E[X] E[Y], so the covariance is zero. The reverse is false in general.
  3. 3. Var[X + Y] equals Var[X] + Var[Y] + 2 Cov(X, Y). When is the simple sum Var[X] + Var[Y] correct?

    The cross term vanishes only when Cov(X, Y) = 0.
  4. 4. X is symmetric around 0 and Y = X**2. What is the correlation of X and Y?

    Correlation detects only linear relationships. Positive and negative x give the same y, so the covariance cancels to zero.
  5. 5. What happens to the correlation coefficient if X is measured in millivolts instead of volts?

    Correlation divides out both standard deviations, so it is unit-free and invariant to positive rescaling.

Summary

  • A discrete random variable has a PMF, a continuous one a density; means and variances use sums or integrals of the same shape.
  • A joint distribution describes two variables at once; marginals add (or integrate) across the other variable.
  • Cov(X, Y) = E[XY] - E[X] E[Y] and Var[X + Y] = Var[X] + Var[Y] + 2 Cov(X, Y).
  • Correlation rho is covariance divided by both standard deviations: unit-free, between -1 and +1, and blind to curved relationships.
  • Independence implies zero covariance, but not the other way round (except for jointly normal variables).

Next: Binomial, Poisson and Other Discrete Distributions.