Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 37 ยท Entropy and Information

Mutual Information

Mutual information I(X;Y) measures how many bits one variable tells you about another. Compute it from a joint table, estimate it from samples and compare with scikit-learn style checks.

How much does seeing Y tell you about X? Mutual information answers in bits. It is the reduction in uncertainty about one variable from observing the other:

TEXT
I(X; Y) = H(X) - H(X | Y) = H(Y) - H(Y | X) = H(X) + H(Y) - H(X, Y)

Unlike correlation, which captures only linear relationships, mutual information detects any statistical dependence. It is symmetric, never negative, and zero exactly when the variables are independent. It is also the quantity that defines channel capacity: the information the output of a channel carries about its input.

Three equal formulas

Start from the chain rule (Joint and Conditional Entropy). Since H(X, Y) = H(X) + H(Y | X), rearranging gives all three forms above. A fourth form works directly with probabilities:

TEXT
I(X; Y) = sum over x,y of p(x, y) log2 [ p(x, y) / (p(x) p(y)) ]

Each term compares the actual joint probability with what independence would predict. If the ratio is 1 everywhere, the logs vanish and I = 0.

Python Mutual information four ways

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All four values agree. Seeing whether someone carries an umbrella tells you about 0.39 bits about the weather, out of a weather entropy of about 0.97 bits.

Extreme cases

  • Independent variables: I(X; Y) = 0.
  • Identical variables (Y = X): I(X; X) = H(X). A variable carries exactly its own entropy about itself, which is why entropy is sometimes called self-information.
  • In general 0 <= I(X; Y) <= min(H(X), H(Y)).
Python Independence, identity and in between

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Mutual information beats correlation

Correlation misses nonlinear relationships. Take X uniform on {-2, -1, 0, 1, 2} and Y = X^2. The relationship is perfect and deterministic, but the Pearson correlation is exactly zero by symmetry. Mutual information sees it.

Python A relationship correlation cannot see

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The correlation is 0, but I(X; Y) = H(Y) โ‰ˆ 1.52 bits because X determines Y completely. That is what "any dependence" means.

Estimating mutual information from samples

The code above computes the plug-in estimate from counts. It is easy and works well when there are many samples per cell. With a continuous variable you must bin it first, and the estimate is biased upward for small samples (random noise looks like a bit of dependence). Always compare against a shuffled baseline.

Python Bias and a shuffle baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The true value is 0, yet the small-sample estimates are clearly positive and shrink as n grows. Subtracting the average MI of shuffled data is a simple correction.

Scikit-learn's mutual_info_score computes the same quantity in nats. Divide by ln 2 to get bits. This unit mismatch is a classic source of confusion when comparing tools.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which is NOT a correct expression for mutual information?

    H(X,Y) - H(X) - H(Y) is the negative of the mutual information.
  2. 2. If X and Y are independent, I(X; Y) equals

    Independence means p(x,y) = p(x)p(y), so every log term is 0.
  3. 3. Y = X^2 with X symmetric around 0. What do correlation and mutual information report?

    Pearson correlation measures only linear dependence; mutual information detects any dependence.
  4. 4. What is I(X; X)?

    A variable's information about itself is its entropy.

Summary

  • I(X; Y) = H(X) - H(X | Y) = H(X) + H(Y) - H(X, Y): bits that one variable reveals about the other.
  • It is symmetric, non-negative, zero iff independent, and I(X; X) = H(X).
  • It detects nonlinear dependence that correlation misses.
  • Plug-in estimates from few samples are biased upward; use a shuffle baseline and remember library functions may report nats.

Next: KL Divergence and Cross-Entropy