Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 37 ยท Getting Started

A Probability Refresher

The probability you need for information theory: distributions, expectation, joint and marginal tables, conditional probability, independence and Bayes' rule, with NumPy.

Information theory is probability theory seen through a logarithm. Before building on it, let us fix notation and revisit the handful of ideas used in every later lesson: distributions, expected value, joint and conditional probability, independence and Bayes' rule.

If you know this material already, skim the code and move on. If not, take your time: every formula here returns in Entropy and Mutual Information.

Random variables and distributions

A discrete random variable X takes values from a finite alphabet, for example {a, b, c}. Its distribution assigns each value a probability p(x). The probabilities are non-negative and sum to 1. We represent a distribution as a NumPy vector.

Python A distribution as an array

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The observed frequencies are close to the theoretical probabilities. This law of large numbers is the reason entropy, an average over a distribution, predicts the average cost of real data.

Expected value

The expectation of a function f(X) is the probability-weighted average

TEXT
E[f(X)] = sum over x of p(x) * f(x)

Entropy will turn out to be E[-log2 p(X)], the expected surprisal. Expectation is linear: E[aX + bY] = aE[X] + bE[Y], even if X and Y are dependent. That linearity is behind many later proofs.

Python Expectation by formula and by simulation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first line is exactly 1.875; the simulated mean lands very close to it.

Joint, marginal and conditional

Two variables X and Y have a joint distribution p(x, y), naturally a 2-D table whose entries sum to 1. From it:

  • Marginals sum out the other variable: p(x) = sum_y p(x, y).
  • Conditional probability restricts to a row or column and renormalizes: p(y | x) = p(x, y) / p(x).
  • The chain rule: p(x, y) = p(x) * p(y | x).
Python Joint table, marginals and conditionals

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The conditional rows each sum to 1 because each is a proper distribution. When it rains, the umbrella probability is 0.30 / 0.40 = 0.75.

Independence

X and Y are independent when knowing one tells you nothing about the other: p(x, y) = p(x) p(y) for every pair. Equivalently p(y | x) = p(y). In the weather table above they are clearly not independent. Independence is the situation where information adds (Bits and Surprise), and it is the baseline against which mutual information measures dependence.

Python Test independence numerically

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Bayes' rule

Because p(x, y) can be factored either way, p(x) p(y | x) = p(y) p(x | y), and so

TEXT
p(x | y) = p(y | x) p(x) / p(y)

Bayes' rule updates a belief p(x) into a posterior p(x | y) after seeing evidence y. It is central to decoding: a receiver sees the noisy output y and asks which input x was most likely sent.

Python Bayes: a noisy test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Even with a good detector, the posterior is only about 11%, because the symbol is rare. Priors matter, and a decoder that ignores them can do worse than one that uses them.

Whenever a probability problem feels tangled, write down the joint table. Marginals, conditionals and Bayes all fall out of it with sums and divisions.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A joint table p(x, y) has entries summing to 1. How do you get the marginal p(x)?

    Marginalizing means summing out the other variable: p(x) = sum_y p(x, y).
  2. 2. X and Y are independent exactly when

    Independence means the joint factors into the product of marginals.
  3. 3. Which statement about expectation is always true?

    Linearity holds for any random variables. The other equalities fail in general.
  4. 4. In the detector example, why is P(present | fire) so low despite a 95% detection rate?

    With a 1% prior, false alarms from the 99% absent cases outnumber true detections.

Summary

  • A distribution is a non-negative vector summing to 1; frequencies converge to it.
  • E[f(X)] = sum p(x) f(x) and expectation is linear.
  • Marginals sum out a variable; conditionals renormalize a slice; the chain rule is p(x, y) = p(x) p(y | x).
  • Independence means the joint is the product of marginals.
  • Bayes' rule turns p(y | x) into p(x | y) and underlies decoding.

Next: The Communication System