Skip to content
Tutorials Center
My learning

Topics

Lesson 53 of 68 ยท Probability and Statistics for Engineers

Conditional Probability and Bayes' Theorem

Learn conditional probability, the law of total probability and Bayes' theorem with a fault-diagnosis example, tree diagrams and Python simulation.

A smoke alarm sounds. A diagnostic test flags a circuit board. A sensor reading drifts upward. In each case you hold some evidence, and the real question is: given this, how likely is it that something is wrong? That is a conditional probability, and the rule that turns "how likely is this evidence if there is a fault" into "how likely is a fault given this evidence" is Bayes' theorem.

The previous lesson, Probability and Random Variables, described a single quantity with a density. This lesson works with events and their probabilities, which is how most reliability, diagnosis and signal-detection problems are phrased. The arithmetic is only multiplication and addition, but the answers are often surprising.

Conditional probability

The probability of event A given that event B has happened is

P(A | B) = P(A and B) / P(B), defined when P(B) > 0.

Knowing B shrinks the world of possibilities to just the outcomes inside B, and P(A | B) is the share of that smaller world that also lies in A. Rearranged, this is the multiplication rule: P(A and B) = P(A | B) P(B).

Two events are independent when learning one tells you nothing about the other, that is P(A | B) = P(A), or equivalently P(A and B) = P(A) P(B). Independence is an assumption you must justify (two components on the same power rail rarely fail independently). It is not the same as being mutually exclusive: exclusive events with nonzero probabilities are strongly dependent, because if one happens the other cannot.

The law of total probability

Often the world splits into cases, and you know the probability of the outcome within each case. If the events B1, B2, ..., Bn are mutually exclusive and cover every possibility (a partition), then

P(A) = P(A | B1) P(B1) + P(A | B2) P(B2) + ... + P(A | Bn) P(Bn)

This is a weighted average of the case probabilities, with the weights being how common each case is. It is the discrete cousin of E[g(X)] = integral of g(x) f(x) dx from the previous lesson: add up (or integrate) outcome times weight.

flowchart LR
    S["Start"] -- "0.02" --> F["Faulty"]
    S -- "0.98" --> OK["Healthy"]
    F -- "0.95" --> FP["Alarm<br/>0.02 x 0.95 = 0.0190"]
    F -- "0.05" --> FN["No alarm<br/>0.0010"]
    OK -- "0.08" --> OP["Alarm<br/>0.98 x 0.08 = 0.0784"]
    OK -- "0.92" --> ON["No alarm<br/>0.9016"]

A tree diagram: multiply along a branch, add across branches that end in the same outcome.

Bayes' theorem

Now reverse the conditioning. Since P(A and B) can be written two ways, P(A | B) P(B) = P(B | A) P(A), so

P(A | B) = P(B | A) P(A) / P(B)

and with a partition in the denominator (the law of total probability),

P(Bi | A) = P(A | Bi) P(Bi) / (P(A | B1) P(B1) + ... + P(A | Bn) P(Bn))

The vocabulary is worth learning. P(Bi) is the prior: your belief before the evidence. P(A | Bi) is the likelihood: how probable the evidence is under each case. P(Bi | A) is the posterior: the updated belief.

A fault-diagnosis example

Suppose 2% of manufactured units are faulty. An automatic test raises an alarm for 95% of faulty units (its sensitivity) but also for 8% of healthy units (its false-alarm rate). A unit triggers the alarm. How likely is it faulty?

Python Bayes' theorem for a diagnostic test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A 95%-sensitive test gives a posterior of only about 0.195: roughly four of every five alarms are false. The counts show why. Of 100,000 units, 2,000 are faulty and 1,900 of them alarm, but 98,000 are healthy and 8% of them, 7,840 units, alarm too. The healthy units are so common that even a small false-alarm rate swamps the true alarms. Ignoring the prior (the base rate) is the classic mistake, called the base-rate fallacy. A silent test is very reassuring, though: P(faulty | no alarm) is about 0.0011.

Several causes: which machine made the defect?

Bayes works for any number of cases. Three machines produce 50%, 30% and 20% of the parts, with defect rates of 1%, 2% and 5%. The total defect rate comes from the law of total probability, and each machine's share of the blame from Bayes. The same code also shows sequential updating: yesterday's posterior becomes today's prior when a new, independent test result arrives.

Python Which machine, and updating after more evidence

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Machine C makes only 20% of the parts but is responsible for about 47.6% of the defects. Repeated independent positive tests quickly push a doubtful 19.5% to 74% and then to 97%. Note the word independent: if the second test shares a failure mode with the first (the same flaky sensor, say), the second result carries much less new information, and multiplying likelihoods again overstates your confidence.

Checking by simulation

When a problem has too many branches for a clean tree, simulate it. Generate a million units, apply the test, and count. The simulated conditional probability is just the fraction among the units where the condition holds.

Python Estimate P(faulty | alarm) by simulation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The simulated values, about 0.0974, 0.1956 and 0.00111, match the exact calculation to the accuracy a million samples allows. Seeded generators make the run reproducible, as in the previous lesson.

Common slips

  • Confusing P(A | B) with P(B | A). "95% of faulty units alarm" says nothing by itself about how many alarms are faulty units.
  • Forgetting the prior. A rare condition needs a very small false-alarm rate before a positive test means much.
  • Treating dependent evidence as independent when chaining updates.
  • Using a partition that is not exhaustive: the denominators then fail to add up to 1. Check that the posteriors sum to 1.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does P(A | B) mean?

    P(A | B) is the share of the outcomes in B that are also in A, which equals P(A and B) / P(B).
  2. 2. A test has 99% sensitivity and a 5% false-alarm rate. The condition affects 0.1% of the population. Which statement is most accurate?

    Per 100,000 people there are 99 true positives and about 4,995 false positives, so the posterior is about 2%. The low base rate dominates.
  3. 3. Which law gives P(A) as a weighted sum of P(A | Bi) P(Bi) over a partition?

    The law of total probability splits an event across exhaustive, mutually exclusive cases and weights each case by how common it is.
  4. 4. If A and B are independent and P(B) > 0, what is P(A | B)?

    Independence means that learning B does not change the probability of A, so P(A | B) = P(A).

Summary

  • P(A | B) = P(A and B) / P(B); multiplying along a tree branch uses P(A and B) = P(A | B) P(B).
  • The law of total probability averages case probabilities over a partition.
  • Bayes' theorem turns a likelihood and a prior into a posterior; counting outcomes in a large imaginary population makes it concrete.
  • A positive result from a good test on a rare condition can still be mostly false alarms (the base-rate fallacy).
  • Posteriors can be updated sequentially only when the pieces of evidence are independent given the cause.

Next: Random Variables, Covariance and Correlation.