Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 111 ยท Getting Started

Probability and Statistics for Machine Learning

Learn the probability behind machine learning: distributions, Bayes rule, maximum likelihood, the central limit theorem and the bootstrap, with NumPy and SciPy.

A model never sees the whole world. It sees a sample: a few thousand customers, a few hundred photos. Probability is the language for reasoning about what a sample can tell you, and statistics is the craft of going from the sample back to conclusions about the world. Almost every loss function in this course has a probabilistic reading: squared error comes from assuming Gaussian noise, log loss comes from assuming coin-flip labels, and Naive Bayes is Bayes' rule applied directly.

This lesson covers the five ideas you will use most: random variables and distributions, Bayes' rule, maximum likelihood, the central limit theorem and the bootstrap. Every idea is checked by simulation, because the best way to trust a formula is to watch it agree with a million random draws. The linear algebra and calculus side of the foundations is in Math for ML.

Random variables and distributions

A random variable is a quantity whose value comes from a random process: a person's height, the number of customers per hour, whether an email is spam. A distribution says how likely each value is. Three families cover a surprising amount of ground:

Distribution Models Parameters Mean Variance
Normal (Gaussian) A measurement with many small influences, such as height mean ฮผ, spread ฯƒ ฮผ ฯƒยฒ
Binomial Number of successes in n yes/no trials n, success chance p nยทp nยทpยท(1โˆ’p)
Poisson Count of events in a fixed window rate ฮป ฮป ฮป

Discrete variables (counts) have a probability mass function, P(X = k). Continuous ones have a density, and you ask about ranges: P(X > 186) is an area under the curve, which is what the cumulative distribution function (cdf) gives you. SciPy's scipy.stats module has every common distribution with pdf/pmf, cdf and rvs (random samples).

Python Sampling from distributions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The simulated means and variances land on the theory column, and the exact probability from stats.binom.pmf matches the share of simulated draws. The last line is the famous 68-95-99.7 rule: about 68% of a normal variable falls within one standard deviation of the mean, 95% within two, and 99.7% within three. Remember it, because it is the quickest way to spot an outlier.

Conditional probability and Bayes' rule

P(A | B) means "the probability of A once you know B happened". It is not the same as P(B | A), and mixing them up is one of the most common reasoning errors in data work. Bayes' rule converts one into the other:

P(A | B) = P(B | A) ยท P(A) / P(B)

In machine learning terms, P(A) is the prior (what you believed before seeing data), P(B | A) is the likelihood (how well A explains the data), and P(A | B) is the posterior (your updated belief). The denominator is found by adding up all the ways B can happen.

The classic demonstration is a medical test, and it applies to any rare-event classifier such as fraud detection. Suppose 1% of people have a condition, the test catches 95% of real cases (sensitivity) and correctly clears 90% of healthy people (specificity). You test positive. How likely is it that you have the condition? Most people guess about 90%. Run the numbers.

Python Bayes' rule and a base-rate surprise

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only about 9% (0.088) of positive tests are real, because healthy people are so numerous that their 10% false-positive rate produces far more positives than the sick people's 95% hit rate. This is the base-rate effect, and it is why precision of a classifier collapses on rare classes even when accuracy looks great. The second block shows Bayesian updating: the posterior after one test becomes the prior for the next, and a second positive result lifts the probability from about 9% to about 48%. Naive Bayes does exactly this once per feature.

Maximum likelihood: choosing parameters

How do you pick the parameters of a distribution, or the weights of a model? Maximum likelihood estimation (MLE) gives a principled answer: choose the parameters under which the data you actually saw was most probable. The likelihood is the product of the probabilities of each observed point; because products of many small numbers underflow, we use the log-likelihood, a sum, which has its peak in the same place.

For a normal distribution, the MLE of the mean is the sample mean and the MLE of the variance is the average squared deviation. For a coin, the MLE of the heads probability is the fraction of heads. These are the "obvious" answers, and MLE explains why they are obvious.

It also explains models. If you assume labels equal a linear function plus Gaussian noise, maximizing the likelihood is the same as minimizing squared error, which is linear regression. If you assume labels are Bernoulli coin flips with a probability from a sigmoid, maximizing the likelihood is the same as minimizing log loss, which is logistic regression. Loss functions are not arbitrary; they are negative log-likelihoods.

The next example finds the mean by brute force, scanning a grid and keeping the value with the highest log-likelihood, then compares it with the closed form and with scipy.stats.norm.fit.

Python Maximum likelihood by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The grid search, the formula and SciPy agree (up to the grid's resolution), and the Bernoulli scan peaks at the fraction of ones. Notice that the estimates are close to, but not exactly, the true values 5 and 0.37 we simulated from. That gap is sampling error, which the next two ideas quantify.

The central limit theorem

Take any distribution, however lopsided. Draw n values and average them. Repeat many times. The averages follow an approximately normal distribution centered on the true mean, with a spread (the standard error) of ฯƒ / โˆšn. This is the central limit theorem (CLT), and it is why the normal distribution is everywhere: averages and sums of many small effects look normal.

Two practical consequences follow. First, quadrupling your sample size only halves the uncertainty of an average, because of the square root. Second, the CLT is what justifies the confidence intervals and p-values you see in classical statistics, even when the raw data are not normal.

The bootstrap: uncertainty from one sample

What if you cannot derive a formula for the uncertainty of your statistic, such as a median or a model's accuracy? The bootstrap gives a general recipe. Treat your sample as a stand-in for the population: resample it with replacement many times to create fake new samples of the same size, compute your statistic on each, and look at how much it varies. The 2.5th and 97.5th percentiles of those values form a 95% bootstrap interval.

The example demonstrates both ideas on a deliberately skewed distribution, the exponential. One draw is very lopsided, yet the means of 100 draws are nearly symmetric and their spread matches 1/โˆšn.

Python Central limit theorem and a bootstrap interval

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Skewness (a measure of lopsidedness; 0 means symmetric) falls toward 0 as n grows, and the standard deviation of the means follows the 1/โˆšn curve. The bootstrap interval reports how far the sample mean could plausibly be from the true mean (which is 1.0 here), using nothing but the 40 numbers. Later in the course the same trick, applied to a model's score, gives a confidence band around a cross-validation result.

A confidence interval is a statement about the procedure, not about this one interval: if you repeated the experiment many times, about 95% of intervals built this way would contain the true value. It is not "a 95% probability that the truth lies in this particular range".

Descriptive statistics: shape, spread and correlation

The ideas above describe populations. In day-to-day ML work you first describe your sample: where is it centered, how spread out is it, what shape does it have, and which columns move together?

  • Center: the mean is pulled by extreme values, the median is not. When they differ a lot, the data is skewed or has outliers.
  • Spread: the standard deviation measures typical distance from the mean; the IQR (the gap between the 25th and 75th percentiles) is its outlier-proof cousin. A common outlier rule flags values more than 1.5 IQRs beyond the quartiles.
  • Skewness measures lopsidedness. Zero means symmetric, a positive value means a long right tail (incomes, prices, waiting times), a negative value a long left tail. A log transform often tames a right-skewed column before a linear model.
  • Kurtosis measures how heavy the tails are compared with a normal distribution. SciPy reports excess kurtosis, which is 0 for a normal. A large positive value means more extreme values than a bell curve predicts, so squared-error models will be pulled around by them.
  • Covariance says whether two columns rise together, but its size depends on the units. Correlation rescales it to the range -1 to 1, so you can compare columns.

Three correlation coefficients cover most needs. Pearson measures linear association. Spearman works on ranks, so it measures any monotonic relationship (always rising or always falling), and it is robust to outliers. Kendall's tau also uses ranks and is better behaved on small samples with ties. None of them can see a relationship that goes up and then down.

Python Skewness, kurtosis and three kinds of correlation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The symmetric column has skew and excess kurtosis near 0. The log-normal column has skew 2.65 and a mean (24) well above its median (20), and the heavy-tailed column has the same mean and median but an excess kurtosis of 16.6, which is the numeric sign of its extreme values. In the correlation table, Pearson is highest for the straight-line relationship, Spearman beats Pearson on the exponential curve (0.88 against 0.82) because the curve is monotonic but not a line, and all three coefficients are near zero for the U shape, even though y is almost entirely determined by x. A near-zero correlation means "no linear or monotonic trend", never "no relationship". The last two lines confirm that multiplying a column by 100 multiplies its covariance by 100 and leaves its correlation untouched.

Inferential statistics: from a sample to a claim

Inference asks whether what you see in a sample is real or could be chance. The standard toolkit has four pieces.

  • A confidence interval gives a range of plausible values for a quantity such as a mean. For small samples, use the t distribution rather than the normal. Remember what it means (see the note above): the procedure covers the truth about 95% of the time, so some intervals miss.
  • A hypothesis test starts from a null hypothesis H0 (no difference, no effect) and asks how surprising your data would be if it were true. The p-value is the probability of seeing a result at least this extreme when H0 holds. A small p-value is evidence against H0. It is not the probability that H0 is true, and it says nothing about the size of an effect.
  • Which test? For two group means, use the two-sample (Welch) t-test. For two measurements on the same units (such as two models scored on the same folds), use the paired t-test. For counts in categories (clicked or not, by device), use the chi-square test of independence.
  • Multiple testing: test 100 things and about 5 will come out "significant" at 0.05 by luck. Divide the threshold by the number of tests (Bonferroni), or use a less strict correction, whenever you look at many features or many models.

Power is the chance that a test detects a real effect. Small samples have low power, so a failure to find a difference is weak evidence of no difference.

Python Confidence interval, t-test, chi-square, multiple testing and power

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first interval (89.1 to 98.5) misses the true mean of 100. That is not a bug: about 1 interval in 20 does, which is the real meaning of "95% confidence". The t-test and chi-square test both report p-values below 0.05 for effects we planted (a 5-point shift, and click rates of 13% against 17%). The paired test on the folds is the standard way to compare two models scored on the same splits, though cross-validation folds share training data, so treat that p-value as a rough guide and prefer repeated runs for important decisions. The multiple-testing block shows 8 false alarms out of 100 tests on pure noise, and none after Bonferroni. The power block shows why sample size matters: a medium effect (half a standard deviation) is caught only 37% of the time with 20 samples per group, 78% with 64, and almost always with 200.

For machine learning, these tools answer practical questions: "is the new model really better than the old one?", "did the A/B test change click-through?", "is this feature related to the target, or is it noise?". Treat a p-value as one input to a decision, alongside the effect size and its cost, never as a verdict.

How these ideas connect to machine learning

  • Train/test split is sampling; the test score is a random variable with its own standard error, so a difference of 0.2% between two models is often just noise.
  • Priors become regularization: a Gaussian prior on weights adds an L2 penalty to the likelihood.
  • Independence is what Naive Bayes assumes, and what cross-validation for time series needs you to question.
  • Distribution shift (training and production data come from different distributions) is the statistical name for drift.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A test is 95% sensitive and 90% specific, and the condition affects 1% of people. Why is the chance of having the condition after one positive test only about 9%?

    With a 1% base rate, 99% of people are healthy and 10% of them test positive, which is many more positives than the 95% of the 1% who are sick. This is the base-rate effect.
  2. 2. What does maximum likelihood estimation choose?

    MLE picks the parameter values under which the data you saw has the highest probability. With Gaussian noise this is least squares, and with Bernoulli labels it is log loss.
  3. 3. You quadruple the size of your sample. By what factor does the standard error of the mean shrink?

    The standard error is sigma divided by the square root of n. Quadrupling n divides it by sqrt(4) = 2.
  4. 4. What does a bootstrap do?

    Each resample plays the role of a new data set. The spread of the statistic across many resamples estimates its uncertainty.

Summary

  • Random variables have distributions; the normal, binomial and Poisson cover most everyday cases, and scipy.stats provides pdf/pmf, cdf and rvs.
  • Bayes' rule turns P(B | A) into P(A | B): posterior = likelihood ร— prior / evidence. Base rates matter enormously for rare events.
  • Maximum likelihood picks the parameters that make the data most probable; squared error and log loss are negative log-likelihoods in disguise.
  • The central limit theorem: averages of many draws are approximately normal with standard error ฯƒ/โˆšn.
  • The bootstrap estimates the uncertainty of almost any statistic by resampling your own data.
  • Simulate whenever you doubt a formula; a few lines of NumPy settle most arguments.

Continue with a different kind of learning in Association Rule Mining, or return to the main path at Linear Regression.