Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 39 ยท Exploration and Visualization

Probability Distributions for Data Mining

Meet the distributions that data mining uses: normal, binomial, Poisson, exponential, uniform, lognormal and more, with simulations and fits in scipy.

A probability distribution describes how likely each value of an attribute is. Many mining techniques quietly assume one: z-scores and Gaussian anomaly detectors assume a normal shape, Naive Bayes models each attribute with a distribution, and time between events in a log often follows an exponential law. Knowing the common distributions helps you choose transformations, tests and models. This lesson tours the ones you will meet most, with a quick simulation of each.

Discrete versus continuous

  • A discrete variable takes countable values and has a probability mass function (PMF).
  • A continuous variable takes any value in a range and has a probability density function (PDF); probability is the area under the curve.
  • Both have a cumulative distribution function (CDF): P(X <= x).
flowchart TD
    D["Distributions"] --> Di["Discrete"]
    D --> Co["Continuous"]
    Di --> B["Bernoulli, Binomial"]
    Di --> P["Poisson"]
    Co --> N["Normal, Lognormal"]
    Co --> E["Exponential, Gamma"]
    Co --> U["Uniform, Beta"]

Bernoulli and binomial

A Bernoulli variable is one yes/no trial with success probability p (a customer buys or not). The binomial distribution counts successes in n independent trials: how many of 20 visitors buy?

Python Binomial probabilities

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Poisson

The Poisson distribution counts events in a fixed interval when they happen independently at a constant average rate lambda: calls per hour, errors per page, arrivals per minute. Its mean and variance both equal lambda.

Python Poisson: events per hour

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If the variance is much larger than the mean in real count data (overdispersion), plain Poisson is wrong and a negative binomial fits better.

Normal (Gaussian)

The bell curve, defined by mean mu and standard deviation sigma. Sums and averages of many small independent effects tend toward it (the central limit theorem), which is why heights, measurement errors and sample means look normal. The 68-95-99.7 rule: about 68 percent of values lie within 1 sigma of the mean, 95 percent within 2, 99.7 percent within 3. The z-score (x - mu) / sigma says how many sigmas a value is from the mean and is the basis of simple outlier detection.

Python The 68-95-99.7 rule

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The central limit theorem

Python Averages of skewed data become normal

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Uniform

All values in a range are equally likely. Random number generators produce it; a flat histogram is also a sign of identifiers or hashed values.

Exponential and gamma

The exponential distribution models the waiting time between Poisson events: minutes between customer arrivals, time until a machine fails. It is memoryless: the chance of waiting another 5 minutes does not depend on how long you have already waited. The gamma distribution generalizes it (the wait for the k-th event) and models positive, right-skewed values such as claim sizes.

Python Waiting times

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Lognormal

If log(X) is normal, X is lognormal: positive, right-skewed, with a long tail. Incomes, house prices, file sizes and many sensor readings behave this way. Taking logs turns it into a normal variable, which is why the log transform from Data Transformation works so well.

Python Lognormal becomes normal after log

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Beta and chi-squared

  • Beta: a distribution on [0, 1], ideal for probabilities and rates (a conversion rate you are unsure of). It is the building block of Bayesian A/B testing.
  • Chi-squared: the distribution of a sum of squared standard normals; it underlies the chi-square test and variance estimation. Also the t and F distributions come from combinations of these and drive the t-test and ANOVA.
Python Beta as belief about a rate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Fitting a distribution to data

Given a column, try candidate distributions, estimate their parameters (by maximum likelihood) and compare fit, for instance with a Kolmogorov-Smirnov test or by comparing a histogram with the fitted curve.

Python Which distribution fits?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The smallest KS distance marks the best fit; here the lognormal should win. When no standard distribution fits, use non-parametric methods such as a histogram or kernel density estimate, as many anomaly detectors do.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which distribution counts the number of buyers among 20 independent visitors?

    Binomial counts successes in a fixed number of independent yes/no trials.
  2. 2. Waiting time between independent random arrivals follows which distribution?

    If counts per interval are Poisson, waiting times between events are exponential.
  3. 3. About what share of a normal distribution lies within 2 standard deviations of the mean?

    The 68-95-99.7 rule: 95 percent within 2 sigma.
  4. 4. Incomes are right-skewed and positive. Which distribution is a common model?

    Lognormal data has a normal logarithm and a long right tail.
  5. 5. The central limit theorem says that:

    Sample means become approximately normal as the sample size grows, whatever the original shape.

Summary

  • Discrete variables have a PMF, continuous ones a PDF; both have a CDF.
  • Bernoulli and binomial for yes/no counts; Poisson for event counts; exponential and gamma for waiting times.
  • The normal distribution, the 68-95-99.7 rule and z-scores underlie much of statistics; the central limit theorem explains its ubiquity.
  • Lognormal fits skewed positive data; beta fits rates between 0 and 1.
  • Fit candidate distributions and compare with the KS statistic, or go non-parametric.

Next: Frequent Itemsets and Market Basket Analysis