Skip to content
Tutorials Center
My learning

Topics

Lesson 52 of 68 ยท Probability and Statistics for Engineers

Probability and Random Variables

See probability as calculus: PDFs and CDFs as integrals, expectation and variance, the normal and exponential distributions, and scipy.stats in Python.

Real systems are noisy. A machine shop never produces two bolts with exactly the same diameter, a network packet takes a slightly different time every trip, and a sensor reading wobbles around the truth. Engineers cannot predict individual values, but they can describe the pattern: how likely each value is. That description is a probability distribution, and for continuous quantities the tool that builds it is the integral.

This lesson treats a continuous random variable as a quantity whose behaviour is described by a function, and shows that everything you need (probabilities, averages, spread) comes from integrals you already know from The Integral as Accumulated Change and Improper Integrals and Convergence.

PDF and CDF: density and accumulation

A continuous random variable X has a probability density function (PDF) f(x). The density is not a probability itself. The probability that X falls between a and b is the area under the density:

P(a <= X <= b) = integral of f(x) dx from a to b

Two rules follow. The density is never negative, and the total area is 1, because something must happen. Because a single point has zero width, P(X = c) = 0 for any exact value c; only intervals have probability.

The cumulative distribution function (CDF) is the running total, F(x) = P(X <= x), which is the integral of f from minus infinity up to x. By the Fundamental Theorem of Calculus, F'(x) = f(x): the PDF is the derivative of the CDF. And P(a <= X <= b) = F(b) - F(a), the same net-change idea as in The Fundamental Theorem of Calculus.

flowchart LR
    A["PDF f(x)<br/>density"] -- "integrate" --> B["CDF F(x)<br/>P(X <= x)"]
    B -- "differentiate" --> A
    B --> C["P(a <= X <= b)<br/>= F(b) - F(a)"]
    A --> D["E[X] = integral of x f(x) dx"]

The PDF and CDF are linked by integration and differentiation.

Suppose the time to failure T (in hours) of a component has density f(t) = (t / 2500) exp(-t**2 / 5000) for t >= 0. We can check everything with scipy.integrate.quad.

Python Probabilities, mean and variance as integrals

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The total area is 1, the chance of failing within 100 hours is about 0.8647, and the chance of failing between 50 and 100 hours is about 0.4712. The CDF here has a closed form, F(t) = 1 - exp(-t**2/5000), which is why the "exact" column matches.

Expectation and variance

The expected value (mean) is the density-weighted average of the values:

E[X] = integral of x f(x) dx

It is the balance point of the density, just like the centre of mass in Applications of Integrals with density playing the role of mass. More generally E[g(X)] = integral of g(x) f(x) dx, which is how you average any function of the variable (energy, cost, power).

The variance measures spread: Var[X] = E[(X - mu)**2] = E[X**2] - mu**2. Its square root is the standard deviation sigma, in the same units as X. In the example above, E[T] is about 62.67 hours and Var[T] about 1073, a standard deviation of roughly 32.8 hours. Two useful rules: E[aX + b] = a E[X] + b, and Var[aX + b] = a**2 Var[X]. For independent variables, the means add and so do the variances.

The normal distribution

The normal (Gaussian) distribution has density

f(x) = exp(-(x - mu)**2 / (2 sigma**2)) / (sigma sqrt(2 pi))

It is symmetric around the mean mu with spread sigma. Its CDF has no elementary closed form, since the integral of exp(-x**2) is not an elementary function (see the Gaussian integral in the improper-integrals lesson), so you use tables or software. It matters everywhere because of the central limit theorem: sums and averages of many small independent effects look normal, whatever the individual effects look like. Measurement noise and manufacturing variation are typical examples.

The 68-95-99.7 rule says about 68% of the probability lies within one standard deviation of the mean, 95% within two, and 99.7% within three.

The exponential distribution

The exponential distribution models the waiting time between independent random events (customer arrivals, radioactive decays, failures with a constant failure rate). With mean 1/lambda, its density is lambda exp(-lambda x) for x >= 0, and its CDF is 1 - exp(-lambda x). It has a striking property: it is memoryless. If you have already waited 4 minutes, the distribution of the remaining wait is the same as for a fresh start. In formulas, P(X > s + t | X > s) = P(X > t).

scipy.stats: the practical interface

scipy.stats wraps these (and many more) distributions in objects with the same methods: pdf, cdf, sf (survival function, 1 - cdf, more accurate in the tail), ppf (the inverse CDF, or quantile), mean, var, and rvs (random samples). Note that scipy parametrizes with loc (shift) and scale, so a normal with mean 10 and standard deviation 0.05 is stats.norm(loc=10, scale=0.05), and an exponential with mean 4 is stats.expon(scale=4).

Python Normal and exponential with scipy.stats

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For the bolts, the specification limits 9.9 to 10.1 mm are two standard deviations from the mean, so about 95.45% of bolts pass and about 5.5% are above 10.08 mm. The 95th percentile is about 10.082 mm. For the arrivals, the chance of waiting more than 6 minutes is about 0.2231, and the conditional probability of waiting 6 more minutes after already waiting 4 gives the identical number: that is memorylessness in numbers.

Seeing and simulating

A picture helps to hold the PDF/CDF relation in mind: the shaded area under the density up to x is the height of the CDF at x. Simulation (Monte Carlo) is the other practical tool. When combining several random effects, such as tolerances in an assembly, the exact distribution may be hard to derive, but sampling is easy. A seeded generator keeps the run reproducible.

Python Density versus CDF, and a tolerance stack-up by simulation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each uniform part has variance 0.2**2 / 12, so three independent parts have three times that, giving a standard deviation of 0.1 mm. The simulation reproduces it. The sum of three flat distributions is already nearly bell-shaped, so the normal approximation (about 0.9545) lands close to the simulated probability (about 0.958). This is the central limit theorem at work in a design calculation.

Common slips

  • A density can be larger than 1 (the normal pdf with sigma = 0.05 peaks near 8). Only the area is bounded by 1.
  • P(X = c) is 0 for a continuous variable; ask for intervals instead.
  • Variances add for independent variables; standard deviations do not.
  • Check the parametrization: scale in scipy for the exponential is the mean, not the rate.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How is the probability that a continuous X lies between a and b computed from its PDF?

    Probability is area under the density curve. Equivalently it is F(b) - F(a) for the CDF F.
  2. 2. What is the relationship between the PDF f and the CDF F?

    F(x) is the integral of f up to x, so by the Fundamental Theorem of Calculus F'(x) = f(x).
  3. 3. X is normal with mean 50 and standard deviation 2. Roughly what fraction of values lie between 46 and 54?

    46 and 54 are two standard deviations from the mean, and about 95% of a normal distribution lies within two standard deviations.
  4. 4. What does the memoryless property of the exponential distribution say?

    P(X > s + t | X > s) = P(X > t): past waiting gives no information about the future wait.

Summary

  • A PDF is a density; probabilities are areas, and the total area is 1. The CDF is its running integral, and the PDF is the CDF's derivative.
  • E[X] is the integral of x f(x); variance is E[X**2] - mu**2; independent variances add.
  • The normal distribution arises from sums of many small effects; the exponential models waiting times and is memoryless.
  • scipy.stats provides pdf, cdf, sf, ppf, mean, var and rvs for dozens of distributions.
  • Simulation with a seeded generator is a quick way to check a probability calculation.

Next: Conditional Probability and Bayes' Theorem.