Skip to content
Tutorials Center
My learning

Topics

Lesson 58 of 68 ยท Probability and Statistics for Engineers

Statistics, Curve Fitting and Uncertainty

Derive least squares with calculus, fit lines and curves, build confidence intervals and propagate measurement uncertainty with derivatives in Python.

Every measurement has an error, and every model has parameters that must be estimated from imperfect data. Statistics answers three practical questions: What are the best-fit parameters? How uncertain are they? And how does the uncertainty in my inputs spread into the quantity I actually care about? Calculus supplies the machinery for all three: least squares is an optimization problem, and uncertainty propagation is the chain rule in disguise.

This lesson follows on from Probability and Random Variables. It also reuses the optimization ideas from Optimization in Several Variables and the partial derivatives of Partial Derivatives.

First, describe the data: scales and summary statistics

Before fitting anything, look at what kind of data you have. The scale of measurement decides which summaries mean something:

Scale What it tells you Example Sensible summaries
Nominal category only error code, sensor model mode, counts
Ordinal category with an order severity: low / medium / high mode, median
Interval equal steps, arbitrary zero temperature in degrees Celsius mean, standard deviation
Ratio equal steps, true zero length, mass, current, time all of these, plus ratios ("twice as long")

Averaging error codes is meaningless, and "40 degrees Celsius is twice as hot as 20" is false because Celsius has no true zero, while "40 ohms is twice 20 ohms" is true. Data about one quantity are univariate; data recorded as pairs (temperature and drift, load and deflection) are bivariate, and you describe the relationship between the two with covariance and correlation, as in Random Variables, Covariance and Correlation.

The standard summaries of a numeric sample x1, ..., xn are:

  • Mean x_bar = (x1 + ... + xn) / n: the balance point. It uses every value, so one outlier drags it.
  • Median: the middle value of the sorted data. It ignores the size of the extremes, so it is robust.
  • Mode: the most frequent value; the only average defined for nominal data.
  • Variance s**2 = sum of (xi - x_bar)**2 / (n - 1) and standard deviation s. Dividing by n - 1 (not n) corrects the bias that comes from using the sample mean in place of the unknown true mean; in numpy this is ddof=1.
  • Range and the interquartile range (IQR, the distance between the 25th and 75th percentiles): spread measures, the second one robust.
Python Mean versus median, and the n-1 variance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The single slow response of 480 ms pulls the mean up to 138.3 ms while the median stays at 100.5 ms, and without the outlier the mean falls back to 100.33. The sample standard deviation (120 ms) is just as distorted, while the IQR of 4.5 ms still describes the typical spread. For skewed data or data with outliers, report the median and IQR, or both sets of summaries. The bivariate pair has a covariance of 10.15 and a correlation of 0.9926: drift rises almost linearly with temperature, which sets up the least-squares fit that follows.

Least squares is calculus

Suppose you measured y at several values of x and believe y = a + b x plus noise. For any candidate (a, b), each point has a residual y_i - a - b x_i. The least-squares line minimizes the sum of squared residuals

S(a, b) = sum of (y_i - a - b x_i)**2

S is a smooth function of two variables, so at the minimum both partial derivatives vanish:

  • dS/da = -2 sum (y_i - a - b x_i) = 0
  • dS/db = -2 sum x_i (y_i - a - b x_i) = 0

These are the normal equations. They say the residuals add up to zero and are uncorrelated with x. Solving them gives b = (n Sxy - Sx Sy) / (n Sxx - Sx**2) and a = (Sy - b Sx) / n, where the S terms are sums.

flowchart LR
    A["Data (x, y)"] --> B["Choose model<br/>y = a + b x"]
    B --> C["Sum of squared residuals S(a, b)"]
    C --> D["Set dS/da = 0 and dS/db = 0"]
    D --> E["Best-fit a and b"]
    E --> F["Check residuals,<br/>estimate uncertainty"]

Least squares: define the error, differentiate, set the derivatives to zero.

Python Least squares from the normal equations

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The data were generated from the true line y = 2 + 1.5 x, and the fit recovers an intercept of about 1.68 and a slope of about 1.54: close, but not equal, because of the noise. The gradient of S is zero at the solution (as it must be at a minimum), the residuals sum to zero, and moving the intercept by 0.5 raises S by exactly 12 * 0.25 = 3, since the curve is a parabola in a with curvature 2n.

Why squares? They make S smooth and differentiable, they give a unique closed-form answer for linear models, and they are the maximum-likelihood choice when the noise is normal. The cost is sensitivity to outliers, which a squared penalty weighs heavily.

How uncertain is the fit?

A fitted slope is itself a random quantity: with another set of measurements you would get a slightly different slope. The standard error estimates the typical size of that variation. It grows with the noise in the data and shrinks with more points and a wider spread of x.

A confidence interval turns it into a range. For a 95% interval on a fitted parameter or a sample mean, you take the estimate plus or minus t * (standard error), where t comes from the Student t distribution with the right degrees of freedom (the number of points minus the number of fitted parameters). The t distribution is slightly wider than the normal when you have few points; with many points it approaches the 1.96 you may remember.

What "95%" means is a statement about the procedure: if you repeated the experiment many times and built an interval each time, about 95% of those intervals would contain the true value. It is not the probability that this particular interval contains it.

Python Regression with standard errors and confidence intervals

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The slope is 1.545 with a standard error of about 0.062, so its 95% interval is roughly 1.41 to 1.68. The true slope, 1.5, lies inside. The interval for the mean of the eight repeated measurements is about 9.987 to 10.036. The last experiment checks the meaning of "95%": over 2000 repeated experiments, the intervals contained the true mean 10 about 95% of the time (the exact count depends on the random seed).

R squared measures the fraction of the variation in y explained by the model. A high value does not prove the model is right. Always plot the residuals: a curved pattern means the model is missing something, and a funnel shape means the noise is not constant.

Fitting a curve

Not every model is a line, and many are nonlinear in their parameters. For example, a cooling object follows T(t) = T_env + (T_0 - T_env) exp(-t / tau). There are no closed-form normal equations here, so scipy.optimize.curve_fit minimizes S iteratively, in the spirit of the gradient methods you will see in Project: Gradient Descent from Calculus. It also returns a covariance matrix of the parameters: the square roots of its diagonal are their standard errors.

Give curve_fit a sensible starting guess p0; a poor one can lead to a bad local minimum or a failure to converge.

Python A nonlinear fit with parameter uncertainties

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All three fitted parameters land within about one standard error of the values used to generate the data (the noise had a standard deviation of 0.8, and the residual standard deviation comes out near that). That agreement is a useful sanity check: when the reported errors are consistent with the known truth, you can trust them on real data where the truth is unknown.

Error propagation with derivatives

Often you do not measure the quantity you want directly. You measure voltage and resistance and want power, P = V**2 / R. If V and R each carry an uncertainty, how uncertain is P?

For a small error, the linear approximation from Linearization and Newton's Method says dP ~ (dP/dV) dV + (dP/dR) dR. For independent errors with standard deviations sigma_V and sigma_R, the contributions add in quadrature:

sigma_P = sqrt((dP/dV * sigma_V)**2 + (dP/dR * sigma_R)**2)

The partial derivatives are the sensitivities: they say which input matters most. A Monte Carlo simulation (sample the inputs, compute P each time, look at the spread) checks the formula without any calculus, and is the fallback when the function is strongly nonlinear.

Python Propagating uncertainty with sympy and checking by simulation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The power is about 3.064 W. The voltage term contributes about 0.0511 W and the resistance term about 0.0326 W (with opposite sign: a larger R lowers P), so the combined uncertainty is about 0.061 W, and the simulation agrees. The voltage matters more even though its relative error (0.8%) is smaller than that of the resistance (1.1%), because V is squared and so its relative error counts twice. A rule of thumb for products and quotients: relative errors add in quadrature, each multiplied by the absolute value of its exponent.

Good habits

  • Plot the data and the residuals before and after fitting.
  • Report a result with its uncertainty and units: 3.06 +/- 0.06 W.
  • Do not extrapolate far beyond the data range.
  • Beware of correlated errors: the quadrature formula assumes independence.
  • Use more data points, or a wider range of x, to shrink the standard errors.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What condition defines the least-squares line mathematically?

    S(a, b) is minimized where dS/da = dS/db = 0, which gives the normal equations.
  2. 2. What does a 95% confidence interval mean?

    The 95% refers to the long-run success rate of the interval-building procedure, not to a probability for one fixed interval.
  3. 3. For P = V**2 / R with independent small errors, how are the contributions of V and R combined?

    sigma_P = sqrt((dP/dV sigma_V)^2 + (dP/dR sigma_R)^2) for independent errors.
  4. 4. Why does curve_fit need a starting guess p0 for a nonlinear model?

    There is no closed-form solution, so the optimizer improves a guess step by step from where it starts.

Summary

  • Least squares minimizes the sum of squared residuals; setting the partial derivatives to zero gives the normal equations.
  • Standard errors and confidence intervals (estimate plus or minus t times standard error) describe how uncertain a fitted value is.
  • scipy.stats.linregress fits a line; scipy.optimize.curve_fit fits nonlinear models and returns a covariance matrix.
  • Uncertainty propagates through derivatives: sigma_f is the quadrature sum of (df/dx_i) * sigma_i; check it with Monte Carlo.
  • Always plot residuals, report uncertainties, and avoid extrapolating.

Next: Project: Projectile Motion with Air Drag.