Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 38 ยท Optimization

Curve Fitting and Least Squares

Fit nonlinear models to noisy data with curve_fit and least_squares. Learn residuals, parameter errors from the covariance matrix, weights and robust loss.

You have measurements and a model with unknown parameters: a decaying signal a * exp(-k * t) + c, a bell curve, a growth law. Curve fitting finds the parameter values that make the model agree best with the data. "Best" almost always means the smallest sum of squared residuals, where a residual is the difference between a measured value and the model's prediction.

In Least Squares and Pseudo-inverses the model was linear in its parameters, so one matrix solve was enough. Here the parameters sit inside functions like exp, so the solver must iterate from a starting guess. SciPy offers two layers: curve_fit, a friendly wrapper where you pass the model and the data, and least_squares, the engine underneath, where you write the residuals yourself and get control over loss functions and bounds.

curve_fit: the everyday tool

The recipe has three steps. Write a model f(x, *params) whose first argument is the independent variable. Call popt, pcov = curve_fit(f, x, y, p0=...). Then use popt (the best parameters) and the covariance matrix pcov (their uncertainty).

flowchart LR
    D["data (x, y)"] --> C["curve_fit(model, x, y, p0)"]
    M["model f(x, a, b, ...)"] --> C
    P0["initial guess p0"] --> C
    C --> POPT["popt:<br/>best parameters"]
    C --> PCOV["pcov:<br/>covariance matrix"]
    PCOV --> E["errors = sqrt(diag(pcov))"]

curve_fit returns the parameters and their covariance; the square roots of the diagonal are the standard errors.

Let us fit an exponential decay with an offset to noisy data we generate ourselves, so we know the true answer. The plot has a second panel with the residuals, which is the first thing to inspect after any fit.

Python Fit an exponential decay

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The recovered values land within about one or two standard errors of the truth, which is what honest uncertainty looks like. The noise we added had a standard deviation of 0.1, and the residual spread comes out close to it. In the lower panel, good residuals scatter randomly around zero. A visible curve or trend in them means the model is wrong, however nice the top plot looks.

Reading pcov

pcov is the estimated covariance matrix of the parameters. Its diagonal holds the variances, so np.sqrt(np.diag(pcov)) gives one-sigma errors. Off-diagonal entries show how parameters trade off against each other: a large correlation between a and k means many (a, k) pairs fit almost equally well. Converting to correlations is a one-liner: pcov / np.outer(perr, perr).

By default curve_fit treats the scatter of the residuals as the noise level and scales pcov accordingly. If you know the true measurement errors, pass them as sigma and set absolute_sigma=True so the covariance reflects them.

Weights: not all points are equally trustworthy

Often the noise is not the same everywhere. Instruments get worse at large readings, or some samples come from a handful of observations and others from thousands. The sigma argument gives the standard deviation of each y value. The fit then weights each residual by 1 / sigma, so noisy points count for less.

Python Weighted fitting with sigma

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last line computes the reduced chi-square: the weighted sum of squared residuals divided by the degrees of freedom (points minus parameters). When your model is right and your sigma values are honest, it should land near 1. A value far above 1 suggests a poor model or underestimated errors; far below 1 suggests overestimated errors. With only ten points it fluctuates quite a bit, so do not over-interpret one run.

Choosing a good starting guess

Iterative fitting can fail if it starts in the wrong place. Without p0, curve_fit sets every parameter to 1, which is often poor. Make your own guess from the data: the peak height for an amplitude, the x of the maximum for a center, the half-width for a spread. If you know physically sensible ranges, pass bounds=([low...], [high...]).

Python A bad start can still 'succeed'

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The bad start, with a tiny width far from the peak, does not raise an error. It happily returns a nonsense curve (a narrow spike at the edge, fitted to noise). That is the dangerous failure: no exception, just a wrong answer. Always plot the fit. Sensible bounds keep parameters like the width positive and in range, which also helps the solver; note that passing bounds switches curve_fit to a different method (trf) than the unbounded default (lm).

least_squares: full control

curve_fit is a wrapper. scipy.optimize.least_squares takes a function that returns the vector of residuals model - data and minimizes the sum of their squares. Use it when you need robust loss functions, want the Jacobian, have residuals that are not simply y - f(x) (for example fitting several datasets together), or need a particular method.

The signature is least_squares(fun, x0, args=(...), bounds=..., loss=..., f_scale=...), with fun(params, *args) returning an array. The result carries x (parameters), cost (half the sum of squared residuals), fun (the residuals), jac, nfev and success.

Robust loss against outliers

Squaring residuals means that one bad point can pull the whole curve towards it. least_squares can swap the squared loss for one that grows more slowly for large residuals: soft_l1, huber, cauchy or arctan. The parameter f_scale says roughly how large a residual must be before it counts as "large".

Python Robust loss versus plain least squares

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The plain fit is dragged off by the three outliers, notably in the amplitude. The robust fit largely ignores them and lands almost on the truth. The Jacobian has one row per data point and one column per parameter (30 by 2 here). The price of robust loss is that the textbook error formulas no longer apply exactly, so treat reported uncertainties more cautiously.

Parameter errors from the Jacobian

least_squares does not hand you a covariance matrix, but you can build one from the Jacobian J at the solution. For plain squared loss, cov โ‰ˆ inv(J.T @ J) * sยฒ, where sยฒ is the residual variance estimate, 2 * cost / (n - p). This is exactly what curve_fit does internally.

Python Standard errors from the Jacobian

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two routes agree. In practice: reach for curve_fit for standard fits and for least_squares when you need robust loss, custom residuals, or extra control.

Errors in both variables: orthogonal distance regression

curve_fit assumes that only y is noisy and that x is exact. If your x values are also measured (a thermometer reading plotted against a pressure reading, both with error), ordinary least squares is biased: it systematically flattens the slope, a phenomenon called regression dilution. Orthogonal distance regression (ODR) minimizes the distance from each point to the curve in both directions, weighted by the measurement errors in x and y.

The module is scipy.odr. You wrap the data in RealData(x, y, sx=..., sy=...) (the standard deviations of x and y), the model in Model(f) where f(beta, x) takes the parameter vector first, and run ODR(data, model, beta0=[...]).run(). The output has beta (fitted parameters), sd_beta (their standard errors) and res_var (the residual variance).

scipy.odr is deprecated since SciPy 1.17 and scheduled for removal in 1.19, because it wraps an old Fortran library (ODRPACK) that is now maintained as the separate package odrpack. It still runs in the SciPy 1.18 used here (with a DeprecationWarning), and you will meet it in existing code. For new projects install odrpack, or, for a quick answer, use the total least squares trick shown at the end of this section.

Python OLS versus ODR when x is noisy too

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Ordinary least squares gives a slope clearly below the true 2.0 (about 1.8, flattened by the noise in x), while ODR and total least squares land close to 2. Use ODR when both variables have comparable errors and you know their size; if x is controlled precisely, plain curve_fit is the right and simpler tool. If outliers rather than errors in x are your problem, use the robust loss functions of least_squares shown earlier.

A checklist for trustworthy fits

  1. Plot data and fit, then plot the residuals. Structure in the residuals means a wrong model.
  2. Give a reasonable p0 and, where possible, bounds.
  3. Report parameter errors from pcov, not just values.
  4. Check correlations. Highly correlated parameters are poorly determined individually.
  5. Do not extrapolate far beyond the range of the data; a model that fits well can still diverge outside it.
  6. Prefer simple models. Extra parameters always reduce residuals, even when they only fit noise.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does np.sqrt(np.diag(pcov)) give you after curve_fit?

    The diagonal of the covariance matrix holds the variances of the parameters, so its square root gives their standard errors.
  2. 2. Your data points have different, known uncertainties. What should you pass to curve_fit?

    The sigma argument weights each residual by 1/sigma, so noisy points count for less. absolute_sigma=True treats the values as real standard deviations when computing pcov.
  3. 3. A few gross outliers drag your fitted curve away from the bulk of the data. Which tool handles this best?

    Robust losses grow more slowly than the square for large residuals, so outliers have less influence on the parameters.
  4. 4. Which sign indicates the model is probably wrong, even when the curve looks close to the data?

    If the model were right, residuals would be random noise. A trend or wave in them reveals structure the model misses.

Summary

  • curve_fit(model, x, y, p0=...) returns the best parameters popt and the covariance pcov; standard errors are np.sqrt(np.diag(pcov)).
  • A residual is data minus model; fits minimize the sum of squared residuals. Always plot residuals.
  • Use sigma (and absolute_sigma=True) to weight points by their uncertainty; the reduced chi-square near 1 signals a consistent fit.
  • A poor p0 can give nonsense without an error. Choose starting values from the data and use bounds.
  • least_squares works on a residual function and offers robust losses (soft_l1, huber, cauchy), bounds and the Jacobian.
  • Errors can be rebuilt from the Jacobian: inv(J.T @ J) * sยฒ.

Next up: Linear and Integer Programming.