Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 45 ยท Real-World Projects

Project: Fit a Model to Data

Build a linear regression in MATLAB and Octave: simulate data, solve the normal equations and backslash, compute R-squared, plot residuals and predict, with Python stand-ins.

In this project you fit a straight line to noisy measurements, step by step, the way an engineer or a data analyst would. You will build a data set with a known answer, set up the least-squares problem as a matrix equation, solve it two ways, judge the fit with R-squared and a residual plot, and finally use the model to predict a new value. Every step uses ideas from earlier lessons: matrices, the backslash operator, and polynomial fitting.

MATLAB and Octave are not available in your browser here, so the MATLAB code is read-only. Each step has a runnable Python stand-in using NumPy and matplotlib that does the same matrix operations. Because a seeded random generator gives different digits in MATLAB, Octave and NumPy, the numbers you see in the stand-ins will differ slightly from a real MATLAB session; the method and the shape of the results are the same.

The project in five steps

Make data A known line plus noise.
Fit Normal equations and backslash.
Judge R-squared and residuals.
Predict Use the model on new inputs.
1 / 2

Step 1: build data with a known answer

To check a method, test it on data whose true answer you know. Here the "true" relationship is y = 3 + 2x, and the measurements have random noise on top. Think of x as hours of study and y as an exam score, or x as temperature and y as sensor voltage.

MATLAB
rng(1)                          % repeatable noise
n = 40;
x = linspace(0, 10, n)';        % column vector of inputs
noise = 1.5 * randn(n, 1);
y = 3 + 2*x + noise;            % measurements
plot(x, y, 'o')

The noise has standard deviation 1.5, so we expect the fitted line to land close to a slope of 2 and an intercept of 3, but not exactly on them. That gap between the truth and the estimate is the nature of fitting noisy data.

Step 2: set up and solve the least-squares problem

A line is y = b0 + b1*x. For all n points at once, write it as a matrix equation X*b = y, where X has a column of ones (for the intercept b0) and a column of x values (for the slope b1):

MATLAB
X = [ones(n, 1), x];            % n-by-2 design matrix

There are 40 equations and only two unknowns, so no line passes through every point and the system has no exact solution. Least squares picks the b that makes the sum of squared residuals as small as possible. Setting the derivative to zero gives the normal equations:

MATLAB
b1 = (X' * X) \ (X' * y);       % normal equations
b2 = X \ y;                     % backslash: least squares directly
b3 = polyfit(x, y, 1);          % [slope intercept], note the reversed order

X'*X is a small 2-by-2 matrix, and the backslash solves the 2-by-2 system. Written with inv, inv(X'*X)*X'*y, the same formula works on paper but is less accurate and slower, so avoid it. For a rectangular X, X \ y does not use the normal equations at all: it uses a QR factorization, which is more accurate because squaring X in X'*X also squares its sensitivity to rounding errors (its condition number). On well-behaved data the two answers agree to many digits; on badly scaled columns the difference can matter. polyfit returns the coefficients highest power first, so b3(1) is the slope and b3(2) the intercept, the opposite order of b2.

Python Build the data and fit the line three ways

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 3: judge the fit with R-squared

How good is the fit? R-squared compares the model's leftover error with the error of the simplest possible model, "always predict the mean". It is the share of the variation in y that the line explains: 1 means a perfect fit, 0 means no better than the mean.

MATLAB
yfit = X * b2;                          % model predictions
res = y - yfit;                         % residuals
SSres = sum(res.^2);                    % what the model leaves unexplained
SStot = sum((y - mean(y)).^2);          % total variation around the mean
R2 = 1 - SSres / SStot                  % about 0.95 for this much noise
rmse = sqrt(mean(res.^2))               % typical error in y units, near 1.5

R-squared alone can mislead. It never decreases when you add more columns to X, even useless ones, and a high value does not prove the model is right. A curve fitted with a straight line can still reach an R2 of 0.9 or more, as the next TryIt shows. That is why we also look at the residuals.

Step 4: plot the residuals

If a straight line is the right model, the residuals should look like random scatter around zero, with no pattern and about the same spread everywhere. A visible curve, a funnel or a trend means the model is missing something.

MATLAB
plot(yfit, res, 'o')
yline(0)                                % zero reference line (R2022b+)
xlabel('fitted value'), ylabel('residual')

Octave has no yline in older versions; draw it with plot(xlim, [0 0], 'k-') or line. To see why the check matters, imagine the same procedure on data that really follows y = x.^2. The line has a respectable R-squared, but the residuals form a clear U shape, announcing that you need an x.^2 column.

Python R-squared, RMSE and a residual plot

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 5: predict, and know the limits

Once you have coefficients, a prediction is a dot product: the new point's row [1, x_new] times b. To predict many values, build the same kind of design matrix.

MATLAB
xnew = [2.5; 7.2; 12];
Xnew = [ones(size(xnew)), xnew];
ynew = Xnew * b2                % predictions
polyval(flipud(b2), xnew)       % the same with polyval (needs highest power first)

Predictions for x = 2.5 and x = 7.2 lie inside the range of the data (interpolation) and are trustworthy. The prediction for x = 12 lies outside it (extrapolation): the line may continue, but your data never showed that. Any error in the slope grows with distance, and real relationships often bend. Statistics Toolbox functions such as fitlm and predict also give confidence intervals; with the core you can get a rough interval from the RMSE (about two RMSE either side covers most points if the noise is bell-shaped).

You can add more columns to X for more complex models without changing the solving code. A quadratic needs [ones(n,1), x, x.^2]; two predictors need one column each. Adding columns is where the normal equations begin to lose accuracy, so on larger problems prefer X \ y.

Python Predict, extend the model and compare conditioning

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Putting it together as a function

When the steps work, wrap them in a function so you can reuse them, as covered in Scripts and Functions:

MATLAB
function [b, R2] = linfit(X, y)
    b = X \ y;
    res = y - X * b;
    R2 = 1 - sum(res.^2) / sum((y - mean(y)).^2);
end

Call it as [b, R2] = linfit([ones(n,1) x], y). Because X is an argument, the same function fits a line, a quadratic or any number of predictors.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the design matrix X = [ones(n,1), x] represent?

    Each row is [1, x_i], so X*[b0; b1] equals b0 + b1*x_i for every point at once.
  2. 2. For an n-by-2 matrix X with n > 2, what does X \ y compute?

    For overdetermined systems, backslash returns the least-squares solution (via QR), which is why it can be used directly for regression.
  3. 3. A model has R-squared = 0.95, but its residual plot shows a clear U shape. What should you conclude?

    Patterned residuals mean systematic error. A high R-squared can still hide it, so always inspect the residuals.
  4. 4. What does polyfit(x, y, 1) return?

    polyfit lists coefficients from the highest power down, so the slope comes first and the intercept second.
  5. 5. Why is predicting at x = 12 riskier than at x = 5 when the data covers 0 to 10?

    Inside the data you interpolate. Outside it you assume the relationship continues, and the uncertainty in the slope also grows with distance.

Summary

  • Test a method on data with a known answer: y = 3 + 2*x + noise with a seeded generator.
  • Set up X = [ones(n,1) x]; solve with the normal equations (X'*X)\(X'*y), with X \ y (preferred) or with polyfit.
  • R-squared is 1 - SSres/SStot, the share of variation explained; RMSE gives the typical error in y units.
  • Residuals should look like random scatter around zero; a pattern means the model is wrong, whatever R-squared says.
  • Predictions are Xnew * b; trust interpolation more than extrapolation.
  • More predictors or powers are just more columns in X, and backslash stays accurate where the normal equations lose digits.

Next: Project: Predator-Prey Simulation.