Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 40 ยท Regression

Linear Regression

Fit LinearRegression in scikit-learn, read coef_ and intercept_, predict, measure error with MAE, RMSE and R-squared, and check residuals on a real dataset.

Regression means predicting a number: a house price, tomorrow's temperature, the minutes a delivery will take. Linear regression is the simplest and most useful starting model. It predicts with a weighted sum of the features plus a constant:

prediction = intercept + w1*x1 + w2*x2 + ...

Training finds the weights that make the predictions closest to the true values, by minimising the sum of squared errors (ordinary least squares).

Fit a line

Python LinearRegression on a simple dataset

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

coef_ has one weight per feature; intercept_ is the value when all features are 0. Because we generated the data from a known line, we can see the model recovered it.

Multiple features

With several features the model fits a plane (or hyperplane). Each coefficient says how much the prediction changes when that feature rises by one unit with the others held fixed.

Python Coefficients on the diabetes dataset

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The features in this dataset are already standardised, so coefficient sizes can be compared. With raw features of different units they cannot, unless you scale first (Scaling and Normalization).

Measuring error

A single number is not enough, so use several. All of them live in sklearn.metrics:

  • MAE (mean absolute error): the average miss in the target's own units. Easy to explain.
  • MSE and RMSE: squares the errors first, so big misses cost more. RMSE is back in the target's units.
  • R-squared (Rยฒ): the share of the target's variance the model explains. 1 is perfect, 0 is no better than always predicting the mean, and it can be negative on test data for a bad model.
Python Regression metrics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Always compare with a simple baseline. If your model barely beats "predict the average", it has learned little. More on this in the evaluation chapter.

Residuals: what the model gets wrong

A residual is true - predicted. For a healthy linear model they should look like random noise with no pattern and mean near zero.

Python Residuals plot

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A curved pattern or a funnel shape means the relationship is not linear or the noise changes with the level. The fixes are polynomial features, transformed targets or a different model.

Assumptions and weaknesses

Linear regression works best when the relationship is roughly linear, features are not strongly correlated with each other, and there are no wild outliers. Its main weaknesses:

  • Outliers drag the line because errors are squared.
  • Correlated features (multicollinearity) make coefficients unstable, even though predictions stay fine. Regularization fixes this.
  • It cannot extrapolate non-linear trends.
Python One outlier moves the line

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Under the hood: the normal equation

For curiosity, LinearRegression solves the least-squares problem exactly, with no iteration and no learning rate, so there is nothing to tune. For millions of rows, SGDRegressor trains by gradient descent instead.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the intercept_ attribute represent?

    The intercept is the constant term, the output when every feature is 0.
  2. 2. An R-squared of 0 on test data means:

    R-squared compares your model with always predicting the mean of the target.
  3. 3. Which metric is in the same units as the target and treats big errors more harshly than MAE?

    RMSE is the square root of MSE: same units as the target, with squared errors emphasising large misses.
  4. 4. A residual plot shows a clear U-shaped curve. What does that suggest?

    Structure in the residuals means the model is missing something, here non-linearity.

Summary

  • Linear regression predicts intercept + sum(weight * feature) and is trained by least squares.
  • coef_ and intercept_ are readable results; score returns R-squared.
  • Judge with MAE, RMSE and R-squared on a test set, and compare with a baseline.
  • Residual plots reveal missed patterns.
  • Outliers and correlated features are its main weaknesses.

Next: Ridge, Lasso and Elastic Net