Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 111 ยท Regression

Linear Regression

Learn how linear regression fits a line by least squares, how to read the slope and intercept, and how to use LinearRegression to predict and plot results.

Regression means predicting a number: a price, a temperature, the minutes a delivery will take. Linear regression is the oldest and simplest way to do it. It assumes the target changes in a straight-line way with the input, and it finds the one line that fits your examples best. Despite its age, it is still the first model professionals try, because it is fast, easy to explain and a strong baseline for anything fancier.

In this lesson you will see what "best fit" means, compute the answer by hand with NumPy, get the same answer from scikit-learn's LinearRegression, and learn to read the two numbers the model learns: the slope and the intercept. It builds on the estimator pattern from The scikit-learn API and the split from Train, Validation and Test Sets.

A line has two knobs

Slope How much the target changes when the input grows by one unit.
Intercept The prediction when the input is zero.

Prediction = intercept + slope x input.

1 / 3

The model: a straight line

With one input feature x (say, a flat's size in square meters) and one target y (its monthly rent), the model is:

TEXT
prediction = intercept + slope * x

The slope (scikit-learn calls it coef_) says how much the prediction rises per extra unit of x. The intercept (intercept_) is where the line crosses the vertical axis, meaning the prediction when x is zero. Training means choosing those two numbers. Everything else (predicting, scoring, plotting) follows from them.

Notice what is learned and what is not. The slope and intercept are learned parameters: you never type them in. There are no hyperparameters to tune in plain linear regression, which is part of its charm. The terms are explained in The scikit-learn API.

What "best fit" means

Imagine drawing any line through a scatter plot. For each point, the vertical distance between the point and the line is the residual (actual minus predicted). A good line makes residuals small. Because some are positive and some negative, we square them before adding, so they cannot cancel out. The total is the sum of squared errors (SSE), and the line that makes it smallest is the least squares line.

For one feature there is a closed-form answer:

TEXT
slope     = sum((x - mean_x) * (y - mean_y)) / sum((x - mean_x) ** 2)
intercept = mean_y - slope * mean_x

Read the slope formula as "how much do x and y move together, relative to how much x moves on its own". The intercept formula says the best line always passes through the point of means. Try it with generated data, and check that nudging the slope in either direction makes the error larger.

Python Least squares by hand with NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The data was generated from a "true" slope of 3.2 and intercept of 50, then blurred with noise. The fit recovers numbers close to those, not exactly them, because 30 noisy points never tell the whole truth. Every nudge you try raises the SSE: that is what "best" means here.

The same thing with scikit-learn

In practice you do not write the formulas. LinearRegression follows the usual estimator routine: create, fit, predict. The one gotcha is shape: scikit-learn wants X as a two-dimensional array of shape (n_samples, n_features), even with a single feature, while y is one-dimensional.

Python LinearRegression: fit, inspect, predict

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The slope and intercept match the hand-computed values from the previous example, because both minimize the same quantity. coef_ is an array (one entry per feature), which is why we read coef_[0]. Names ending in an underscore, like coef_, always mean "learned during fit".

model.score(X, y) returns R squared, the share of the target's variation that the line explains: 1.0 is perfect and 0.0 is no better than always predicting the average. It is introduced properly in Regression Metrics.

Note: Under the hood, scikit-learn does not use the textbook formula. It solves the least squares problem with a numerically stable linear algebra routine (an SVD-based solver), which also copes with several features at once. The answer is the same.

Seeing the fit

A regression is easiest to trust when you can look at it. Plot the points, draw the line, and draw the residuals as short vertical segments. This is also a quick check that a straight line is a reasonable idea at all.

Python Plotting the fitted line and its residuals

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On the left, the grey segments are the residuals that least squares minimizes. On the right, the residuals are plotted against the predictions: you want a shapeless cloud around zero. A curve, or a funnel that gets wider to one side, would mean the straight-line assumption is wrong. Notice also that the mean residual prints as almost exactly zero: with an intercept in the model, least squares always balances the positive and negative misses.

Interpreting and misusing the line

The slope is a statement about association in your data, not a law of nature. "Each extra square meter goes with about 3 more rent units" is a fair reading. "Adding a room to your flat will raise the rent by that much" is a causal claim the model cannot support.

Three habits keep you honest:

  • Do not extrapolate far. A line fitted on flats of 40 to 160 m2 says nothing reliable about a 600 m2 mansion. The line keeps going, but the world might not.
  • Watch the intercept. If x = 0 is outside your data (a flat of zero square meters), the intercept is just a mathematical anchor, not a meaningful prediction.
  • Beware outliers. Because errors are squared, one wildly wrong point pulls the line toward itself. Look at the plot before trusting the numbers.
Python One outlier can tilt the line

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One point out of 26 changes the slope noticeably. Later lessons cover remedies: robust loss functions, regularization and sensible data cleaning.

When a straight line is the right tool

Linear regression works well when the relationship is roughly linear, the errors are fairly symmetric and similar in size across the range, and you care about an explanation as much as a prediction. It can fail when the pattern curves (see Polynomial Regression and Nonlinearity) or when inputs interact in complicated ways (trees and ensembles, covered later in the track). Even then, it stays useful as a baseline: if a complex model cannot beat a straight line by a clear margin, the complexity is not paying for itself.

The assumptions behind linear regression

You can fit a line to any data, but the p-values, confidence intervals and the trust you place in the coefficients rely on assumptions. Predictions can still be useful when some are bent; interpretation is what suffers. The classic list:

  1. Linearity. The average of y changes in a straight line with each feature (or you have transformed the features until it does). A curved residual plot signals a violation.
  2. Independence of errors. One row's error tells you nothing about another's. Time series and repeated measurements of the same person break this (the errors are autocorrelated). The Durbin-Watson statistic is near 2 when there is no first-order autocorrelation.
  3. Constant variance (homoscedasticity). The scatter of the errors is the same for every prediction. A funnel shape means heteroscedasticity: the model is much less certain for large values than for small ones, and the usual standard errors are wrong. The Breusch-Pagan test checks it.
  4. Normally distributed errors. The residuals are roughly bell-shaped. It matters mostly for small samples and for confidence intervals; with many rows, the coefficients are close to normal anyway. A histogram, a Q-Q plot or the Shapiro-Wilk test check it. (With several features this is often phrased as "multivariate normality", meaning that the features and errors are jointly normal.)
  5. No (severe) multicollinearity. The features are not near-copies of each other, otherwise coefficients become unstable (multicollinearity).
  6. Additivity. Each feature's effect is the same whatever the others are. If the effect of advertising depends on the season, add an interaction term (polynomial features and interactions).

The example below fits two datasets with statsmodels: one that obeys the assumptions, and one built to break them (the noise grows with x, and the errors are carried over from one row to the next). It prints the tests and shows what the violations do to the slope's standard error.

Python Diagnosing the assumptions with statsmodels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The good dataset passes everything: a large Breusch-Pagan p-value (0.95, no sign of a funnel), a Durbin-Watson value of 2.14 (close to 2) and a Shapiro-Wilk p-value of 0.67 (the residuals look bell-shaped). The bad dataset fails: a Breusch-Pagan p-value of essentially zero (the spread depends on x), a Durbin-Watson value of 0.63 (strong positive autocorrelation) and a rejected normality test. Notice that the slope is still estimated reasonably in both cases; what breaks is the claim about its precision. Over 400 repeated datasets the slope of the good data varies by 0.035, in line with the 0.033 the formula reports. For the bad data the slope really varies by 0.181, while the formula reports only 0.074: the textbook standard error is too optimistic by a factor of about 2.5, so confidence intervals and p-values would claim far more certainty than the data justify.

What to do when an assumption fails:

  • Curved residuals: add polynomial terms, transform features, or move to a non-linear model.
  • Funnel-shaped residuals: take the logarithm of the target, use weighted least squares, or heteroscedasticity-robust standard errors (fit(cov_type="HC3") in statsmodels).
  • Autocorrelated errors: use time-aware features and models (time series) and time-ordered validation.
  • Non-normal errors: transform the target, use robust regression (outliers), or rely on the bootstrap for intervals.
  • Collinearity: drop or combine features, or use ridge regression.

If you only need predictions and you validate them properly (cross-validation), mild violations are tolerable. If you need to explain effects or give confidence intervals, check them first.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does least squares choose?

    Least squares minimizes the total of the squared gaps between actual and predicted values. It usually passes through very few points exactly.
  2. 2. You have one feature stored in a 1D array size of length 30. What do you pass to fit?

    scikit-learn expects X with shape (n_samples, n_features). One feature means one column, so reshape to (30, 1).
  3. 3. What does model.coef_[0] hold after fitting a one-feature LinearRegression?

    coef_ stores the learned weights, one per feature. With a single feature it is the slope. The intercept lives in intercept_.
  4. 4. Why is it risky to predict for a flat far outside the sizes in your training data?

    A fitted line extends forever, but the model only saw evidence inside the training range. Extrapolating assumes the pattern continues unchanged.

Summary

  • Linear regression predicts a number as intercept + slope * x, with one slope per feature when there are several.
  • Least squares picks the line with the smallest sum of squared residuals; for one feature it has a simple closed-form formula.
  • LinearRegression().fit(X, y) needs a 2D X; read the learned values from coef_ and intercept_.
  • Plot the fit and the residuals: a shapeless cloud around zero means the line is a fair description.
  • The slope describes association, not cause; avoid extrapolating and watch for outliers.

Next: use several inputs at once in Multiple Linear Regression.