Skip to content
Tutorials Center
My learning

Topics

Lesson 47 of 76 ยท Modelling

Linear Regression for Inference and Prediction

Fit linear regression with statsmodels and scikit-learn, read the coefficient table, check residual diagnostics and multicollinearity, and interpret results.

Linear regression is the workhorse of data science. It models a numeric outcome as a weighted sum of inputs plus an error: price = intercept + b1 * size + b2 * age + .... It is quick to fit, easy to explain, and often hard to beat on small, tidy tables. It is also the cleanest place to learn the difference between two goals that people mix up: inference (what is the relationship, and how sure are we?) and prediction (how well can we guess new cases?).

This lesson uses two libraries on purpose. statsmodels gives you the statistician's view: a full coefficient table with standard errors, intervals and p-values. scikit-learn gives you the engineer's view: a model object with fit and predict that plugs into validation tools. If you are new to the modelling loop, read the modelling overview first. For the linear algebra behind least squares, see least squares in SciPy.

Two goals, one model

For inference you care about the coefficients. "Holding age and garage fixed, each extra square metre is associated with about 1.9 more price units, give or take 0.07." You need the standard errors, and you need the model's assumptions to be roughly true, because the intervals rely on them.

For prediction you care about error on data the model has not seen. Coefficients may be unstable or hard to interpret and the model can still predict well. The assumptions matter less; honest validation matters more (see train/test splits).

flowchart TD
    A["Linear regression"] --> B["Inference<br/>(explain)"]
    A --> C["Prediction<br/>(forecast)"]
    B --> D["Coefficients, intervals,<br/>p-values, assumptions"]
    C --> E["Held-out error,<br/>cross-validation"]

The same fitted line serves two different questions, and each needs different evidence.

Fitting with statsmodels and reading the table

We generate a small housing table with a known recipe, so we can check whether the model recovers it: the true effects are +2.0 per size unit, -1.5 per year of age, and +20 for a garage, with noise on top.

Python OLS coefficient table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

sm.add_constant adds the intercept column; without it, statsmodels forces the line through zero. The summary table has one row per term. Read it left to right:

  • coef is the estimated effect of one unit of that input, with the other inputs held fixed. garage is 0/1, so its coefficient is the average difference between houses with and without a garage.
  • std err says how much the estimate would wobble across repeated samples. Smaller is better, and it shrinks with more data.
  • t is coef / std err, and P>|t| is the p-value for the null hypothesis "this true coefficient is zero". See hypothesis testing for what that does and does not mean.
  • [0.025, 0.975] is a 95% confidence interval. This is usually the most useful column: it shows the plausible range, and here every interval contains the true value we planted.

The other headline numbers: R-squared is the share of the outcome's variance the model explains on the training data, and adjusted R-squared penalizes extra inputs. The residual standard error is the typical miss, in the units of the outcome. A high R-squared does not prove the model is right, and a low one does not make it useless; in noisy human data, an R-squared of 0.2 can still hold a real, important effect.

Using formulas and categories

statsmodels.formula.api lets you write the model like an equation. Categorical columns are expanded into 0/1 columns automatically with C(...). One level is left out as the baseline, and every other coefficient is a difference from it.

Python Formula interface with a categorical column

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

City A is the baseline, so C(city)[T.B] means "city B compared with city A, at the same size and age". The model recovers the planted +30 and -15 reasonably well. A coefficient is a conditional association: it describes the relationship after accounting for the other columns in the model, which is not the same as the raw difference between group averages.

A regression coefficient is not a causal effect by default. If an important driver is missing from the model, the coefficients of the included inputs can absorb its influence. Causal thinking explains when a coefficient can be read as an effect.

Checking the assumptions with residuals

Ordinary least squares assumes, roughly, that the relationship is linear, that the errors have constant spread, that they are not strongly dependent on each other, and (for the small-sample intervals) that they are close to normal. Everything you need to check these lives in the residuals, the leftover differences actual - fitted.

Two plots do most of the work. A residuals-versus-fitted plot should look like a shapeless cloud around zero. A curve means the relationship is not linear; a funnel means the spread changes. A normal Q-Q plot should follow the straight line.

The next example fits a straight line to two datasets: one truly linear, one truly curved. The summary numbers alone might not warn you; the plots do.

Python Residual plots: linear truth vs curved truth

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Even the curved case has a respectable R-squared, yet its residual plot is a clear U-shape: the line is too high in the middle and too low at the ends. The fix is to change the model, for example by adding a squared term (x + I(x**2) in a formula) or transforming a variable. Look for outliers too; a handful of extreme points can pull a line a long way, and fit.get_influence() can rank points by their leverage.

Multicollinearity

When two inputs carry almost the same information, the model cannot tell which one deserves the credit. Predictions stay fine, but coefficients become unstable and their standard errors blow up. The common diagnostic is the variance inflation factor (VIF). As a rough guide, values above 5 to 10 deserve a look.

Python Two near-duplicate columns inflate the standard errors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Height in centimetres and in inches are the same fact twice. With both in, the standard error of height_cm is roughly ten times larger than with one. The cure is to drop or combine the redundant columns, or to accept the instability if you only care about predictions.

The scikit-learn version

For prediction work, scikit-learn has the same model with a different interface: LinearRegression().fit(X, y), then .predict(X_new). It has no p-values or intervals, but it works with train/test splitting, cross-validation and pipelines. It needs a purely numeric matrix, so expand categories with pd.get_dummies first.

Python Predicting held-out houses with scikit-learn

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The fitted coefficients are close to what statsmodels found on the same recipe. The difference is the workflow: here the numbers that matter are test MAE and test R2, computed on rows the model never saw. A tool for the metrics themselves is coming in evaluating models.

Interpreting results responsibly

  • Say what the coefficient means in units. "Each extra year of age goes with 1.5 lower price, at the same size" is better than "age is significant".
  • Report the interval, not only the p-value. A tiny effect can be very significant in a large sample, and a big effect can be non-significant in a small one.
  • Do not extrapolate. The line holds only inside the range of data you fitted. A model trained on 50-150 square metre houses says nothing reliable about a 600 square metre mansion.
  • Compare with a baseline. If always predicting the average price is nearly as good, the model adds little.
  • Log-transform skewed outcomes. For positive, right-skewed values such as prices, modelling log(price) often fixes curvature and unequal spread, and coefficients become approximate percentage changes.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In a statsmodels OLS summary, what does the interval [0.025, 0.975] for a coefficient give you?

    It is the confidence interval for the coefficient. It describes the uncertainty of the estimate, not the spread of the data and not the probability that the effect is zero.
  2. 2. A residuals-versus-fitted plot shows a clear U-shaped curve. What does that suggest?

    Residuals should look like shapeless noise. A systematic curve means the straight line misses a pattern, so add a transformation or a squared term.
  3. 3. Two predictors are nearly identical copies of each other. What is the usual result?

    This is multicollinearity. The model cannot split credit between the near-duplicates, so individual coefficients wobble, but the combined prediction is often still good.
  4. 4. Which statement about scikit-learn's LinearRegression compared with statsmodels OLS is true?

    Both minimize squared error. scikit-learn focuses on fit and predict, so it works with splits, cross-validation and pipelines, but does not produce a statistical table.

Summary

  • Linear regression models a numeric outcome as an intercept plus weighted inputs; use it to explain relationships or to predict.
  • statsmodels gives the coefficient table: estimates, standard errors, p-values and 95% intervals. Add the constant yourself, or use the formula interface.
  • A coefficient is a conditional association, not automatically a causal effect.
  • Check residual plots for curvature, funnels and outliers; R-squared alone can hide a wrong model.
  • Near-duplicate inputs inflate standard errors; spot them with VIF.
  • scikit-learn's LinearRegression is the prediction-oriented version, judged on held-out data.

Next, switch from numbers to categories with classification with logistic regression.