Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 37 · Correlation and Regression

The Least-Squares Regression Line

Find the line of best fit y = a + bx using least squares, interpret slope and intercept, make predictions, and avoid extrapolation mistakes.

When a scatter plot shows a roughly linear pattern, you can summarize it with a straight line and use that line to predict. But many lines could be drawn through a cloud of points. Which is "best"? Statisticians use the least-squares regression line: the line that makes the total of squared vertical misses as small as possible.

This lesson uses the study-hours data from the previous lesson.

The model

The line is written:

Text
ŷ = a + b x

where ŷ (read "y-hat") is the predicted y, b is the slope and a is the intercept. The formulas are:

Text
b = Σ(dx · dy) / Σ(dx²)
a = ȳ - b · x̄

An equivalent form of the slope is b = r × (s_y / s_x), which shows the link between slope and correlation. The line always passes through the point (x̄, ȳ).

Worked example

Using hours studied (1 to 5) and scores (52, 55, 61, 66, 71), the previous lesson found Σ(dx·dy) = 49, Σdx² = 10, x̄ = 3 and ȳ = 61. So:

Text
b = 49 / 10 = 4.9
a = 61 - 4.9 × 3 = 46.3
ŷ = 46.3 + 4.9 x
Python Fit a regression line from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Interpreting slope and intercept

Always interpret in context:

  • Slope 4.9: for each additional hour studied, the predicted score rises by about 4.9 points.
  • Intercept 46.3: the predicted score for zero hours. This is only meaningful if x = 0 is a sensible value in the data's range.

Pay attention to the units: slope is "units of y per unit of x".

Prediction

To predict, substitute x. For 4.5 hours: 46.3 + 4.9 × 4.5 = 68.35. This is interpolation, because 4.5 lies inside the data's x-range (1 to 5), so it is reasonably trustworthy.

For 12 hours the formula gives 105.1, which is above the maximum score of 100. That is extrapolation: predicting outside the observed range. It can give nonsense, because the linear pattern need not continue.

Avoid extrapolating far beyond the data. Also, the line predicts the average y for a given x, not the exact value for one individual.

Why "least squares"?

For each point, the vertical distance from the line is a residual: y - ŷ. Positive and negative residuals would cancel if you just added them, so we square them first. The least-squares line minimizes Σ(y - ŷ)². You can check this experimentally: try other slopes and intercepts and see whether any beat the formula.

Python Try to beat the least-squares line

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The grid search lands on (or extremely near) the formula's answer, because the formula is exactly the minimum.

Explanatory vs response

Regression is not symmetric. The line predicting y from x is generally not the same as the line predicting x from y. Choose x as the variable you use to predict, and y as the one you want to predict.

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. A regression line is ŷ = 20 + 3x, where x is hours and ŷ is predicted points. What does the slope mean?

    The slope is the change in predicted y per one-unit increase in x.
  2. 2. Which point always lies on the least-squares line?

    Because a = ȳ - b x̄, substituting x̄ gives ȳ.
  3. 3. Data cover x from 10 to 30. Predicting y at x = 20 is... and at x = 80 is...

    20 is inside the observed range; 80 is far outside it.
  4. 4. What does the least-squares line minimize?

    Residuals sum to zero for any such line, so we minimize the sum of their squares.

Summary

  • The least-squares line ŷ = a + bx has b = Σdxdy / Σdx² and a = ȳ - b x̄.
  • It minimizes the sum of squared vertical errors and passes through (x̄, ȳ).
  • Interpret slope and intercept with units and context.
  • Interpolation is safer than extrapolation.

Next: Residuals and R-Squared