Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 57 · Regression

Simple Linear Regression

Fit a straight line with least squares, interpret slope, intercept and R², build confidence and prediction intervals, and check residuals, with Python examples.

Correlation tells you that two variables move together. Regression goes further: it gives you an equation you can use to predict one variable from the other, and to describe how much the outcome changes per unit of the predictor. Simple linear regression uses one predictor x and one numeric outcome y.

The model is y = β₀ + β₁x + ε. Here β₀ is the intercept, β₁ the slope, and ε the random error that the line cannot explain. You never know β₀ and β₁; you estimate them from data as b₀ and b₁.

Least squares: choosing the line

Many lines pass near a cloud of points. Ordinary least squares (OLS) picks the one that minimizes the sum of squared residuals, where a residual is the vertical gap eᵢ = yᵢ − ŷᵢ between the observed value and the line's prediction. The solution has a closed form:

  • slope: b₁ = r · (s_y / sₓ) or equivalently Σ(xᵢ − x̄)(yᵢ − ȳ) / Σ(xᵢ − x̄)²
  • intercept: b₀ = ȳ − b₁x̄

The line always passes through the point of means (x̄, ȳ), and the residuals always sum to zero.

Python Fit a line by hand and with SciPy

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Interpreting the coefficients

  • Slope b₁: the average change in y for a one-unit increase in x. A slope of 4.0 means each extra hour of study goes with about 4 more points on average.
  • Intercept b₀: the predicted y when x = 0. It is only meaningful if x = 0 is realistic and inside your data range. Often it is just an anchor for the line.
  • Association, not causation: unless the data came from a randomized experiment, the slope describes a pattern, not what would happen if you intervened.

Do not extrapolate far beyond the observed x range. A line fitted to 1 to 10 hours says nothing reliable about 40 hours.

How good is the fit? R² and the standard error

Split the total variation in y into two parts:

  • SST = Σ(yᵢ − ȳ)² total sum of squares,
  • SSE = Σ(yᵢ − ŷᵢ)² unexplained (residual) sum of squares.

Then R² = 1 − SSE/SST is the share of variation explained by the line. In simple regression R² = r². An R² of 0.9 means 90% of the variation in y is accounted for by x.

A high R² does not mean the model is correct, and a low R² does not mean it is useless. When the outcome is naturally noisy (human behavior), R² of 0.2 can still be valuable. A more tangible measure is the residual standard error s = √(SSE/(n−2)), in the units of y: roughly the typical size of a prediction error.

Inference for the slope

The slope has a sampling distribution. Its standard error is SE(b₁) = s / √Σ(xᵢ − x̄)², and t = b₁ / SE(b₁) follows a t-distribution with n − 2 degrees of freedom when the assumptions hold. This gives:

  • a p-value for H₀: β₁ = 0 (no linear relationship),
  • a confidence interval b₁ ± t* · SE(b₁).

Confidence vs prediction intervals

There are two different intervals around the line at a new value x₀:

  • The confidence interval for the mean response answers "what is the average score of all students who study 6.5 hours?" It is narrow and shrinks with more data.
  • The prediction interval answers "what score might one new student get?" It adds the individual noise ε, so it is always wider and never shrinks to zero.
Python Intervals for the slope and for predictions

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Assumptions and residual checks (LINE)

The inference above relies on four conditions, remembered as LINE:

  1. Linearity: the true relationship is a straight line.
  2. Independence: the errors are independent (watch time series and clustered data).
  3. Normality: the errors are roughly normal (matters mostly for small samples).
  4. Equal variance: the spread of errors is the same at all x (homoscedasticity).

Check them by plotting residuals against fitted values. A good plot is a shapeless horizontal band around zero. A curve means the relationship is not linear; a funnel means unequal variance; one far-away point means an outlier, and a point far from x̄ can be a high-leverage point that drags the line towards itself.

flowchart LR
  A["Plot y vs x"] --> B["Fit the line (OLS)"]
  B --> C["Plot residuals vs fitted"]
  C --> D{"Random band?"}
  D -->|"yes"| E["Interpret slope, R², intervals"]
  D -->|"curve or funnel"| F["Transform, add terms, or change model"]
Python Spotting a curved pattern in residuals

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The R² of the straight line looks high, yet the residuals are positive at both ends and negative in the middle: the classic signature of a curve. R² alone would have hidden the problem.

Check your understanding

Quick check · 5 questions

of 5 answered You scored / 5 · perfect!

  1. 1. What does ordinary least squares minimize?

    OLS chooses the intercept and slope that give the smallest sum of squared vertical distances between the points and the line. The plain sum of residuals is always zero.
  2. 2. A regression of weight on height gives slope 0.9 kg per cm. How should you read it?

    The slope is an average difference associated with a one-unit change in x. It does not by itself prove cause, and it is not R squared.
  3. 3. Which interval is always wider at the same x?

    The prediction interval includes the random error of one new observation on top of the uncertainty in the line, so it is wider.
  4. 4. A residual plot shows a clear U-shape. What does it suggest?

    Systematic structure in residuals means the linearity assumption fails. Consider a transformation or a quadratic term.
  5. 5. In simple linear regression, how is R squared related to r?

    With one predictor, the share of variance explained equals the square of the correlation.

Summary

  • Simple linear regression models y = β₀ + β₁x + ε and estimates it by least squares.
  • The slope is the average change in y per unit of x; avoid extrapolating and avoid causal claims from observational data.
  • R² is the explained share of variation; the residual standard error gives typical error in y units.
  • Confidence intervals are for the average response, prediction intervals for one new case and are wider.
  • Check LINE with residual plots before trusting p-values and intervals.

Next: Multiple Linear Regression