Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 39 ยท Statistics and Modeling

Linear and Logistic Regression in R: lm, glm, predict

Fit and interpret regression models in R: lm(), summary(), coefficients, confidence intervals, predict(), residual diagnostics, multiple regression, glm() logistic regression and odds ratios.

Regression models how an outcome changes with one or more predictors. It answers questions such as "how much fuel does an extra 1,000 pounds of weight cost?" and "what is the chance this customer buys?". R's lm() fits linear regression for numeric outcomes, and glm() fits generalized models such as logistic regression for yes/no outcomes. Both use the same formula syntax and the same helper functions (summary(), coef(), predict()), so the workflow transfers.

R is not available on this site, so the R code is read-only. The numbers in comments were reproduced by fitting the same models in Python (least squares and iteratively reweighted least squares) on the same mtcars data; the Python block at the end lets you re-run them. For the underlying theory see Statsmodels and the Statistics and Probability tutorial.

Simple linear regression

Model the fuel economy mpg of the 32 cars in mtcars as a straight-line function of weight wt (in 1,000 lb). The formula mpg ~ wt reads "mpg as a function of wt":

R
fit <- lm(mpg ~ wt, data = mtcars)
coef(fit)
# (Intercept)          wt 
#   37.285126   -5.344472 

The fitted line is mpg = 37.29 - 5.34 * wt. The slope says that each additional 1,000 lb of weight is associated with about 5.34 fewer miles per gallon. The intercept (37.29 mpg at weight 0) is just where the line crosses the axis; no car weighs zero, so do not interpret it physically.

Draw the data and the line in base graphics: plot(mpg ~ wt, data = mtcars); abline(fit).

Reading summary()

R
summary(fit)

The important parts of the printed output, and what they mean:

Part Value here Meaning
Estimate (wt) -5.3445 change in mpg per unit of weight
Std. Error (wt) 0.5591 uncertainty of that estimate
t value (wt) -9.559 estimate divided by its standard error
Pr(>|t|) (wt) 1.29e-10 p-value for H0: true slope is 0
Residual standard error 3.046 on 30 df typical size of a prediction miss, in mpg
Multiple R-squared 0.7528 share of the variance in mpg explained by weight
Adjusted R-squared 0.7446 R-squared corrected for the number of predictors
F-statistic 91.38 on 1 and 30 DF overall test of the model (p = 1.294e-10)

So weight explains about 75 percent of the variation in fuel economy, and a typical car's mpg is within about 3 mpg of the line. In simple regression the F-statistic equals the square of the slope's t value (9.559 squared is 91.38).

A confidence interval for each coefficient: confint(fit). A 95 percent interval is the estimate plus or minus about two standard errors, so the slope is roughly -6.5 to -4.2.

Predicting

predict() evaluates the model on new data. The new data frame must use the same column names as the model's predictors:

R
new_cars <- data.frame(wt = c(2, 3, 4))
predict(fit, newdata = new_cars)
predict(fit, newdata = data.frame(wt = 3), interval = "confidence")
predict(fit, newdata = data.frame(wt = 3), interval = "prediction")

For wt = 3 the prediction is 37.285126 - 5.344472 * 3 = 21.25171 mpg. A "confidence" interval is for the average mpg of all cars that weigh 3,000 lb; a "prediction" interval is for the mpg of a single new car, which is wider because it includes the scatter around the line. Be careful with extrapolation: the data cover weights of about 1.5 to 5.4, and the line says nothing reliable about a 10-ton truck.

Checking the model

A model is only as good as its assumptions: a linear relationship, residuals with constant spread (homoscedasticity), roughly normal residuals and independent observations. Residuals are the vertical gaps between the points and the line; residuals(fit) returns them, and plot(fit) draws four standard diagnostic charts.

R
par(mfrow = c(2, 2))
plot(fit)
Chart What you want to see
Residuals vs Fitted a flat, patternless band (a curve means non-linearity; a funnel means unequal variance)
Normal Q-Q points near the diagonal line
Scale-Location an even horizontal band
Residuals vs Leverage no points beyond the Cook's distance lines

For mpg ~ wt the first chart shows a mild curve: fuel economy falls faster for light cars than for heavy ones. A transformation (lm(log(mpg) ~ log(wt))) or a quadratic term (lm(mpg ~ wt + I(wt^2))) can capture it. Note the I() wrapper, which makes ^ mean arithmetic instead of formula syntax.

Multiple regression

Add predictors with +:

R
fit2 <- lm(mpg ~ wt + hp, data = mtcars)
coef(fit2)
# (Intercept)          wt          hp 
# 37.22727012 -3.87783074 -0.03177295 

Now each coefficient is the effect of that variable holding the others fixed: for cars of the same horsepower, each extra 1,000 lb costs about 3.88 mpg, and for cars of the same weight, each extra horsepower costs about 0.032 mpg. The weight coefficient shrank from -5.34, because horsepower and weight are correlated and some of the effect that weight got credit for belongs to horsepower. R-squared rises to 0.8268.

Compare nested models with anova(fit, fit2) (an F-test) or by information criteria AIC(fit, fit2) (lower is better). R-squared always increases when you add a variable, so judge by adjusted R-squared, AIC or out-of-sample accuracy.

Categorical predictors

A factor is automatically turned into dummy variables against a reference level:

R
lm(mpg ~ wt + factor(cyl), data = mtcars)

You get one coefficient for 6 cylinders and one for 8 cylinders, each measuring the difference from 4 cylinders at the same weight. Use relevel() to change the reference. An interaction wt * factor(am) lets the slope differ between groups.

Things that go wrong

  • Multicollinearity: highly correlated predictors make coefficients unstable. Check cor() or car::vif().
  • Outliers and leverage: one extreme point can tilt the line; look at cooks.distance(fit).
  • Overfitting: too many predictors for the data fit noise. Validate on held-out data.

Logistic regression for yes/no outcomes

When the outcome is binary (0/1, yes/no), a straight line can predict impossible probabilities below 0 or above 1. Logistic regression models the log-odds of the outcome instead, using glm() with family = binomial. Here the outcome is am, 1 for a manual transmission and 0 for automatic:

R
fit3 <- glm(am ~ wt, data = mtcars, family = binomial)
coef(fit3)
# (Intercept)          wt 
#    12.0404     -4.0240 

Coefficients are on the log-odds scale. Exponentiate them to get odds ratios:

R
exp(coef(fit3))["wt"]       # 0.0179

Each extra 1,000 lb multiplies the odds of a manual transmission by about 0.018: heavier cars are overwhelmingly automatic in this data set.

To get probabilities, ask for type = "response":

R
predict(fit3, newdata = data.frame(wt = 3), type = "response")   # 0.4921

A 3,000-lb car has about a 49 percent chance of being a manual. Turn probabilities into classes with a threshold (0.5 is the usual starting point) and compare with the truth:

R
pred <- ifelse(predict(fit3, type = "response") > 0.5, 1, 0)
table(predicted = pred, actual = mtcars$am)
#           actual
# predicted  0  1
#         0 18  2
#         1  1 11
mean(pred == mtcars$am)     # 0.90625

The model classifies 29 of 32 cars correctly (90.6 percent), but that is measured on the same data it was fitted on, so it is optimistic. A real evaluation holds some rows out, or uses cross-validation. For rare outcomes accuracy is misleading; look at sensitivity, specificity and ROC curves (package pROC).

summary(fit3) reports z-values and p-values for each coefficient and the deviance, which plays the role that residual sum of squares plays for lm(). Other families cover other outcomes: poisson for counts and Gamma for positive skewed values.

Reproduce the numbers in Python

The Python block below re-fits all three models on the same data with plain NumPy: ordinary least squares from the normal equations, and logistic regression with Newton's method (the same algorithm R's glm() uses, called IRLS).

Python Python comparison: lm and glm by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common mistakes

  • Reading a regression coefficient as a causal effect. Observational data show association.
  • Interpreting the intercept when x = 0 is outside the data.
  • Extrapolating far beyond the range of the predictors.
  • Judging a model by R-squared alone, or by accuracy on the training data.
  • Forgetting that logistic coefficients are on the log-odds scale; use exp() for odds ratios.
  • Passing predict() new data with misspelled column names.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In lm(mpg ~ wt, data = mtcars), what does the slope -5.34 mean?

    The slope is the average change in the outcome for a one-unit increase in the predictor.
  2. 2. What does an R-squared of 0.75 say?

    R-squared is the share of the outcome's variance accounted for by the model.
  3. 3. How do you get predicted probabilities from a logistic regression fitted with glm()?

    By default predict() returns log-odds. type = 'response' converts them to probabilities.
  4. 4. A coefficient of -4.02 in a logistic model corresponds to an odds ratio of about:

    The odds ratio is exp(coefficient); exp(-4.02) is about 0.018.
  5. 5. Why is a prediction interval wider than a confidence interval for the mean response?

    A single new observation varies around the line as well as the line being uncertain.

Summary

  • lm(y ~ x1 + x2, data) fits linear regression; summary(), coef(), confint(), predict() and plot() are the standard helpers.
  • Coefficients are effects per unit holding other predictors fixed; R-squared and residual standard error describe fit.
  • Check residual plots; beware of extrapolation, collinearity and overfitting.
  • glm(y ~ x, family = binomial) models log-odds; exp(coef()) gives odds ratios and predict(type = "response") gives probabilities.
  • Evaluate on data the model has not seen.

Next: Functional Programming in R