Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 39 ยท Classification

Prediction in Data Mining

Predict numbers instead of categories: linear regression, regularization, tree regressors, error metrics (MAE, RMSE, R2) and a first look at time-based forecasting, with runnable Python examples.

Classification predicts a category. Prediction in the numeric sense, also called regression, predicts a continuous value: tomorrow's sales, a house price, the number of minutes a delivery will take. The process is the same as before (learn from known records, evaluate on unseen ones), but the target is a number, so we measure errors in distance rather than counting right and wrong answers.

Some books use "prediction" for both tasks. In this lesson it means numeric prediction. Models you already know have regression versions: trees, k-NN and ensembles; plus the classic linear regression.

Linear regression

Linear regression fits a straight line (or plane) y = b0 + b1*x1 + b2*x2 + ... by choosing the coefficients that minimize the sum of squared errors (least squares). The coefficients are directly interpretable: b1 is the average change in y when x1 rises by one unit and everything else stays the same.

Python Least squares by hand and with scikit-learn

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Measuring prediction error

With a numeric target, the residual of a record is actual - predicted. Summaries of the residuals:

  • MAE (mean absolute error): average size of the error, in the units of the target.
  • MSE / RMSE: squares the errors first, so big mistakes weigh more; RMSE returns to the original units.
  • R-squared (R2): the share of variance explained, 1 is perfect and 0 is no better than always predicting the mean (it can be negative on test data).
  • MAPE: error as a percentage of the actual value (breaks when actual values are near zero).
Python Regression metrics on a held-out set

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Always compare with a naive baseline such as the training mean. A model that cannot beat it has learned nothing.

Overfitting and regularization

With many features or a flexible model, the fit follows noise. Regularization adds a penalty on the size of the coefficients. Ridge penalizes squared coefficients (shrinks them), Lasso penalizes absolute values (can push some to exactly zero, so it also selects features).

Python Ridge and Lasso on noisy features

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Plain least squares overfits when there are almost as many features as records. Regularized models do better, and Lasso keeps only a few features.

Non-linear relationships

If the relationship bends, a straight line underfits. Options: add polynomial terms, or use models that adapt on their own such as a regression tree, k-NN regression or a random forest regressor.

Python Linear model versus trees on a curve

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Predicting over time

When records are ordered by time (daily sales), the future must be predicted from the past, and the test set must be the latest part of the series, never a random sample: shuffling would let the model peek at the future. A simple, strong approach is to turn the series into a table of lag features (yesterday's value, last week's value) and fit an ordinary regressor.

Python Forecasting with lag features

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The lag model uses the 7-day pattern and beats "tomorrow equals today". For dedicated tools see time series with statsmodels.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does least squares minimize?

    Ordinary least squares chooses the coefficients with the smallest total squared error.
  2. 2. An R-squared of 0 on test data means the model is:

    R2 compares the model with the mean baseline; 0 means no improvement and negative means worse.
  3. 3. What is special about Lasso compared with Ridge?

    The absolute-value penalty of Lasso zeroes out weak coefficients, which performs feature selection.
  4. 4. How should you split time-ordered data for evaluation?

    A random split leaks future information into training. The test set must come after the training set in time.

Summary

  • Numeric prediction (regression) estimates continuous values; evaluate with MAE, RMSE and R2 against a naive baseline.
  • Linear regression is interpretable; Ridge and Lasso reduce overfitting and Lasso selects features.
  • Trees, k-NN and forests handle non-linear patterns.
  • For time series use lag features and a chronological split.

Next: Clustering Basics