Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 40 ยท Regression

Decision Tree and k-NN Regression

Fit non-linear regression with DecisionTreeRegressor and KNeighborsRegressor: tune depth and neighbors, see step-shaped predictions, and learn their limits.

Linear models assume a straight-line relationship, and polynomial features assume a smooth curve. Two other families make almost no assumption about the shape: decision trees and nearest neighbors. Both predict by looking at similar training examples, which makes them flexible, and also prone to overfitting if you let them.

DecisionTreeRegressor

A regression tree asks a series of yes/no questions about the features ("is x < 2.3?") until it reaches a leaf. The leaf predicts the average target of the training samples that landed there. The whole tree is therefore a set of flat steps.

Python A tree fits a curve with steps

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With max_depth=2 the tree has at most 4 leaves, so it can output only 4 different values. More depth means more steps and a closer fit.

The depth dial

An unlimited tree gives every training point its own leaf: a perfect training score and poor generalisation. Limit it with max_depth, min_samples_leaf or min_samples_split.

Python Train versus test by tree depth

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

None means no limit: training Rยฒ reaches 1.0, but test Rยฒ stopped improving at a depth of about 3 to 5, so the extra leaves only memorise noise. More on this in Overfitting and Underfitting.

Trees cannot extrapolate

Because leaves hold averages of training targets, a tree can never predict outside the range it saw. A linear model, in contrast, continues the trend.

Python Beyond the training range

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For forecasting growth beyond past values, a tree is the wrong tool.

KNeighborsRegressor

k-nearest neighbors predicts the average target of the k closest training points. There is no training in the usual sense: fit just stores the data. The key choices are n_neighbors and the distance measure.

Python Effect of k

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

  • k=1 copies the nearest point: perfect on training data, noisy on new data (high variance).
  • Very large k averages almost everything, producing a flat line (high bias).
  • The best k lies in between. weights="distance" lets closer neighbors count more.

k-NN needs scaled features

Distances add up differences in every column, so a column with large numbers drowns the others. Always scale (Scaling and Normalization).

Python Scaling and k-NN regression

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Compare the two on one dataset

Python Tree versus k-NN versus linear on the diabetes data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

cross_val_score repeats the train/test split five times for a steadier estimate. There is no universal winner; the data decides.

When to use each

Tree k-NN
Needs scaling No Yes
Handles interactions Yes, automatically Through distance only
Training / prediction speed Fast / fast Instant / slower on big data
Interpretable Yes (small trees) Somewhat
Extrapolates No No

Single trees are mostly a building block: forests and boosting combine many trees for much better accuracy.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a regression tree leaf predict?

    Each leaf holds the mean of the training targets that reach it.
  2. 2. An unlimited-depth DecisionTreeRegressor scores 1.0 on training and much lower on test. This is:

    The tree memorised the training data, including noise.
  3. 3. With k=1, KNeighborsRegressor tends to have:

    Copying the single nearest neighbor follows every noisy point.
  4. 4. Which statement is true about extrapolation?

    Both average training targets, so predictions stay inside the training range.

Summary

  • Trees predict leaf averages as steps; control them with max_depth and min_samples_leaf.
  • k-NN averages the k closest targets; tune n_neighbors and scale features.
  • Neither extrapolates beyond the training targets.
  • Both overfit when given too much freedom.

Next: Ensembles and Support Vector Regression