Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 40 ยท Classification

Decision Tree Classifier

Build and read a DecisionTreeClassifier: how Gini impurity picks splits, controlling depth and leaves, export_text, plot_tree, feature importances and pruning.

A decision tree classifies by asking a chain of simple questions, like a flowchart: "Is the petal width below 0.8 cm? If yes, setosa. If no, is it below 1.75 cm?..." Trees are easy to read, need no feature scaling, handle numeric and encoded categorical data, and capture interactions without being told. Their weakness is overfitting, which is why you will limit them or combine them into forests.

Grow a tree

Python DecisionTreeClassifier and export_text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each indented line is a question; each class: line is a leaf with a prediction. Because the whole model prints as rules, you can show it to a non-expert.

How a split is chosen: impurity

At each node the tree searches every feature and threshold for the split that makes the two child groups as pure (single-class) as possible. Purity is measured by Gini impurity, 1 - sum(p_k^2), where p_k is the share of class k in the node. It is 0 for a pure node and largest when classes are evenly mixed. The alternative criterion is entropy (information gain).

Python Gini impurity by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The tree picks, greedily, the split that lowers impurity the most, then repeats on each side until it stops.

Visualise the tree

Python plot_tree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each box shows the rule, its Gini value, the number of samples and the class counts. Colours show the majority class.

Overfitting and how to control it

A tree with no limits keeps splitting until every leaf is pure, memorising the training data.

Python Depth versus accuracy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Training accuracy climbs to 1.0, but test accuracy stops improving earlier. The main controls are:

  • max_depth: the maximum number of questions in a row.
  • min_samples_leaf: every leaf must contain at least this many samples.
  • min_samples_split: a node needs this many samples to be split.
  • max_leaf_nodes: a cap on the total number of leaves.
  • ccp_alpha: cost-complexity pruning, growing the tree and then removing branches that do not pay for themselves.
Python Pruning with ccp_alpha

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Feature importances

feature_importances_ shows how much each feature reduced impurity across the whole tree, summing to 1.

Python Which features matter?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Probabilities and instability

predict_proba returns the class shares in the leaf. These are coarse for shallow trees (a leaf of 3 samples can only say 0, 0.33, 0.67 or 1). Also, trees are unstable: small changes in data can produce a very different tree.

Python Instability

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Even here, the root threshold shifts from sample to sample, and deeper questions change more. Ensembles exploit exactly that variety.

Strengths and weaknesses

Strengths Weaknesses
Readable rules Overfit easily without limits
No scaling needed Unstable to small data changes
Handles interactions, mixed scales Axis-aligned splits struggle with diagonal boundaries
Fast prediction Cannot extrapolate (regression)

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a Gini impurity of 0 mean for a node?

    Gini is 0 when every sample in the node has the same class.
  2. 2. Which parameter caps how many questions in a row a tree may ask?

    max_depth limits the length of the longest path from root to leaf.
  3. 3. Do decision trees need feature scaling?

    Monotonic rescaling of a feature does not change where a tree can split.
  4. 4. A fully grown tree scores 1.0 on training and notably less on test data. What is the usual fix?

    The tree overfits, so constrain its complexity or average many trees.

Summary

  • A tree asks threshold questions to separate classes; splits are chosen to reduce Gini impurity or entropy.
  • export_text and plot_tree make the model readable.
  • Control overfitting with max_depth, min_samples_leaf, max_leaf_nodes and ccp_alpha.
  • No scaling needed, but single trees are unstable; ensembles fix that.

Next: Random Forests and Boosting