Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 39 ยท Classification

Classification Basics

Learn what classification means in data mining: labels, training and test sets, overfitting, the confusion matrix and the metrics (accuracy, precision, recall, F1) with runnable code.

Classification is the data mining task of predicting a category for each record: will this customer churn (yes or no), is this message spam, which species is this flower. You start from records whose category is already known, find patterns that separate the categories, and then apply those patterns to new records. The category is called the class label or target; the other columns are the features (also called attributes).

Classification is supervised learning: a teacher (the known labels) tells the algorithm what the right answer is during training. Compare that with clustering, where no labels exist. In this chapter you will meet the main families of classifiers: trees, nearest neighbors, Bayesian models, rule-based models and ensembles. This lesson covers the common process and, most importantly, how to measure whether a classifier is any good.

The two-step process

Step 1: learn a model

Training data Records with known labels
Algorithm Tree, kNN, Bayes, ...
Model Rules learned from data

The model is just a function from features to a class.

1 / 2

The test records must not be used for training. A model can memorize its training records and look perfect, yet fail on new data. That failure is overfitting. The opposite, a model too simple to capture the pattern, is underfitting.

Splitting the data

The simplest evaluation is the holdout method: keep a fraction (often 20-30%) of the data aside as a test set. stratify keeps the class proportions equal in both parts, which matters when one class is rare.

Python Holdout split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Overfitting you can see

A decision tree allowed to grow without limit can fit the training data perfectly. Watch the gap between the training score and the test score as the tree gets deeper.

Python Depth versus accuracy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Training accuracy climbs to 1.000 for the unlimited tree, while the test accuracy stops improving or drops. A big gap between the two is the classic overfitting signal.

The confusion matrix

Accuracy (the fraction of correct predictions) hides the type of mistakes. A confusion matrix counts every combination of true class and predicted class. For a yes/no problem with "positive" as the class of interest, it has four cells:

Predicted positive Predicted negative
Actually positive True positive (TP) False negative (FN)
Actually negative False positive (FP) True negative (TN)

From these four counts come the standard measures:

  • Accuracy = (TP + TN) / all records.
  • Precision = TP / (TP + FP): of everything I flagged, how much was right?
  • Recall (sensitivity) = TP / (TP + FN): of all real positives, how many did I find?
  • Specificity = TN / (TN + FP): of all real negatives, how many did I clear?
  • F1 = the harmonic mean of precision and recall; it is low if either is low.
Python Metrics from a confusion matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Why accuracy can lie

Suppose only 1% of transactions are fraud. A "model" that always answers "not fraud" is 99% accurate and completely useless: its recall is 0. When classes are imbalanced, look at precision, recall, F1 and the cost of each kind of error. Missing a tumor (false negative) is far worse than a false alarm; flagging a good customer as fraud (false positive) annoys people. Choose the metric that matches the real cost.

Python The accuracy trap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The better model has lower accuracy (0.977 against 0.99) but it actually finds fraud. More on probability-based metrics such as ROC curves is in the machine learning classification metrics lesson.

Cross-validation

One holdout split can be lucky or unlucky. k-fold cross-validation splits the data into k parts, trains k times (each time holding out a different part) and averages the scores, which is a more stable estimate.

Python 5-fold cross-validation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Any step that learns from data (scaling, feature selection, imputation) must be fitted on the training part only, otherwise information leaks from the test set. Use a pipeline to keep this automatic.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why must a classifier be evaluated on records it did not train on?

    Memorizing the training set is overfitting. Only unseen records show how the model behaves in real use.
  2. 2. A fraud model flags nothing and is 99% accurate on data with 1% fraud. What does this show?

    The model finds none of the frauds, so recall is 0 even though accuracy is high.
  3. 3. Precision answers which question?

    Precision = TP / (TP + FP). The first option describes recall.
  4. 4. What does stratified splitting preserve?

    Stratification keeps the share of each class the same in train and test sets.

Summary

  • Classification predicts a category from features using a model learned from labeled records.
  • Always separate training and test data; a large train-test gap means overfitting.
  • The confusion matrix (TP, FP, FN, TN) is the base of accuracy, precision, recall, specificity and F1.
  • On imbalanced data accuracy misleads; pick the metric that matches the cost of each error.
  • Cross-validation gives a steadier estimate than one holdout split.

Next: Decision Trees