Skip to content
Tutorials Center
My learning

Topics

Lesson 113 of 149 ยท Data and the Web

Machine Learning with scikit-learn

Learn machine learning in Python with scikit-learn. Split data, train classifiers and regressors, measure accuracy, build pipelines, cross-validate and spot overfitting.

In ordinary programming you write the rules: "if the petal is shorter than 2.5 cm, it's a setosa". In machine learning you hand the computer examples where the answer is already known, and it works out the rules itself. Once it has learned them, it can answer for examples it has never seen. Spam filters, price estimates, product recommendations and fraud alerts all work this way.

scikit-learn (imported as sklearn) is Python's go-to library for this kind of "classic" machine learning on tables of numbers. Every model in it follows the same small set of methods, so once you've trained one, you can train dozens. In this lesson you'll load a dataset, split it into training and test sets, train classifiers and a regressor, measure how good they are, chain steps into a pipeline, use cross-validation and see overfitting happen with your own eyes. You'll want to know NumPy arrays; pandas and Matplotlib help too.

Note: scikit-learn is a third-party package (pip install scikit-learn on your own computer, then import sklearn). The editors here load it automatically, together with NumPy and SciPy. The first Run downloads them, which takes a little while; after that, every example on this page trains in well under a second.

Features, labels and the two big tasks

Machine-learning data is a table. Each row is one sample (a flower, a house, an email). The columns you learn from are the features, and the column you want to predict is the target or label. By convention, scikit-learn code calls the feature table X (capital, because it's 2-D) and the target y (a 1-D array with one answer per row).

There are two main kinds of prediction problem:

  • Classification predicts a category: which species, spam or not, which digit.
  • Regression predicts a number: a price, a temperature, a delivery time.

Both are supervised learning, because the training data includes the right answers. (Unsupervised learning, such as clustering, finds structure without any labels; it's outside this lesson.)

scikit-learn ships with a few small, classic datasets so you can practice without downloading anything. The best known is iris: 150 flowers, four measurements each, and three species:

Python Loading the iris dataset

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

X has 150 rows and 4 columns; y holds 150 numbers, where 0, 1 and 2 stand for the three names in target_names. Models work with numbers, so categories are almost always encoded as integers like this. Other built-in loaders include load_wine(), load_breast_cancer() and load_digits() for classification and load_diabetes() for regression. Pass as_frame=True to get a pandas DataFrame instead of arrays.

Train/test split: keep some data secret

If you grade a model on the same rows it learned from, you're checking whether it memorized them, not whether it can predict. The fix is simple: hold back part of the data as a test set that the model never sees during training, and measure on that.

train_test_split() shuffles the rows and cuts them into four arrays:

  • test_size=0.25 keeps 25% of the rows for testing.
  • random_state=42 fixes the shuffle, so you get the same split every run.
  • stratify=y keeps the class proportions the same in both parts, which matters for small or unbalanced datasets.

fit, predict, score

Every scikit-learn model is an estimator with the same interface:

  1. Create it, choosing its settings (called hyperparameters): model = KNeighborsClassifier(n_neighbors=5).
  2. Train it: model.fit(X_train, y_train).
  3. Use it: model.predict(X_new) returns predictions; model.score(X_test, y_test) returns a quick quality number (accuracy for classifiers).

Here's the whole loop with a k-nearest neighbors (k-NN) classifier. k-NN is the easiest model to picture: to classify a new flower, it finds the 5 training flowers with the most similar measurements and lets them vote.

Python Your first classifier

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The model gets 37 of the 38 test flowers right, an accuracy of about 0.974. predict() always takes a 2-D input, even for a single sample, which is why new_flower is a list inside a list. predict_proba() shows how confident the model is in each class; here all five neighbors are setosas, so it's 100% sure of class 0.

Comparing classifiers

Because every model has fit() and score(), trying several is a short loop. Three classic classifiers make a good starting set:

  • k-nearest neighbors (KNeighborsClassifier): votes among the closest training samples. Simple, but sensitive to the scale of the features.
  • Logistic regression (LogisticRegression): despite the name, a classifier. It learns one weight per feature and turns the weighted sum into a probability. Fast and a strong baseline.
  • Decision tree (DecisionTreeClassifier): learns a flowchart of yes/no questions such as "is the petal width โ‰ค 0.8?". Easy to explain, but prone to memorizing the training data.

The wine dataset has 13 chemical measurements for 178 wines from three growers. Some features are around 0.1 and others are over 1,000, which makes it a good test of how models react to scale. classification_report() shows the scores per class:

Python Three classifiers on the wine data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

k-NN comes last, and the reason is scale, which you'll fix in the next section. (max_iter=5000 gives logistic regression enough steps to finish training on unscaled data; without it you'd get a warning.)

Reading the results:

  • The confusion matrix has one row per actual class and one column per predicted class. Numbers on the diagonal are correct predictions; anything off the diagonal is a mistake, and its position tells you which classes get confused.
  • Precision for a class: of the samples predicted as that class, how many really were. Recall: of the samples that really are that class, how many the model found. F1 combines the two into one number.

Accuracy alone can mislead. If 99% of emails are not spam, a model that never says "spam" scores 99% accuracy and is useless. Precision and recall per class reveal that immediately.

Scaling and pipelines

k-NN measures distance between samples. When one feature (proline) runs from 278 to 1,680 and another sits between 0.13 and 0.63, the big one drowns out the rest. Standardizing fixes that: StandardScaler shifts and stretches every feature to a mean of 0 and a standard deviation of 1.

There's a trap, though. The scaler must learn its means and deviations from the training data only. If you scale the whole dataset before splitting, information from the test set leaks into training and your score looks better than it really is. A pipeline handles this for you: it chains steps into one estimator, so fit() fits the scaler and the model on training data, and predict() applies the same scaling to new data automatically.

Python StandardScaler in a pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Accuracy jumps from 0.722 to 0.963 just by scaling. Scaling matters for models based on distances or weights (k-NN, logistic regression, support vector machines, neural networks). Tree-based models split on one feature at a time, so they don't care.

Pipelines can hold any number of steps: fill in missing values (SimpleImputer), encode text categories (OneHotEncoder), scale, then predict. For tables with mixed column types, ColumnTransformer applies different steps to different columns.

Regression: predicting numbers

Regression uses exactly the same fit/predict pattern; only the models and the metrics change. The diabetes dataset has 10 measurements for 442 patients and a number that describes how far the disease progressed a year later. LinearRegression learns one weight per feature:

Python Linear regression and its metrics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

  • MAE (mean absolute error) is the average size of a miss, in the target's own units. Lower is better.
  • Rยฒ compares the model with always guessing the average: 1.0 is perfect, 0 means "no better than the average", and it can even go negative. For a regressor, model.score() returns Rยฒ.
  • Always compare with a baseline. Here the model's MAE of about 42 beats the "always predict the mean" baseline of about 62. The Rยฒ of 0.444 shows that these measurements only explain part of the story, which is normal for medical data.

Overfitting: when a model memorizes

A decision tree can keep asking questions until every training sample sits in its own leaf. Then it scores 100% on the training data, but it has learned the noise along with the pattern, and it does worse on new data. That's overfitting. The opposite, a model too simple to capture the pattern, is underfitting.

You can watch it happen by growing trees of increasing depth on a noisy dataset and scoring both sets. make_moons() generates two interleaving half-moons of points with some noise:

Python Watching a tree overfit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Training accuracy climbs all the way to 1.000, but test accuracy peaks at depth 2โ€“3 (0.867) and then slides down to 0.767. The widening gap between the two lines is the signature of overfitting. Ways to fight it:

  • Limit the model's complexity with hyperparameters: max_depth or min_samples_leaf for trees, a larger n_neighbors for k-NN, stronger regularization (smaller C) for logistic regression.
  • Collect more training data.
  • Use an ensemble such as RandomForestClassifier, which averages many slightly different trees.

Cross-validation and tuning

A single train/test split is a bit of a lottery: a lucky split flatters the model, an unlucky one hides its strengths. Cross-validation removes the luck. With 5-fold cross-validation, the data is cut into 5 parts; the model is trained 5 times, each time tested on a different part and trained on the other 4. You get 5 scores, and their mean is a far more reliable estimate.

GridSearchCV builds on this to pick hyperparameters: it cross-validates every combination you list and keeps the best. Because it only uses the training data, the test set stays a fair final check:

Python cross_val_score and GridSearchCV

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Inside a pipeline, a hyperparameter's name is the step name (the class name in lowercase when you use make_pipeline), two underscores and the setting: kneighborsclassifier__n_neighbors. After the search, search itself behaves like the best model, retrained on all the training data, so you can call search.predict() or search.score() directly.

A typical workflow

Putting it all together, most scikit-learn projects follow the same steps:

  1. Load the data into X and y (often from a pandas DataFrame; plot it with Matplotlib to understand it).
  2. Split off a test set and don't touch it until the end.
  3. Build a pipeline of preprocessing steps and a model.
  4. Compare models and tune hyperparameters with cross-validation on the training set.
  5. Score the final model once on the test set and report that number.
  6. Save the trained model with joblib.dump(model, "model.joblib") and load it later with joblib.load(). Only load model files you trust, because, like pickle, loading can run code.

Tip: start with a simple model and a baseline. scikit-learn's DummyClassifier and DummyRegressor predict the most common class or the mean. If your clever model can't beat them, something is wrong with the data or the features, not the algorithm.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why do you score a model on a test set it never saw during training?

    A model can memorize its training rows. Only unseen data tells you whether it learned a pattern that generalizes.
  2. 2. A tree scores 1.000 on training data and 0.767 on test data, while a shallower tree scores 0.871 and 0.867. What is going on?

    A big gap between training and test scores means the deep tree learned the noise in its training data.
  3. 3. Why should a StandardScaler go inside a pipeline instead of scaling the whole dataset first?

    Fitting the scaler on all rows lets the test set influence training; a pipeline fits each step on the training data only.
  4. 4. You want to predict tomorrow's temperature in ยฐC. What kind of problem is this, and which metric fits?

    The target is a number, so it's regression; MAE reports the average miss in degrees.
  5. 5. What does cross_val_score(model, X, y, cv=5) return?

    The data is split into 5 folds and the model is trained and scored 5 times; the mean of the scores is a steadier estimate than one split.

Summary

  • Machine learning learns rules from examples. Features go in X (2-D), the target in y (1-D). Classification predicts categories; regression predicts numbers.
  • Split with train_test_split(X, y, test_size=..., random_state=..., stratify=y) and judge models only on data they didn't train on.
  • Every estimator has the same interface: create with hyperparameters, fit(X_train, y_train), then predict() and score().
  • k-NN, logistic regression and decision trees are good first classifiers; LinearRegression is the first regressor to try.
  • Look beyond accuracy: confusion matrix, precision, recall and F1 for classifiers; MAE and Rยฒ for regressors; always compare with a baseline.
  • Put preprocessing such as StandardScaler in a pipeline so it learns from training data only.
  • Overfitting shows up as a high training score with a much lower test score; limit model complexity and use cross-validation (cross_val_score, GridSearchCV) to choose settings.

Next: Neural Networks: TensorFlow and PyTorch at a Glance, where you'll build a tiny neural network from scratch and see how the big deep-learning frameworks fit in.